compsci.science - Towards predictive theory of computational realities.

Common sense in Scholarly Peer Review

A great instruction for reviewers can be found here. It mentions some important points for good reviews:

Review accurately and completely reflects the contributions of the paper (think about what the authors really wanted to say).
Concerns raised are specific with enough details so that the authors can act on them.

Proposal: Positive Review Bidding

Reviewer number three is a popular monicker for reviewers that make peer-review an unnecessarily unpredictable and unpleasant experience. Fortunately, if the problem lies with just one out of three reviewers, it may be feasible to weed them out with a little help from the authors (who gain a bit of power in the process):

Submission. For each paper ask the authors:
- To write a description of the target audience of the paper.
Bidding. Each reviewer bids on papers based on:
- Abstract + target audience (can be at the end of abstract).
I. Positive Reviews. For each paper ask more than 3 reviewers to write :
- Any questions about things that were unclear from the paper.
- The neutral part (summary) + positive part (strengths) of the review.
- no decision!
II. Reviewer Selection. For each paper ask the authors of the paper:
- To select 3 reviewers based on the neutral/positive part of the review.
- To answer all reviewer questions (not knowing who posed which question).
III. Full Reviews. Ask the 3 author-selected reviewers to complete their reviews (including the decision) with access to all reviewer questions and answers.

The most important underlying assumptions of the proposal:

Reviewing antagonistic in nature (e.g., burden of proof lies with authors)
Suitable reviewer → gets the point of a paper (e.g. used progress metric)
Positive reviews → hints to authors who are suitable reviewers
Effort writing positive reviews < effort writing full reviews
Open-mindedness before decision > after making/justifying decision

Some potential advantages and disadvantages to consider:

	Pros	Cons
More scientific value?	Reviewers entirely missing the point of the paper less likely to be selected by authors	Authors may avoid some more critical reviewers (that seem to be less able to write a convincing positive review)
Meaningful interaction?	Reviewers receive author answers (to questions posed at positive review stage) before being asked for a decision	Dismissive reviewers may ask leading questions to influence others (mitigated by author responses)
Respectful?	Little bit of power given to authors (that are also experts deserving respect!)	Some wasted effort of reviewers not selected by authors
Considers human nature?	Some negative instincts may be inhibited if only a positive review is requested at first	Some reviewers may try to trick authors by making up positives (easy to spot?)
Implementable?	Can be implemented in CMTs [1,2,3] as rebuttal to positive reviews	No rebuttal to full reviews (which is perhaps in any case too late)
Feasible?	Positive reviews without a decision are easier to write?	Some reviewers may refuse to write a purely positive review without a decision?
Emotionally sustainable?	Hope for authors with bad experiences?	Unconventional and potentially confusing for reviewers
Prevents accidental bidding?	More clarity on targeted progress metrics through explicit description of targeted audience	Authors/reviewers may be confused by audience descriptions (that are only visible during reviewing process)

Some additional questions and answers:

Technical Implementation
- Q1: Does this proposal require changes / adding functionalities to common reviewing systems?
- A1: No. The target audience can be described at the end of the abstract and the reviewer selection / question answering can be implemented as a rebuttal.
Positive Reviews
- Q2: How many positive reviews should be collected per paper?
- A2: Generally 4-5 is a good number. It would be difficult to get more positive reviews for each paper and at some point it gives authors too much power.
- Q3: Why let reviewers write only positive reviews first?
- A3: To force reviewers to approach the paper positively at first without any need to make a decision (which is often the hard part) and to allow authors to select reviewers.
- Q4: Why is the reviewer selection only based on positive reviews?
- A4: Authors would obviously want to avoid negative reviews, but all suitable reviewers should be able to write a competent positive review.
Reviewer Selection
- Q5: Why let authors select reviewers?
- A5: The authors know best which reviewers may lack a sufficient understanding to review the paper and this is a way to give authors a little bit of power in the process.
- Q6: Why is the reviewer selection only based on positive reviews?
- A6: Authors would obviously want to avoid negative reviews, but all suitable reviewers should be able to write a competent positive review.
- Q7: Why give authors any power, would they not just abuse it?
- A7: It's not enough power to realistically avoid critical reviews. Giving a little bit of power can remind everyone involved that this is meant to be a meaningful scientific interaction rather than one-sided judging akin to casting shows.
Non-Selected Reviewers
- Q8: Do the non-selected reviewers simply waste their time?
- A8: No, the positive review is visible to the authors and their questions (no breakdown of who posed which question) are visible to everyone along with author answers. The author answers are also given to non-selected reviewers.
- Q9: Are the positive reviews of the non-selected reviewers visible to selected ones?
- A9: No, but mostly due to the positive reviews potentially not being positive reviews.
- Q10: What if non-selected reviewer(s) spot important problems?
- A10: These can be raised in the questions posed to the authors. The selected reviewers get all reviewer questions as well as author answers.
- Q11: Can questions be abused to influence selected reviewers?
- A11:: Potentially, yes. Reviewers may try to share their opinions through leading questions, but they are never presented without the author response.
- Q12: Are reviewers which rarely get selected by authors (for full reviews) necessarily bad reviewers?
- A12: No, not necessarily. One would still need to look at the details to draw any meaningful conclusions.

Accountability: Reviewing in the Open?

An interesting idea to help authors and reviewers to reflect upon the process is to open up the reviews to the public. This is for instance practiced in machine learning conferences such as ICLR and NeurIPS. Such an approach maintains the anonymity of reviewers for the most part and introduces a minimal amount of accountability into the process, as sloppy and presumptious reviews may reflect poorly on a conference. While most computer science conferences move towards double-blinded reviewing, it may also be useful to consider experimenting with non-anonymous reviewing. Ideally, multiple models could coexist such that authors and reviewers are given the choice. Everything being out in the public could potentially hurt some relationships and put too much stress on reviewers, but anonymity also has well-known disadvantages and disassociates reviewers from the valuable work they do. Which approach works best depends a lot on the authors and reviewers involved. If everyone involved is motivated by a search for knowledge, then a system is barely needed. If instead everything is just about the egos and careers of reviewers and authors, even the best system can only do so much. If the problem is more of a cultural nature, it likely cannot be solved behind closed doors and would require to shine a light on the problem. In conclusion: A look behind the curtain could be very helpful.