Why?

There are many reasons, and you can read what e.g., the Journal of Public Health Research, Elsevier or Wiley think about it. In short, reviewing let you

1. Learn about recent trends in your field,
2. Give back to your community,
3. Network.

Yes, it is voluntary, and yes, it is actually free labor for editors that (sometimes) make huge benefits (cf. “The Cost of Knowledge” or, more succinctly, this xkcd), but it is also a cornerstone of scientific progress (ok, you can find articles that disagree with that).

How Do I Volunteer?

Generally, you don’t.

There can be, in some rare cases, the possibility of self-nominating yourself to be part of a program committee (as PLDI Research Artifacts did), but generally, a program committee member will reach out to you directly, because they know you, because your work is referenced in the work they need to review, or because your name was recommended to them.

How Do I Review?

There are three excellent sources to learn more about this process (Smith 1990; Cormode 2008; Parberry 1994; Knuth, Larrabee, and Roberts 1996, secs. 15–16), and I believe that they give better advises than I ever could.

You can also read about the process of reviewing on various blogs, by Matt Might, John Regehr or Michael Hicks. You can also read a funny take on this question by Peter Sewell, called Bad Reasons to Reject Good Papers, and Vice Versa or, if you are particularly interested in Mathematics, this short article by Álvaro Lozano-Robledo.

If you prefer a “short, list-style” type of advises, then I believe all the tips in this twitter thread are relevant:

Summarize first. ☂️

Begin by summarizing key ideas/contributions of the paper, even if it is not required to do so. Don’t copy-paste the abstract. This is useful for committees/editors to get context, and for authors to identify and resolve misunderstandings.

Let authors explain their work their way. 📢

Sometimes others will use different terminology to describe their work than you would use. As long as they are not strictly technically wrong, it’s their right to do so, respect it! At most, passively suggest other descriptions.

Make the follow-up discussion easy. ⛵️

If you are critiquing, or asking questions which need a response in a rebuttal, put them in a clear, numbered list of < 4 points near the end of the review. Distinguish them explicitly from smaller nitpicks and asks, typos, etc.

Don’t write reviews at the last minute. ⏰

If you agree to review, immediately budget some time to do so. Put it on your calendar! I like to write my reviews over at least 2 sittings, because sometimes opinions shift when you sleep on them.

Avoid borderline, unsubstantial reviews. 🎭

If you do not express any opinions, you are wasting everyone’s time with your review! Find something to latch on to and give concrete feedback about.

Don’t be afraid to have an opinion. ⚖️

As an early-career scientist, it’s easy to think your opinions are probably wrong. But you were asked to review because someone wanted your opinion! Express it and defend it! Conversely, always be open to changing your mind.

Never be selfish. 🦪

I shouldn’t have to say this, but never treat reviewing as an opportunity to advance your own work. Doing so hurts us all, and embarrasses yourself. Only bring up your own work if it is truly technically relevant (which it sometimes is).

Look for reasons to accept papers. 🚀

Science is rarely perfect; procedural mistakes happen. Ask, could someone benefit from reading this? Is there something interesting to build on? Is it exciting? As long as a paper is not strictly technically wrong, imperfect is okay.

Remember the human. 🫂

If you are giving negative feedback, imagine a grad student reading it who has poured their heart and soul into the project. Always include some positives. Use wording like “this submission does not do X”, rather than “the authors do not do X”.

How Do I Not Review?

Alternatively, if you want to know what you should not do, you can refer to this thread:

In the spirit of David Patterson’s “How to Have a Bad Career in Research/Academia” talk here are 10 tips I just shared with the @PLDI Program Committee for winning the Undistinguished Reviewer Award.

1. Argue to reject papers because they are not to your personal taste. If the topic is something that you are not excited about, it’s unlikely that anyone in the PLDI community will appreciate the work.

2. Argue to reject papers because you would have preferred they be written differently. Better to delay publication of the technical ideas, delay a PhD student’s progress toward their degree, etc. than to have the paper be written in a way you don’t like.

3. Argue to reject papers because you would have used a different approach – especially if they didn’t use the approach you invented! Introducing non-standard approaches into the literature is confusing, even if they reveal connections to other areas that may inspire interesting follow-on work.

4. Argue to reject papers simply because the proofs are not mechanized or the code is not publicly available. Regardless of whether a paper makes a significant advance and provides sufficient evidence to support its claims, the field now expects full mechanization and open artifacts.

5. Argue to reject papers because you’d like to see more experiments, better theorems, etc. Even if the the technical claims are already well-supported by the evidence presented in the paper, it will be even better after the next revision.

6. Argue to reject papers because the ideas are “too simple.” Even though simple approaches that work well are often the ones that have the most impact in the long run, we should be selecting papers that optimize for technical complexity.

7. Argue to reject papers based on anecdotal evidence. For example, “I heard this paper was rejected previously” or “I used tool X once and it didn’t work” are fine things to bring to the discussion of PLDI submissions.

8. Argue to reject papers because the authors forgot to cite a few tangentially-related papers. You should especially do this if the omitted paper is your own – after all, if you’ve written on the topic, you are the expert!

9. Argue to reject papers because they lacks citations to or comparisons with unpublished work. PDFs on the author’s website or manuscripts on arXiv are flags planted in the ground and fair game!

10. Argue to reject papers because they build on prior work that you are unfamiliar with. If you don’t have the background to understand a paper, it’s unlikely that anyone in the PLDI community will appreciate it.

How Do I Answer a Review?

You can read about it on Matt Might and Michael Hicks’s blogs.

What Should I Do if I am a PC Member?

Celebrate! And then, read a bit more about that process, probably (Dulek et al. 2021).

How Secretive Should I Be?

That is a difficult question, very field- and seniority-sensitive, I believe. There are legal obligations that you should review, of course, but whether it is acceptable to ask high-level questions about the papers to colleagues or not is, as far as I know, a very personal matter.

References