Reviewing and being in a program committee

A Short and Partial Guide

March 14, 2022

This (very) short and partial guide aims at sharing some useful references and tips to be a constructive reviewer.

Why?

There are many reasons, and you can read what e.g. the Journal of Public Health Research, Elsevier or Wiley think about it. In short, reviewing let you

  1. Learn about recent trends in your field,
  2. Give back to your community,
  3. Network.

Yes, it is voluntary, and yes, it is actually free labor for editors that (sometimes) make huge benefits (cf. “The Cost of Knowledge”), but it is also a cornerstone of scientific progress.

How Do I Volunteer?

Generally, you don’t.

There can be, in some rare cases, the possibility of self-nominating yourself to be part of a program committee (as PLDI Research Artifacts did), but generally, a program committee member will reach out to you directly, because they know you, because your work is referenced in the work they need to review, or because your name was recommended to them.

How Do I Review?

There are three excellent sources to learn more about this process (Smith 1990; Cormode 2008; Parberry 1994; Knuth, Larrabee, and Roberts 1996, secs. 15–16), and I believe that they give better advises than I ever could.

You can also read about the process of reviewing on various blogs, by Matt Might, John Regehr or Michael Hicks.

If you prefer a “short, list-style” type of advises, then I believe all the tips in this twitter thread are relevant:

Summarize first. ☂️
Begin by summarizing key ideas/contributions of the paper, even if it is not required to do so. Don’t copy-paste the abstract. This is useful for committees/editors to get context, and for authors to identify and resolve misunderstandings.
Let authors explain their work their way. 📢
Sometimes others will use different terminology to describe their work than you would use. As long as they are not strictly technically wrong, it’s their right to do so, respect it! At most, passively suggest other descriptions.
Make the follow-up discussion easy. ⛵️
If you are critiquing, or asking questions which need a response in a rebuttal, put them in a clear, numbered list of < 4 points near the end of the review. Distinguish them explicitly from smaller nitpicks and asks, typos, etc.
Don’t write reviews at the last minute.
If you agree to review, immediately budget some time to do so. Put it on your calendar! I like to write my reviews over at least 2 sittings, because sometimes opinions shift when you sleep on them.
Avoid borderline, unsubstantial reviews. 🎭
If you do not express any opinions, you are wasting everyone’s time with your review! Find something to latch on to and give concrete feedback about.
Don’t be afraid to have an opinion. ⚖️
As an early-career scientist, it’s easy to think your opinions are probably wrong. But you were asked to review because someone wanted your opinion! Express it and defend it! Conversely, always be open to changing your mind.
Never be selfish. 🦪
I shouldn’t have to say this, but never treat reviewing as an opportunity to advance your own work. Doing so hurts us all, and embarrasses yourself. Only bring up your own work if it is truly technically relevant (which it sometimes is).
Look for reasons to accept papers. 🚀
Science is rarely perfect; procedural mistakes happen. Ask, could someone benefit from reading this? Is there something interesting to build on? Is it exciting? As long as a paper is not strictly technically wrong, imperfect is okay.
Remember the human. 🫂
If you are giving negative feedback, imagine a grad student reading it who has poured their heart and soul into the project. Always include some positives. Use wording like “this submission does not do X”, rather than “the authors do not do X”.

How Do I Answer a Review?

You can read about it on Matt Might and Michael Hicks’s blogs.

What Should I Do if I am a PC Member?

Celebrate! And then, read a bit more about that process, probably (Dulek et al. 2021).

How Secretive Should I Be?

That is a difficult question, very field- and seniority-sensitive, I believe. There are legal obligations that you should review, of course, but whether it is acceptable to ask high-level questions about the papers to colleagues or not is, as far as I know, a very personal matter.

References

Cormode, Graham. 2008. “How NOT to Review a Paper: The Tools and Techniques of the Adversarial Reviewer.” SIGMOD Record 37 (4): 100–104. https://doi.org/10.1145/1519103.1519122.
Dulek, Yfke, Stacey Jeffery, Christian Majenz, Christian Schaffner, Florian Speelman, and Ronald de Wolf. 2021. “A Guide for New Program Committee Members at Theoretical Computer Science Conferences.” ArXiv Preprint. https://doi.org/10.48550/ARXIV.2105.02773.
Knuth, Donald E, Tracy Larrabee, and Paul M Roberts. 1996. Mathematical Writing. Mathematical Association of America Notes. Washington, D.C., DC: Mathematical Association of America. https://jmlr.csail.mit.edu/reviewing-papers/ knuth_mathematical_writing.pdf.
Parberry, Ian. 1994. “A Guide for New Referees in Theoretical Computer Science.” Information and Computation 112 (1): 96–116. https://doi.org/10.1006/inco.1994.1053.
Smith, Alan Jay. 1990. “The Task of the Referee.” IEEE Computer 23 (4): 65–71. https://doi.org/10.1109/2.55470.

Miscellaneous