My primary goal in this post is to impart my philosophy on reviewing HCI conference paper submissions. This guides how I think a review ought to be formulated – what are its constituent parts, and what is the role of each part. Thus, a secondary goal is to suggest how a review can be constructed by providing a bit of a template.
As context, I have written and reviewed HCI papers since about 2003 or so. I have received a lot of really great and useful reviews, and I have also received a lot of nasty and not-so-useful reviews. This means I have been heartbroken a lot of times. I have also served on a program committee a few times, which means that as one of the people trying to decide what submissions are accepted as papers, I have been on the “first part of” the receiving end of reviews. In this context, I have seen a lot of good reviews (useful) and also a few not-so-useful reviews.
My Philosophy
- A review is a one-direction communiqué from an “expert.” I always think about the days of Newton, where he would write letters to his contemporaries, and these were the “papers” of the day. Colleagues would write back, offering suggestions and ideas. This was the main form of intellectual exchange of the day: Newton says, “Hey, I think X now, because I did Y and Z, and got R results! What do you think?” And his friend writes back and says, “Yo, X1 makes sense, but X2 totally does not make sense – it’s not supported by R yet. Maybe you can do Z2, and if you got R2, that would help.” These letters and exchanges are about helping one another and “building science” through writing and supporting arguments.
Peer-reviewed conference papers are sort of a “mass-scale” version of this in terms of dissemination. Reviews are, unfortunately, a one-directional communiqué from an “expert.” The one-directional nature means that the author doesn’t really get to do a back-and-forth to clarify problems, and in more recent times, it isn’t clear that the reviewer is always an expert in the space (at least, I rarely feel like I am an expert in most of the things I review).
But, the philosophy should still be the same: the author is saying, “Hey! I think X because I did Y and got result R. What do you think?” And the reviewer is writing back and saying, “Oh yeah, that makes sense,” or “Oh wait, something doesn’t make sense here… Did you think of this?”
-
Authors want to make sure they are understood. One of the most frustrating things as an author is when I feel like the reviewer didn’t understand the paper as I wrote it. To this end, the reviewer should try to make it clear at the outset: “This is how I understood the paper.” This serves as a gut-check for the authors–Did I write it clearly enough for the reviewer to understand?
-
Assume the authors are doing their best: every work likely has some merit. I think that authors, when they submit something, are trying to submit only because they think the work has merit and ought to be published. To this end, I try to figure out what the main idea was – even if the authors didn’t do a good enough job of articulating it themselves. What was the intended contribution, if it wasn’t stated?
-
Authors have blindspots, but as a reviewer, I have blindspots too. Invariably, we all have blindspots when we make arguments. To this end our job as a reviewer is to help identify those blindspots for the authors, and how to try to fix the blindspot. That said, I think particularly in HCI, it is important to be open to new ways of doing research, new ways of doing science, and new ways of contributing knowledge. To this end, I try to be open to new approaches to how research can be done, and avoid being too mono-culture in terms of how I think things can/ought to be done.
-
Most HCI research is not Chemistry. Most HCI research is not going to have results that are as clean as Grade 11 titration experiments/studies. To this end, it is mostly about a well-thought-through and clear argument in a paper. When uncertain, I provide a higher recommendation rather than a lower one. I think on balance, most papers are right around the middle, and over time, what will matter more is whether it contributes to the overall state of knowledge in the research space. In this sense, don’t worry about accepting a few papers that aren’t that strong – over time, the research community will correct that via citations.
-
A paper is an argument. A review is also an argument. While this is the last bullet in this list, it is actually the most important thing. A paper is an argument, and to this end, it is made up of a set of arguments (where evidence is used to support the argument). If I am arguing in my review that there is a weakness or a problem, this should be saying that there is a problem with the argument in the sense that “the evidence does not support the argument,” or “the evidence is not strong enough to support the argument.” It is important to think about it this way because it helps to make clear what my review is. My review is also an argument – one that others can disagree with, but that I can strengthen by providing evidence or through logic. Why is this a weakness? How do I argue that it is a weakness?
My Template
- Paragraph 1: Demonstrate that I understand what the paper is about. I try to write this in 4-5 sentences, and it is an attempt to be fact-based. Sentence 1: What is the problem/question the authors are exploring, and what is the motivation? Sentence 2: What is the approach the authors are taking to study the problem/question? Sentence 3: How do they evaluate their approach, or what is the main (set) of findings from the study? Sentence 4: What is the contribution of the work? Sentence 5. Free sentence.
- Paragraph 2: Describe the primary contribution and strengths of the work. In a longer one paragraph form, I reiterate what the primary contribution of the work is, and particularly in a stronger submission, I discuss why the contribution is sound given the authors’ arguments (work) as described in the paper. This can sometimes be up to two paragraphs in length, and sometimes will be used to praise the mechanics of the submission, too (e.g. “Yo, the framing of the work is super clever”, or “Wow, the study design is ingenious!”, or “Hot damn, I wish I’d thought of this idea.”).
- Paragraph 3: Offer a recommendation to the program committee. First sentence reiterates my recommendation to the committee (accept, reject, etc.). Then, I now build my own mini-essay to support my recommendation. If I am recommending accept, then I reiterate what was in Paragraph 2, and describe why this should be accepted. Otherwise, I briefly state what the weaknesses are (to be elaborated on below).
- Paragraph 4: Describe each weakness in turn, with recommendations on how to improve it. To make this legible, sometimes this is done as a set of bullet points, or mini-paragraphs. Remember that this part is about making an argument. If there is something weak in the submission, then it needs to be a weakness in the sense that it does not support the main argument or it does not do a good enough job of supporting the main argument. So, given this is the case, I always try to provide a fix on how to improve the weakness. This is where things get tricky, but the point about the “weakness of the argument” is to try to get away from an opinion (e.g. “well, I didn’t think they talked enough about X” – the question is WHY is it not enough? is that an opinion (i.e. immaterial), or is it because by not talking enough about X, they are actually hindering the argument?). Really think about how can this be improved and offer this up. Doing so makes the review constructive rather than emotionally destructive. Note: if these weaknesses can be fixed in a half day’s worth of work, I consider them to be, something that can be fixed in a revision. These are not something that a submission should be rejected for.
- Paragraph 5: Describe opportunities for improvement. Here is where the laundry list of “minor nitpicks” can go. None of these are problems with the argument, but usually smaller things with presentation – typos, problems with the figures, and so on.
Each paragraph in this structure serves a distinct purpose. Para 1 tells the author “This is how I understood your paper.” Para 2 makes the author feel good. Para 3 is mainly for the program committee – it is about what your recommendation is, and why you think it is what it is. Para 4 plays two roles: it serves to support your para 3 recommendation, and also for the authors on how to improve the work. Para 5 is mainly for the perfectionist in you, letting you get things off your chest.
Things to Avoid/Other Tips
- Avoid expecting the methods to look exactly like you’d see in a textbook. A textbook example of a method is necessarily a very abstract, simplified study so that for the learner, it is easy to see the main idea how the study/methods are executed. Reality is much messier, and much more difficult to work with. To this end, we need to be more flexible. Avoid making judgements on a submission because its study is not perfect – instead, consider the main argument of the study, and whether the evidence is sufficient to make the case.
- Avoid ticking off “Accept this paper”, but then providing only a list of problems. This is really difficult for the PC to deal with, because it is unclear what you really mean. To avoid this, if you are arguing to accept the paper, then you really want to back it up with what the strengths or contributions of the paper are.
- Avoid providing a decision without any supporting evidence. (See above)
- Avoid providing an opinion that is strictly that – an opinion. A recommendation ought to be based on an argument as to why something is a weakness. Ideally, if there is a weakness, a fix ought to be offered up, too.
- Sometimes one or a handful of citations is missing; however, this is rarely a cause to reject a paper submission. Reflect carefully on whether the missing reference is actually weakening the argument the authors are making.
- Focus on the ideas in the paper rather than the writing quality. The ideas are the central point of the idea – often, writing style is a matter of taste.
- Be aware that there are many ways to “arrive at knowledge” in HCI, and be aware of your own intellectual background. Early on, we tend to think of the world and knowledge mainly from the school of thought from which we were trained; it is difficult to understand other schools of thinking and how these other schools arrive at knowledge, but be aware they exist. Think of this as an ongoing learning opportunity for yourself. For me, I was trained from a quantitative experimental psychology background, so seeing ethnographies and interview studies was really difficult in the beginning.
My thanks to Carman Neustaedter, who offered comments on an earlier draft of this.
Resources
Others have written about this before, too.
- http://mobilehci.acm.org/2015/download/ExcellenceInReviewsforHCICommunity.pdf
- http://chicourse.acagamic.com/wp-content/uploads/2017/05/How-to-Write-and-Review-CHI-Papers-CHI2017.pdf