RICELab Rethinking Interaction Collaboration and Engagement

Before I get too far in, let me just say that overall, CHI has treated me very well. I love the projects that I see at the conferences, I think the people in it are bright and intelligent, and it has helped to give me a career in doing stuff that I really enjoy (research in HCI). I was also very lucky to attend the recent CHI program committee meeting in Seoul, and thoroughly enjoyed the experience (as I described to a friend, it was kind of like going to a really great reading group). I also personally fared reasonably well with several papers (conditionally) accepted. So, overall: good.

This “rant” is about the paper reviewing/selection process that CHI uses, and a structural flaw in its use of external vs. and secondary reviewers. I’ll begin by describing how the process works (right now). Then I’ll get into my rant. Unfortunately, I don’t have any great new solutions… I don’t think I have enough experience to figure out what great new ones would be. Instead, I propose only a few “well-known” solutions.

##CHI Review Process Once an author submits his/her paper to a review subcommittee, the reviewing machine begins. The CHI review process has five separate phases, and three different types of reviewers. The five phases are as follows: first, the subcommittee chairs assign the paper to a primary reviewer (a person that is on the program committee — usually, this is R4); second, the primary reviewer assigns three external reviewers (external to the program committee) to review the paper; third, the authors get to see the first four reviews (from R1, R2, R3 and R4) and write a rebuttal/response; fourth, the submission, if it is “good enough”, is assigned a secondary reviewer (another member of the program committee — usually, R5) to read in time for the program committee meeting; finally, at the program committee meeting, additional committee members (R6, and so on) are assigned to offer an opinion, where necessary.

There are three different kinds of reviewers here:

• the primary (usually R4): a member of the program committee that is responsible for the paper submission, first by selecting the external expert reviewers, and then by summarizing their reviews — hopefully this person is an expert in the area being covered by the submission;
• the external reviewers (usually R1-R3): three experts in the area called upon by the primary to provide an opinion on the suitability of the paper for publication, and finally,
• the secondary (and additional “secondaries”): members of the program committee that are asked to “weigh in” on submissions at the program committee meeting (with the caveat that the secondary usually reads it prior to the meeting itself).

There is a rationale for the process right now. First, the rebuttal allows for the authors to clarify things that the original set of reviewers (R1-R4) “got wrong.” A secondary (R5) is assigned only to submissions that stand a chance of getting in — this saves resources on the program committee side of things, as submissions that don’t really stand a chance don’t need to get assigned a secondary. Finally, in the PC meeting itself, sometimes the primary and secondary disagree about the fate of a submission (or need additional expertise), and it is here that additional secondary reviewers are assigned (R6+). These additional secondary reviewers give the paper a quick read and offer an opinion.

##The Structural Problem The structural problem in the CHI reviewing process is that reviewers that come online later in the process have essentially veto power over any reviewers earlier (and usually, these reviewers have had even less time with the paper). This happens in at least two ways:

• A common thing I saw, for instance, was when a secondary (R5) disagrees with the original set of reviewers (R1-R4). Let’s say the original set of reviewers thought the submission was good, and now the secondary disagrees. This only comes up at the PC meeting–past the time that the authors can rebut anything–meaning that the authors do not get a chance to respond to R5’s concerns/comments. Not only that, but none of the original reviewers get to respond, either. [Footnote: Let me just say that while I saw this, I didn’t think it ever happened maliciously that R5 would disagree in this way. In fact, R5 is trying to help the community by making sure that only great papers make it in, and not something subpar.]
• I also saw instances where additional program committee members were added to the review team (i.e. other secondaries) to offer up an opinion. Here, the person gives it a quick read (at most they are given the night), and expected to offer up a review. This review usually trumps anything that happened before. (In some ways, the rationale for this trumping is obvious — that is, why would s/he be recruited if the decision was already clear?)

Thus, the structural problem results in two effects: first, subsequent reviewers essentially have veto power over anything that happened before, and second, the authors do not get a chance to rebut anything that happens after R4’s review — this is problematic, particularly if subsequent reviewers actually misread something (or interpreted something incorrectly).

##Why this is crazy

I think fundamentally, this is crazy because the philosophically, the role of the three external reviewers is to offer an expert opinion. Cynically: if members of the PC (who, admittedly, have been entrusted with the direction of the research community) can overturn the opinion of the three experts, then why even bother the experts to begin with?

I also think it is crazy because the rebuttal is almost in the wrong place — if the purpose of the rebuttal is to clarify misinterpretations in the reviewers’ reading of the submission, then why is it that an author only gets to rebut a small portion of the reviewers? Just as an extreme variation on the rebuttal idea to make the point — imagine you wrote the rebuttal after seeing only the first two reviews that came in. Just as external reviewers can sometimes get it wrong (i.e. misread the paper), committee members can also misread something (and frankly, are more likely to, as they are responsible for reading and offering an opinion on submissions with a heavy time pressure).

##Why this ends up being unfair

I saw a few papers (and experienced one myself) where the reviews going into the PC meeting (i.e. from R1-R4) were on average, quite good (i.e. >3.5). This gives the initial impression that, “Things will be fine,” and thoughts of, “Hey! This is a slam dunk! Woohoo!” And then, we (and others) were probably quite surprised to find that the paper had been kaiboshed at the PC meeting. This feels profoundly unfair… (In our case, it feels extremely unfair because the secondaries actually misread something in a factually incorrect way — again, something a rebuttal could have cleared up.)

##Why should we care

Tenure and promotion decisions are made on the basis of this stuff. We should try to get the process right, otherwise, we are unfairly messing with people’s careers due to a (slightly) random process. In the worst case, we’ve messed up someone’s career. In the second worst case, people stop going (and submitting) to CHI.

##Solutions I’m a young guy — still wet behind the ears. I haven’t thought long about this, nor do I have the experience to have seen how else it could have been done. Nevertheless, here are some solutions (half-baked as they may be) to the structural problem I outlined earlier.

• Get rid of external reviewers. Hey, if we don’t care about what they have to say, then why tax the review community more? (I’m being sarcastic. Well, maybe not. Maybe.)
• Submit to a journal. This was Dan Olsen (BYU)’s solution to this mumbo jumbo. He basically said, “Screw CHI’s review process.” His sense was that journal review processes were far saner (no time pressure to get it right), even if it were slower.
• Remove the rebuttal phase. Maybe the reason it feels this profoundly unfair is that you get a glimpse in the process early on, but it sometimes ends up going against you. I kind of equate this to an undergrad whose assignment gets first graded by a TA (and then you get a peek), before it gets graded by the prof (when you finally get kaiboshed). This would feel far more unfair than if you just found out what the result was at the end… yes, still unfair, but you don’t need to get the false sense of hope ever.
• Move the rebuttal phase. As I hinted above, maybe we can move the rebuttal phase to after (at least) the (first) secondary has put in his/her opinion and review. I understand this could further tax the system and timeline, but at least it might feel a little more fair. It removes one of the stumbling blocks (as outlined above), though if both R4 and R5 look uncertain going into a PC meeting (whereupon more secondaries would get assigned), at least it would be obvious to the authors.

Conclusion

In spite of this, I have a lot of faith in the CHI community. When I see the kinds of things that Jofish is talking about on his tumblr blog, it makes me feel better — other people are seeing these problems too. It is a good thing smarter people are working on figuring out good solutions. Crossing fingers.

TLDR; We got screwed. I ranted. Smart people are on the case.