Posted 14 Nov 2014
A program committee meeting (particularly face-to-face meetings) is a very interesting experience. I’ve only been invited to a small handful (ITS 2009, ITS 2010, MobileHCI 2012), but they have been thoroughly eye-opening in terms of the process. I thought I’d post this as a reflection of my experiences. I’ll divide this into about three different sections: (1) a description of how the meeting runs; (2) some commentary about PC meetings, and finally (3) some thoughts on how I prepare for PC meetings. I’d really rather skip straight to (3), but I think some of (1) and (2) are necessary to set it all up. To be forewarned, (1) and (3) are “objective” reports on what I’ve seen. (2) is more of a rant by a jaded guy.
To preface all of this, I think I am a relatively quiet guy in PC meetings. Part of this has been just my way of listening, learning and trying to figure out what’s happening — the other part is that I think of myself as generally quiet anyway.
Setting. The PC members sit around a table, usually each person has a laptop out. At the front of the room, the chair controls (usually) two different projected screens: one of the screens is for the information about the current paper (title, authors, maybe reviewers), and another screen for the papers for which decisions have been made and not.
Papers are stack ranked. The list of paper submissions is typically stack ranked based on weighted score from highest down to lowest. Paper submissions with consistently great scores are binned in the “automatic accept” pile; paper submissions with consistently horrible scores are binned in the “automatic reject” pile. The primary function of the PC meeting is to work through the bulk of the paper submissions. How a PC works through this bulk depends on the chair; a common strategy is as follows: work from top down until it starts to get hard (i.e. it will be a bunch of accepts, until we are not sure), then, start working from the bottom until it gets hard (i.e. there will be a bunch or rejects, until we are not sure). Then how it goes depends a lot on the chair. I’ve seen them alternate between high and low, and I’ve also seen just starting from the top end (of the submissions that don’t yet have a decision) again and working downwards.
Conflicts. When a paper submission comes up for discussion, the chair will ask people with conflicts to leave. This usually means that if you are one of the authors, or have worked with any of the authors, or have worked at the organization, you leave the room and wait until the discussion is over before returning. The rationale here is so that the identity of the AC that handled the submission (and the reviewers) stays confidential.
Presentation and discussion. Once the conflicts leave, the presentation begins. The AC that handled the paper normally gives a short blurb on the paper: what it was about, what they did, what the contribution is to the field. They also quickly summarize the major points raised by the reviewers (both positive and negative). The committee is then free to discuss whatever they would like about the paper. Often, the committee asks questions of the AC–perhaps what the method was, whether the problem issues raised by the reviewers can be addressed in small revisions, and so forth. The questions can take a long time, but they shed a lot of light into what the committee is looking for in an accepted paper. The chair can sometimes play a role here in terms of helping the committee to decide how stringent they are going to be (remember: no paper is perfect).
Decisions. A decision is then made by the remaining members of the committee. These can include any of the following: accept, accept as short/poster, shepherd, revisit (or “no decision yet”), reject. Sometimes, when an AC doesn’t have enough experience in the domain, s/he will ask for another AC to help read the paper and offer an opinion. In this case, the committee revisits the paper submission at a later time. Other times, the committee will decide that a paper needs to be shepherded–this means that the committee thinks there’s something really great in here, but there are some problems. These problems can be addressed with a closer relationship that an AC who will actually manage the back-and-forth with the authors to make sure the paper has addressed the problems by the end.
“Calibration.” Something that people will say sometimes is that they’re not sure if they feel “calibrated.” This means that they’re not sure if they’re being too picky or too generous. This is often resolved by simply moving on, and listening to more discussions about other papers. The calibration issue is sometimes raised in the discussion, and if so, the paper is often left to be revisited at a later time.
“Auto-discuss.” I have also seen instance where if there was a lot of variation in reviewer scores, that a paper comes up for discussion. For instance, if a paper were scored (on a 5 point scale) with a 2, 3.5, and a 4, then this would likely come up for discussion, even though on balance, the average is above (say) 3.3. It is the process’s way of giving time to the divergent reviewer, who may be seeing something that the rest of us haven’t.
This is where I’m going to sound cranky and jaded. In writing about it, I don’t mean to distance myself from this process. It is easy to get swept up in the PC mission, and just kind of “become” part of what is happening… that said, being aware that it is happening is sometimes a useful enough guard.
Fatal flaws. Everyone has slightly different things that they like to see in papers. Sometimes, it’s perfect stats and methods; for others, it’s that the incremental contribution needs to be a certain size; for other people, it’s that the authors have been too bold in their claims; for still others, it’s that an evaluation group size of n=4 is not big enough, and so forth. I have my thoughts on each of these kinds of problems, but I think there’s something to be said for ensuring that we don’t “throw the baby out with the bathwater.” That is–just because there a little bit of stink, should we throw away the entire paper? Or should we give it a chance?
Group Think. There are a lot of existing relationships between people on a PC (and many of them budding/growing). Assuming a mostly cordial environment, it is easy to just “go with the flow”–that is, just to agree with what everyone else is saying. For me, it has sometimes felt like we are “piling on” papers that may not really be that bad, but just have a few (tiny) issues with them. While it may feel discourteous to disagree with what one of your esteemed colleagues/heroes think, I think it is worse to simply be complicit. I have often felt relieved when someone expressed concern about an opinion that I also shared, but felt too shy to comment on.
Who are the reviewers? A question that sometimes gets asked in the discussion is, “Who are these reviewers?” The AC that handled the paper now needs to justify the choice of reviewers — what is their expertise, how does it relate to the paper at hand, etc. I always feel a little put off by this question, but it is sometimes fair… sometimes, a paper has a weird rating score because an expert rated him/herself as a novice, or vice versa.
Performance. One thing that bothers me a bit about PC meetings is that it sometimes feels a little bit like theatre… the louder someone is, the more likely they are to get their way. I don’t think anyone actually feels this way, but I have seen it happen. I think it is important to also be self-critical (both of oneself and the group): what are the objective issues at hand here–what are the strengths, and what are the weaknesses.
2AC, 3AC, 4AC, cAC… When the primary AC asks for help, I always die a little inside. Usually, this happens when the primary AC is on the fence about a paper (and/or the reviewers weren’t decisive enough), and so the AC asks for help from someone on the committee. The next person that comes in, 2AC (or 3AC, 4AC if the indecisiveness continues down the chain), has basically single-handed veto power. 2AC wields a ridiculous amount of power, and basically it is his/her decision that will sink or swim the paper. I think this is ridiculous, because 2AC (or 3AC, etc.) have spent the least amount of time with the paper (i.e. they’re reading it during the PC meeting). And, because the general tone of a PC meeting is generally negative anyway, papers often get turfed.
Optics. As a PC member, I think it is important that my meta-review score reflects those of the external reviewers I have recruited. To me, this is important because by recruiting them, I am essentially saying that they are experts, and I value their opinion. I have seen instances where a review panel will give it 4,4,4, while the meta-reviewer gives it a 2. This looks bad for the conference (i.e. “optics”). If the reviewers really have overlooked something, then yes, it is important to point it out, but ideally, this has been raised and addressed through the discussion phase with the reviewers.
Champions. To make it out of this PC meeting, sometimes a paper has a champion. This person believes in this paper, and will push through and help it to make it out of the meeting with an acceptance. The motivation for an AC to become a champion (to me) is usually good–i.e. the AC believes it is a good idea/good piece of work, but it has a subtle issue that needs to be addressed. To me, the champion is therefore the best person to shepherd the paper. What I wonder about, in reflection, is whether the champion is actually being altruistic, or whether they have designs on shifting the paradigms in a field. Maybe it doesn’t matter.
With all this in mind, I’ve been quite careful in my preparation for PC meetings. While it is not exactly a performance, I prepare for it like I’m doing an exam. ;-) Here are some of the things that I do.
Pre-PC Meta Reviews. In preparing my pre-PC meta review, I go through every one of the reviews, marking pros and cons that each of the reviewers are positing for the papers. In my meta-review, I summarise the most important positive points, and the most important negative points about each paper. I also write a short four-five sentence summary of the paper: what is the problem, what is the approach, how do they evaluate the approach, and what is the contribution.
Make a decision. I try to make up my mind on each paper–whether I think it should be in or out. This is weighted on the reviews that my externals have given me, but I want to have my mind made up before I go into the PC meeting. I have been asked to serve on the committee to help steer this ship. This is how I do it, by figuring out where I think the ship (i.e. community as a whole) should go. (albeit only by one paper)
Figure out which papers will be discussed. Anything that looks not horrible (average score >2.5), or with high variance will likely be discussed. I make sure I am ready for each of these.
Presentation. I am generally quite brief. First, I read the one-paragraph summary of the paper (described in the previous bullet). Then, I offer my opinion: I think this paper should be accepted or rejected. Then, I raise the strengths of the paper (two or three), and the weaknesses of the paper (two or three).
Points for discussion. Because the PC meetings tend to be overall fairly negative, I try to make sure to have a lot of strengths ready to say for papers that I think should be in. I also prep before the meeting on papers I am sure will be discussed by re-reading papers, being sure about the arguments, understanding the strengths/weaknesses, how the weaknesses can be addressed, and so forth.
So, that’s it. Have fun at the PC meeting.
Nick Feamster also has some useful thoughts on PC meetings.
http://mobilehci.acm.org/2015/download/ExcellenceInReviewsforHCICommunity.pdf
https://kenhinckley.wordpress.com/2015/10/15/commentary-on-excellence-in-reviews-thoughts-for-the-hci-community/