Changes to the Papers Reviewing and Submission Process

We’re making some changes to the papers reviewing and submission process this year, and thought it would be useful to communicate this and the reasons for these changes to the community. There are two main differences to the ways that we’ll be handling things this year – paper reviewing and paper length. We’ll document these changes below, but we feel strongly that these are necessary for sustaining the conference over the long term, ensuring the quality of the research presented, providing an equitable playing field for authors, and helping us – the CHI community – to work more effectively.

Two Reviewers plus AC review

Last year, we hit two key issues in the reviewing process: 1) finding 3 external qualified reviewers (a problem that has been developing over a long time); 2) having many AC’s overburdened with 2AC reviews that are required in a very short time. The following set of changes addresses both these issues:

• Two external reviews
• Each AC (1AC and 2AC) to identify ONE reviewer per paper
• The 2AC writes a full, expert review of each assigned paper, for a total of 3 expert reviews per paper (thus, 2 external, one from committee)
• The 1AC acts as a managing editor for their assigned papers. They will not be providing an independent review, but rather will manage the external reviewers, author a metareview summarizing important elements from the 3 expert reviews, and manage the process for the accept/reject decision
• Where there is a difference of X in std deviation in reviews (where X is to be determined) a 3AC is assigned to provide an additional review. This process will be jointly managed by the 1AC and subcommittee chairs (SCs)
• The SCs will load balance the 3rd AC assignments so that no 3AC has more than a few additional papers to review.

Only 2 external reviews By having only two rather than three external reviews, we reduce the pressure on the reviewer pool by a third.

Both ACs find an external reviewer By having the primary and secondary AC each add ONE reviewer, we mitigate the risk of the 1AC determining the fate of a paper by having more full reviewer control of a paper.

2AC full review This has three benefits: 1) Authors receive a full review from an expert who is serving on the committee, is privy to full discussions about the paper, and this feedback is provided prior to the rebuttal phase; 2) We keep separate the roles of expert reviews and editorial management–that is, the 1AC will not be responsible for both a personal evaluation as well as the interpretation of external reviews (metareview) to make a decision; and 3) We improve the predictability of the workload for 2ACs. This addresses a key issue from last year, where some 2ACs had one or two papers to review and other had 6-8, after the first round of reviews were complete, all due in the span of about a week while they were also managing their own metareviews.

3AC trigger By having the std deviation of scores automatically trigger the involvement of a 3AC, we further mitigate 1AC having control of the fate of the paper.

3AC bullpen By uncoupling the link between the paper and 2AC after the initial reviewer assignment, we can address a key issue from last year: load balancing of 2nd AC reviews. We can now reduce overload on individual ACs, and additional papers can be more evenly distributed. This should allow ACs to manage their own time more effectively, as well as enabling them to focus on improving the review process. This redistribution is a new role for the SC’s, and we’ll be working with them to understand what this entails.

Variable Length Papers (aka “no more notes”)

Over the past five years, a number of SIGCHI venues have been moving from a Full Paper and Note submission model, simply to “variable length” (4 to 10 pages plus references). We are moving to this model for CHI 2018. We strongly believe that papers should be at a length that is suitable for their contribution, and there is some evidence to show that the binary Notes/Papers distinction is not supporting shorter papers; e.g., in 2017 the mean acceptance rate for Papers was 25% while for Notes it was 14.6%. Of course, another advantage of variable length is that CHI authors would not need to distinguish their contribution as “only a note” vs having “a paper” at CHI.

One challenge of variable length papers is planning for presentations at CHI. In prior years, Notes were given roughly half the time for presentations as full Papers. It is plausible that shorter papers may not require the same amount of time as longer papers, but how this should be determined is still under consideration. Session planning involves many variables, not least the number of accepted papers, so as a result, the presentation time allotted for different paper lengths will be determined after the PC meeting, once we know more about the numbers and spread of accepted papers.

We will be tracking how this change to paper length impacts both submissions and acceptance rates. As we shift to “variable length” this year, we will chart statistics on how many people take advantage of a shorter paper length and track how reviewers respond to length (particularly for shorter papers). A core question here is whether such a shift will improve the ratio of shorter papers being accepted. As Papers Chairs, we will work with SC’s to monitor how discussions of papers less than the maximum length are managed in review and at the PC meeting to ensure that papers of different lengths are handled fairly and equitably.