Home » Uncategorized

Category Archives: Uncategorized

The first part of the review process

This is a brief ‘quick and dirty’ analysis of review data after all reviews were returned. We are aiming to produce more comprehensive analyses of the data in the future, but we wanted to share this information during the rebuttal period in case it is useful to authors in planning their next course of action. We’ve taken every effort to ensure these data are accurate, but given the time pressures, results of the full analysis may vary slightly from these data.

……The TPC team

Headlines

  1. The number of papers submitted has increased by 8% on last year.
  2. Average review scores are very similar to those from CHI2016 (we don’t have the analysis for 2017). There’s no evidence that reviewers or ACs are “more grumpy” this year.
  3. Rebuttals have been shown to increase average scores (2016 data). We expect the same this year and provide links to useful articles about how to write a rebuttal.
  4. The new process for reviews has reduced the burden on the CHI community by approximately 2100 reviews.

Number of papers submitted to CHI2018

Back in September, 3029 abstracts (and associated metadata) were submitted. By the final cut-off date, 86% of these turned into complete submissions resulting in 2592 papers. This represents a rise of 8% (172 papers) on the previous year.

Number of reviews

Of these 2592:

  • 8 were withdrawn
  • 38 were ‘Quick Rejected’
  • 7 were ‘Desk Rejected’
  • 2 were rejected for other reasons

Quick Rejected = papers were read and they were missing something critical that would make replication, analysis or validation of claims impossible.
Desk Rejected = out of scope for chi or some obvious error like over page limit

The 2537 papers remaining received 5105 external reviews. 41 papers received three external reviews. One received four.

Including papers that have already been withdrawn or rejected, reviews were written by 2651 reviewers. The majority (56%) of reviewers completed only one review. One reviewer completed thirteen.

Number of reviews per reviewer Number of reviewers Yield
1 1484 1484
2 550 1100
3 299 897
4 163 656
5 75 375
6 43 258
7 16 112
8 20 160
9 12 108
10 7 70
11 3 33
12 2 24
13 1 13

In addition, 314 ACs (including a few SCs) were assigned to a median of 16 papers (on average, 8 as 1AC, 8 as 2AC). ACs wrote full reviews for 8 papers (as 2AC) and meta-reviews for 8 papers (as 1AC).

Review scores

The mean of the mean scores given to papers for the 2018 conference was 2.56 (SD=0.74). (Yeah, we know we took means of ordinal data here, but luckily there is no R2 of this blog post!). In total, 1356 papers (53%) received an average score between 2.0 and 2.9 (inclusive). Only 129 papers have a mean score of 4.0 or greater. There is one paper with an average of 5.0. The CHI 2016 conference (the last year we have data and detailed analysis for) had an unweighted average of all first round reviewer scores of 2.63 (SD=0.73) and the median was 2.625.

The mean of mean expertise (2018) ratings was 3.22 (SD=0.34) on a 4-pt scale. This shows that reviewers have excellent expertise ratings.

Distribution of mean review scores showing a skewed normal distribution with a peak of 2.5.

This year ACs were instructed to avoid giving papers a score of 3.0. Instead, they were encouraged to give a score of 2.5 or 3.5 in order to give a clear indication to authors as to whether the AC felt they would be able to argue for the paper, in its current state, before the rebuttal. The intent here was that ACs should not be sitting on the fence and avoiding forming a view on the paper. This seems to have been somewhat successful.

Left: Distribution of 1AC scores showing very few 3s. Right: Distribution of 2AC scores showing more 3s and a peak of 2-2.5

Only 33 1AC scores of 3.0 were given (~1%). 2ACs were much more likely to give scores of 3.0 (195, ~8%) when writing their reviews. There are currently 621 papers (~24%) with 1AC score 3+ and 1971 papers with 1AC score <3. We expect these scores to move, as rebuttals are reviewed and discussed.

ACs seem to have tracked their reviewers closely in most, but not all, instances. On average, the mean of 1AC and 2AC scores is ~0.09 less than the mean of the R1 and R2 scores.

Distribution of AC scores relative to the average of the external reviews. Shows a normal distribution centred around 0 with the vast majority being within -1 and 1.

Writing Rebuttals

We are now in the rebuttal phase. Authors can sometimes wonder whether it’s worth their time to write a rebuttal if their paper has received low scores from reviewers. At this stage, of papers not already rejected, 15.6% of papers have a mean score equal to or greater than 3.5.

  • 396 of papers with mean score ≥ 3.5
  • 2141 of papers with mean score < 3.5

While we all understand that papers with higher scores are more likely to be accepted, there is a reason we just don’t auto-accept based on scores, but actually give authors the opportunity to write rebuttals and then discuss the papers at a PC meeting. This important part of the process provides an opportunity for authors to respond to some of the reviewers comments and for the committee to select papers.

Two years ago there was an analysis of whether rebuttals change reviewer scores. (Tl:dr They did) We can therefore expect the number of papers with mean scores above 3.0 to increase when we’ve received the rebuttals and reviewers and ACs have responded to them.

We know that CHI is a highly selective conference. Historic acceptance rates demonstrate that somewhere around 24% of papers are ultimately accepted:

  • 23.6% (2012-16) Papers+Notes acceptance rate
  • 24.5% 2016 Papers+Notes acceptance rate
  • 25.0% 2017 Papers+Notes acceptance rate

Remember, this year there are no more notes so it is also interesting to look at the historic acceptance rates JUST FOR PAPERS

  • 25.1% (2012-16) average Papers only acceptance rate
  • 27.3% 2016 Papers only acceptance rate

So even if your paper has a mean score and a 1AC score below 3.0 there’s every reason to write your rebuttal so that some of the 2.5 1AC scores can move up to 3.5!

A number of members of our community have provided views on how to write a rebuttal:

Anna Cox and Mark Perry
Technical Programme Chairs, ACM CHI 2018

Sandy Gould
Analytics Chair, ACM CHI 2018

Update your reviewing preferences

The ACM CHI Conference on Human Factors in Computing Systems is the premier international conference of Human-Computer Interaction. If you submitted yesterday, thank you for your patience with PCS. We received a record number of submissions. These now need to be peer reviewed by subject experts – and this is why we are reaching out to you. Without the help of thousands of reviewers, we will be unable to ensure that the standard of reviews, and the consequent recognition that research at the event receives, remains as high as it is.

Please take a moment now to update your reviewing preferences and expertise on PCS:

  1. Please visit https://precisionconference.com/~sigchi/
  2. Enter your login credentials
  3. Click on “volunteer center” in the top menu.
  4. Complete the steps as indicated if you are a ‘first time’ reviewer
  5. Or: update the sections marked with “Please update this”

Reviewer Advice:

The selections that you make will support how you will be selected for reviewing papers, so it is important that the content here matches your expertise. Please be realistic, and consider what it is that you are an expert in or are knowledgeable about, but also what you might not have sufficient expertise about to peer review an article in. We’re asking you to volunteer to review a reasonable number of papers – for every paper submitted, three to five (sometime more) people will need to review it, and for the conference to work at all, we need a lot of great reviewers (you!) to volunteer their time for multiple papers. A good rule of thumb would be to volunteer to review at least 3 submissions for every submission you make.

If you haven’t reviewed for ACM CHI before, there are many valuable reasons to do so – not least because you get to see how expert reviewers go about assessing papers for when you submit your own papers to the conference. We strongly encourage new reviewers from industry and research students to take part in reviewing; the committee members will balance topic expertise and prior experience to ensure that less experienced researchers will work alongside at least two other highly experienced researchers as reviewers. For those who do have experience of submitting and reviewing for CHI, we will be working with the original version of the submission system, PCS 1, so please ensure that you access the correct site via https://precisionconference.com/~sigchi/.

There are a number of venues at CHI, and while the ‘Papers’ track is the most prominent, other tracks show important and interesting work. If you haven’t reviewed for other venues such as Late Breaking Work,before, please sign up to review for them on PCS. Some of these tracks are not yet open for volunteering, but will become available very soon. You can find all the CHI 2018 tracks and their deadlines on the CHI 2018 website: https://chi2018.acm.org/authors/. We would be grateful if you would sign up for as many of these as you are able to – reviewing deadlines for these are usually different to the Papers track, so you won’t be overwhelmed with too much reviewing all at the same time.

We really appreciate your enthusiasm and effort in creating a great conference for 2018 – many thanks in advance!

Anna Cox and Mark Perry
Technical Programme Chairs, ACM CHI 2018

Extension to submissions – PCS downtime

As you are aware, PCS – the Precision Conference System that CHI papers are submitted to – has been broken today. There have been several hours where it was been either inaccessible or impossible to submit papers for technical reasons. We are also concerned that the site may be unstable in the next few hours. And also maybe later too. We have therefore decided to extend the deadline to
4pm/1600 Pacific Time (San Francisco, Vancouver),
7pm/1900 Eastern Time (New York, Toronto ),
midnight BST (London),
7am/0700 China Standard Time/Singapore Time (Beijing, Singapore),
8am/0800 Japan Standard Time (Tokyo),
9am/0900 Australian Eastern Time (Sydney).

Please don’t submit at the last possible moment if you can avoid this.

Big shout out to Australia/New Zealand/Other times zones where it is bedtime: #nopaperleftbehind – you can submit tomorrow. Sleep well.
Lots of love TPCs & Paper Chairs

Problems with PCS for Papers submission

We are aware of issues with the submission system (PCS) for submissions of materials for Papers. We are actively working on resolving this issue. We will post regular updates on this page, as well as on Twitter, Facebook, Google+, and across all other social media channels.

Update September 19 2017 at 7am PDT, 2pm GMT, 3pm BST
We are extending the submission deadlines for Papers. Please see this blog post about updated times.

Update September 19 2017 at 5am PDT, 12pm (noon) GMT, 1pm BST
PCS is back online. We continue to monitor the situation.
If you are still experiencing issues, please restart your browser or try a different browser.

Update September 19 2017 at 4am PDT, 11am GMT, 12pm (noon) BST,
We are still working on identifying the root cause of this issue.
Next update at 6am PDT, 1pm GMT, 2pm BST, or earlier if we have any updates.

Update September 19 2017 at 9am GMT, 10am BST, 2am PDT
We are aware that PCS is currently unavailable. We are working to find the cause.
Next update at 4am PDT, 11am GMT, 12pm (noon) BST. We will post hourly updates from 9am PDT, 4pm GMT, 5pm BST

Extension to deadline for CHI2018 Metadata

We are trying to be responsive to the emerging situation on the ground for those affected by the hurricane, and of the huge difficulties and efforts that those affected are putting in in order to try to submit today.

As you’re aware from the previous posts, we have very little leeway to move any of the deadlines. For more information about why this is so please see https://chi2018.acm.org/papers-review-process/

However, we want to extend the opportunity for as many people as possible to be able to submit their papers.  We are therefore moving today’s deadline to the 14th giving an extra couple of days. This is the one area that we really have any real scope to adjust. So, we now have updated this:

  • Submission deadline: 12pm (noon) PDT / 3pm EDT September 14, 2017. Title, abstract, authors, subcommittee choices, and other metadata.

Unfortunately, we have no ability to move the final deadline for submissions. This will remain unchanged:

  • Materials upload deadline: 12pm (noon) PDT / 3pm EDT September 19, 2017.

 

Keep safe!

Ed Cutrell, Anind Dey & m.c. schraefel, CHI 2018 Papers Chairs
Anna Cox & Mark Perry, CHI 2018 Technical Program Chairs

Special note to the CHI community in response to the disruptions caused by Harvey and Irma

For all our friends and colleagues preparing for, and recovering from these catastrophes, our hearts and minds are with you. Watching the news reports and hearing many of the stories coming out of the Caribbean and southern U.S. is truly gut-wrenching. In the past few days, we have received several inquiries about the possibility of a deadline extension for CHI 2018 Papers. Obviously, for those affected by these storms, the preparation of CHI submissions has taken a backseat to much more immediate needs, and we wanted to respond more generally for the whole community.

While we are deeply sympathetic to the disruption these storms have caused, we cannot move the deadline. The CHI papers process is a massive, complex, and somewhat fragile thing—the timing tolerance in going from submissions to reviews to the conference in April is literally on the order of a day or two, and any delays would jeopardize the technical program in Montréal. We have checked to see if there are any creative things we can do, but we are so close to the deadline that there just isn’t any way for us to change the process at this point. We are deeply sorry—we know how important CHI is for students, scholars and professionals in our community and these tragedies hit the whole community.

We hope that all those affected are safe and managing the aftermath with their loved ones. We stand with you and hope to see you all in Montréal next spring.

Ed Cutrell, Anind Dey & m.c. schraefel, CHI 2018 Papers Chairs
Anna Cox & Mark Perry, CHI 2018 Technical Program Chairs

Welcoming Submissions to the Panels & Fireside Chats Venue

At CHI 2018, the theme will be engage. The technical and social programs at CHI will provide several ways for attendees to engage in discussion. One of these is through the curated Panels & Fireside Chats Venue. We are inviting stimulating panel submissions that inspire new ideas, are innovative in approach, and likely to engage the conference attendees.

One change this year is that we are welcoming a range of formats for the presentation of panels. They can take the form of a traditional panel of discussants with a moderator, a fireside chat in which an individual gets interviewed by a moderator, or a roundtable in which the moderator(s) pose the questions to the audience for discussion. Our goal is to facilitate discussion – panels that propose a series of mini presentations are not what we are looking for. We would like you to be creative in proposing a format for your panels and welcome new ideas!

The deadline for submissions to the Panels & Fireside Chats venue is January 15, 2017, but we welcome ideas for high-profile plenary session panels and fireside chats at any time. If you have ideas for a high-profile panel, fireside chat, or roundtable discussion that would be of interest to a broad range of CHI attendees, please contact the venue’s co-chairs (panels@chi2018.acm.org) and/or the conference co-chairs (generalchairs@chi2018.acm.org) with your ideas as soon as possible.

Panels & Fireside Chats Chairs
Parmit Chilana, Simon Fraser University, Canada
Julie Kientz, University of Washington, USA

Conference Chairs
Regan Mandryk, University of Saskatchewan, Canada
Mark Hancock, University of Waterloo, Canada

Changes to the Papers Reviewing and Submission Process

We’re making some changes to the papers reviewing and submission process this year, and thought it would be useful to communicate this and the reasons for these changes to the community. There are two main differences to the ways that we’ll be handling things this year – paper reviewing and paper length. We’ll document these changes below, but we feel strongly that these are necessary for sustaining the conference over the long term, ensuring the quality of the research presented, providing an equitable playing field for authors, and helping us – the CHI community – to work more effectively.

Two Reviewers plus AC review

Last year, we hit two key issues in the reviewing process: 1) finding 3 external qualified reviewers (a problem that has been developing over a long time); 2) having many AC’s overburdened with 2AC reviews that are required in a very short time. The following set of changes addresses both these issues:

  • Two external reviews
  • Each AC (1AC and 2AC) to identify ONE reviewer per paper
  • The 2AC writes a full, expert review of each assigned paper, for a total of 3 expert reviews per paper (thus, 2 external, one from committee)
  • The 1AC acts as a managing editor for their assigned papers. They will not be providing an independent review, but rather will manage the external reviewers, author a metareview summarizing important elements from the 3 expert reviews, and manage the process for the accept/reject decision
  • Where there is a difference of X in std deviation in reviews (where X is to be determined) a 3AC is assigned to provide an additional review. This process will be jointly managed by the 1AC and subcommittee chairs (SCs)
  • The SCs will load balance the 3rd AC assignments so that no 3AC has more than a few additional papers to review.

Only 2 external reviews By having only two rather than three external reviews, we reduce the pressure on the reviewer pool by a third.

Both ACs find an external reviewer By having the primary and secondary AC each add ONE reviewer, we mitigate the risk of the 1AC determining the fate of a paper by having more full reviewer control of a paper.

2AC full review This has three benefits: 1) Authors receive a full review from an expert who is serving on the committee, is privy to full discussions about the paper, and this feedback is provided prior to the rebuttal phase; 2) We keep separate the roles of expert reviews and editorial management–that is, the 1AC will not be responsible for both a personal evaluation as well as the interpretation of external reviews (metareview) to make a decision; and 3) We improve the predictability of the workload for 2ACs. This addresses a key issue from last year, where some 2ACs had one or two papers to review and other had 6-8, after the first round of reviews were complete, all due in the span of about a week while they were also managing their own metareviews.

3AC trigger By having the std deviation of scores automatically trigger the involvement of a 3AC, we further mitigate 1AC having control of the fate of the paper.

3AC bullpen By uncoupling the link between the paper and 2AC after the initial reviewer assignment, we can address a key issue from last year: load balancing of 2nd AC reviews. We can now reduce overload on individual ACs, and additional papers can be more evenly distributed. This should allow ACs to manage their own time more effectively, as well as enabling them to focus on improving the review process. This redistribution is a new role for the SC’s, and we’ll be working with them to understand what this entails.

Variable Length Papers (aka “no more notes”)

Over the past five years, a number of SIGCHI venues have been moving from a Full Paper and Note submission model, simply to “variable length” (4 to 10 pages plus references). We are moving to this model for CHI 2018. We strongly believe that papers should be at a length that is suitable for their contribution, and there is some evidence to show that the binary Notes/Papers distinction is not supporting shorter papers; e.g., in 2017 the mean acceptance rate for Papers was 25% while for Notes it was 14.6%. Of course, another advantage of variable length is that CHI authors would not need to distinguish their contribution as “only a note” vs having “a paper” at CHI.

One challenge of variable length papers is planning for presentations at CHI. In prior years, Notes were given roughly half the time for presentations as full Papers. It is plausible that shorter papers may not require the same amount of time as longer papers, but how this should be determined is still under consideration. Session planning involves many variables, not least the number of accepted papers, so as a result, the presentation time allotted for different paper lengths will be determined after the PC meeting, once we know more about the numbers and spread of accepted papers.

We will be tracking how this change to paper length impacts both submissions and acceptance rates. As we shift to “variable length” this year, we will chart statistics on how many people take advantage of a shorter paper length and track how reviewers respond to length (particularly for shorter papers). A core question here is whether such a shift will improve the ratio of shorter papers being accepted. As Papers Chairs, we will work with SC’s to monitor how discussions of papers less than the maximum length are managed in review and at the PC meeting to ensure that papers of different lengths are handled fairly and equitably.

Welcome to the CHI 2018 Blog

At CHI 2018, the theme will be engage. Our focus will be to engage with people, to engage with technology, to engage with newcomers, to engage with world-class research, to engage with your community of designers, researchers, and practitioners… to engage with CHI!

As organizers, our goal is to engage with the larger CHI community throughout the planning process and to be transparent about the decisions that we make. As such, we will be using this space to post articles on how we have made decisions and also to solicit input from all of you!

Comments are open and can be signed or anonymous. They aren’t moderated and we hope to keep it this way, so please be respectful and professional in communicating your opinions, which we look forward to hearing.

Conference Chairs
Regan Mandryk, University of Saskatchewan, Canada
Mark Hancock, University of Waterloo, Canada

Assistants to General Chairs
Max Birk, University of Saskatchewan, Canada
Cayley MacArthur, University of Waterloo, Canada

Technical Program Chairs
Mark Perry, Brunel University London, UK
Anna Cox, University College London, UK

Assistant to the Technical Program Chairs
Frederik Brudy, University College London, UK
Marta Cecchinato, University College London, UK

Scroll Up