The gargantuan effort of soliciting, submitting and assessing submissions at CHI is over for the 2018 programme. (The process for 2019 has already started!) In this blog post we look across all of the tracks to look at CHI’s big picture. How many submissions did we get? How many were accepted? For whatever reasons you might want to know these statistics – comparisons across different years, assessment of your own success rates against track averages, comparison to other conferences, or something else – we have tried to be as open as possible here about the conference data for the broad set of submissions and reviews.

Overall submissions

CHI 2018 solicited submissions to 14 tracks. Across all of these tracks we had a total of 3955. The conference is, unsurprisingly, dominated by the Papers track (66% of all submission to the conference). Late Breaking Work is also a significant contributor (16%). The rest of the tracks combined account for 18% of the submissions to the conference.

Table 1: Submissions and acceptance rate by track.

A complete listing of all tracks, sorted by submission rate is provided in Table 1. Note that some of these numbers vary slightly from data you may have seen previously: this is because some submissions may have been removed from the submission system (for example, if they were withdrawn).

The Student Design Competition had the lowest acceptance rate (15.2%). SIGs were the most likely to be accepted (82.6%). Of course, the overall acceptance rate is skewed by the fact that we have so many submissions to the Papers track. The average unweighted acceptance rate (i.e., the mean of the individual track acceptance rates) is 43%.

Thank you everyone who submitted their work to CHI 2018! High quality submissions are the lifeblood of any conference.

Overall external reviews

Of course, all 3955 submissions had to have some kind of review, whether through the Paper track’s full peer-review, or through a slightly more lightweight approach. Here we consider just the external reviewers in the process – we’ll consider committee members in the next section.

Figure 1: Number of reviews contributed by each reviewer in the Papers track. This is a plot of the data from a previous blog post.

The conference defines ‘Refereed’ content as that which is rigorously reviewed by members of the program committee and peer experts. The process includes an opportunity for authors to respond to referees’ critiques. Submitters can expect to receive formal feedback from reviewers, and the program committee may ask authors for specific changes as a condition of publication. Reviewers are also involved in ‘Juried’ content  (Late Breaking Work, Workshops/Symposia, Case Studies, alt.chi, Student Design and Research Competitions). Note that some tracks do not have external reviewers because they are ‘Curated’ (Panels, Courses, Doctoral Consortium, EXPO, Special Interest Group (SIG) meetings, Video Showcase), and while these tracks are highly selective, they do not usually follow a committee-based reviewing process. Fuller details on these processes can be found in Selection Processes on the CHI website.

In total, submissions to the tracks that made use of external reviewers generated 1.93 external reviews per submission. The Papers track attracted the lion’s share of reviews (73.9%), averaging 2.01 external reviews per submission. Outside the Papers and Late Breaking Work tracks, fewer external reviewers (516) were involved in the process (7.5%), and averaging 1.65. reviews per submission. Although 3837 non-committee reviewers are listed here across all tracks, we have not collated the contributions of individual reviewers across tracks, so we cannot yet say how many unique CHI external reviewers we had this year across all tracks.

Table 2 contains information for each track on the number of reviewers for a track and the total number of reviews that they produced. Note once again that these breakdowns are based on final-end-of process data and may vary very slightly from what has been reported previously (e.g., due to reviewers being added, or incomplete reviews being removed from the system).

Table 2: External reviewers and number of reviews by track

Thank you everyone who provided an external review for CHI 2018. People know that when they see work at CHI, it has been rigorously reviewed by experts. Without your reviews we wouldn’t be able to put together a programme of this quality.

Overall committee reviews

Submissions need reviewers. Committee members help to find them. Once reviewers give their opinion, someone has to synthesise them to come to a decision. Committee members do this too.

Between them CHI 2018 committee members wrote 6,321 reviews, meta-reviews or opinions. This is not far off the total number of external reviews – committee members work extremely hard in their roles. Once again, the Papers track generates the vast majority of the work (83.8%) for the committee in terms of reviews or meta-reviews written.

Table 3: Committee members and number of reviews by track (NB – includes only PC members with meta-reviewing responsibility and not, for instance, track chairs).

Table 3 shows the contributions of committee members to the different tracks. Note that, again, these figures may not match up precisely because of changes over time (or, for instance, data not appearing in the correct place, a switching of 1AC and 2AC roles during the process, for instance). Also note that not all tracks have a process that involves committee members writing formal reviews.

The CHI 2018 committee members have worked meticulously, often on a large number of submissions, to make sure that the right decisions are made for the the conference programme. We thank all of them for their efforts.

Summary

As you can see, the CHI process generates huge volumes of work – for those working on committees, and as reviewers, as well as for authors, and while we have tried to minimise the effort for everyone involved, the review process requires vast amounts of time to ensure its sustained quality of outputs and topical relevance. Here we have only quantified the number of submissions and the amount of work that these submissions generate for reviewers and committee members. Of course, not all the tracks at CHI generate work that can be measured using the metrics we have used in this post, nor is this a complete account of thousands of hours of organizational work and planning that also has to take place for all of these parts to come together. We can only run a conference of this scale and quality with the dedicated effort of the researchers and practitioners that offer their time, energy, and expertise. The ACM CHI conference stands at the forefront of its discipline, but its pillars are supported by the community it serves. We thank those who have stepped up to do their bit, and encourage the next generation of researchers emerging to help take on these demanding but rewarding roles. The future of the conference lies in your hands!

Anna Cox and Mark Perry
Technical Programme Chairs, ACM CHI 2018

Sandy Gould
Analytics Chair, ACM CHI 2018