Paper length in the reformulated Papers track

Headlines

1. There are no Notes at CHI 2018. Papers with a length proportional to their contribution were invited in the CfP.
2. There are no Note-length papers in the programme this year – the shortest papers have five pages of content.
3. Only 7% of accepted submissions have 7 or fewer content pages.
4. Papers with 10 pages of content make-up 81% of all accepted submissions.
5. Looking at rejected submissions, it looks like longer submissions are more likely to be accepted.
6. Length somewhat predicts scores. Scores predict acceptance. Length somewhat predicts acceptance.

The Papers track at CHI

CHI Notes, short submissions with a four-page limit, were introduced at CHI 2006. In a twist of fate, CHI 2006 took place in Montréal and with CHI 2018, which is also in Montréal, CHI Notes are no longer part of the programme. The last 12 CHIs have had Papers and Notes side by side from submission through to presentation. Papers had a 10 page limit. Notes had a 4 page limit. (Latterly, these limits have excluded references.) The reviewing process for Notes and Papers was the same. They were published in the same format. Notes, however, often fared badly in terms of relative acceptance rates. Table 1 shows the relative acceptance rates for Notes and Papers at past CHIs. It has been more challenging to get notes accepted at CHI.

Table 1. Historical acceptance rates of Papers and Notes

Furthermore, the rates of Paper and Note submissions have diverged over recent CHIs. The number of submissions of Papers has increased substantially. Notes, already submitted in fewer numbers have not kept up.

Finally, Notes and Papers were given different lengths of talk slots at previous CHI conferences. Having to accommodate two different durations of talks complicated the planning process for the sessions at the conference.

With all of this in mind, the decision was taken to drop Notes from the conference, and instead accept papers of any length up to the limit of ten pages to the Papers track.

Paper lengths – accepted submissions

Have Notes ‘lived on’ in the new Papers track? Are we still getting those kind of short papers? First we will consider accepted papers. When authors of accepted submissions were preparing their final submissions, we asked them to record how many pages of content were in their manuscript. This allows us to augment our data from the number of pages in the submitted PDF, which also includes references.

The mean number of content pages was 9.6 (SD=1.14). The mean number of pages, including references, was 12.1 (SD=1.8). On average, papers came with 2.6 pages of references (SD=1.1). The shortest submissions were 4.5 pages (2). So many of the submissions (512, 77%) report 10 pages of content that a histogram is not informative. These data are tabularized instead, rounded to the nearest whole page (e.g., 4.5 pages and 5 pages are both rounded to five whole pages).

Table 2. Author indicated content length of accepted submissions

These data seem show that there are no Note-length papers being presented at CHI this year. Shorter papers (5-7 pages of content) account for only 7% of all of the papers being presented at the conference. Does this reflect reviewer’s historical preference for longer papers, or has the loss of the Note format encouraged people to submit generally longer papers?

Paper lengths – rejected submissions

Is there a preference for longer papers, or do acceptances simply reflect the underlying distribution of submissions? We do not have the number of content pages for rejected papers, because this information was not solicited from authors. However, from the accepted submissions we know that, on average, 79% of the total length of accepted submissions is made up of content. We can use this to estimate the number of content pages in the rejected submissions (NB – rejects only, not withdrawn, quick reject papers etc) made to the papers track from the length of the PDFs submitted. This yields Table 3.

Table 3. Estimated content length of rejected submissions

Recall that these are only estimates. Nevertheless, these estimates seem to suggest a much more even distribution of paper content lengths amongst the rejected submissions. The accepted papers seem to be disproportionately 10 pages in length, compared to rejected papers.

Relationship between score and length

Are longer papers more likely to be accepted, then? We looked at the relationship between the average score a paper received and its total (i.e., including references) length.

Figure 1. Relationship between mean submission score and submission length in pages

This can be inferred from what has already been reported, but higher scores imply acceptance and longer submissions imply higher scores.

It seems clear that CHI reviewers want to see a CHI-sized contribution to get them tending toward accepting submissions. Whether this preference reflects a preference for longer papers or a preference for the more substantial contribution that longer papers should be able to make is not something that we can assess using the data at our disposal, but we might speculate.

It might be that the kind of disciplines that CHI encompasses generally don’t produce outputs that are well suited to low page limits (consider, e.g., work reporting the results of an interview study). It might also be that experienced researchers perceive full-length papers to have more cachet (e.g., in the context of tenure or national research assessments) and so focus their  efforts on full-length papers. It might also be the case that shorter submissions represent work that has been quickly –and therefore not fully– developed. These submissions might be of lower quality. All of these factors might play a role in the relationship between length and acceptance, amongst many others. We can’t be sure of the answer, but perhaps all of these things are worth considering when you come to submit your CHI 2019 papers in September.

Anna Cox and Mark Perry
Technical Programme Chairs, ACM CHI 2018

Sandy Gould
Analytics Chair, ACM CHI 2018