Reviewing Submissions

Introduction

Thank you for volunteering to review for CHI.  If you aren’t familiar with reviewing for CHI, or haven’t reviewed for the last couple of years, please read this thoroughly; our approach has changed considerably. This short guide explains what you need to do in order to assist the Co-chairs in selecting a top-quality program.  It also addresses your equally important task of providing helpful comments to feed back to authors.

This page describes how to summarize a submission's contribution, what is needed in the review (a review is not just a vote for or against acceptance, it is input to a discussion amongst the 30+ chairs and associate chairs), and provides examples of both suitable and unsuitable ways of writing CHI reviews. It also emphasizes the importance of being complete, constructive and polite, and of turning your reviews in on time. If you cannot complete a thoughtful review by the review deadline, please contact the co-chairs as soon as possible.

Summarizing the Contribution

When you first log on you will see a list of the submissions assigned to you.  Please follow the instructions on filling in the review form, and you will see that item 4 asks you to state, in two or three sentences, what contribution the submission aims to make to the field of Human-Computer Interaction.  The Call for Participation advises authors that their submissions will be reviewed in terms of this contribution, so we ask you to take some care to identify it concisely and accurately.  Give some thought to the type of contribution, which may be:

• a design briefing
• a development methodology or tools
• an interaction technique
• an interactive system
• a reflective analysis
• results from fieldwork and ethnography, e.g., findings, guidelines, etc.
• results from laboratory studies, e.g., findings, techniques, methods, etc.
• a theory or model.

Writing the Review: What is Needed

The key point you should keep in mind is that your review is not just a vote for whether a submission will be accepted: it is input to a discussion amongst the committe members. You are assisting the committee by providing an argument for or against acceptance. In some cases there will be wide divergence amongst reviewers’ numerical ratings of the submission. In these cases your argument, if clearly and succinctly stated, can ultimately have more influence than the rating alone. To do so your review need not be of great length. But if it provides just a rating without an adequate rationale it will be virtually useless. Please therefore help your co-chairs put together an overall case for acceptance or rejection by attending to:

Reading the submission thoroughly. To write a thorough review you will need to understand in detail what the authors have achieved and how they achieved it. It is often necessary to read a submission more than once, or to spend considerable time focusing on complex arguments. Please set aside sufficient time to read each of your assigned submissions.

Summarizing your main points.  Your co-chairs and fellow committee members will need to understand your argument quickly, and an initial summary can help them a lot.  Use the remainder of the review to expand on the summary’s main points and mention other matters.

Relevant past work.  If you’re aware of past work against which this submission can be judged, please mention it, whether or not the authors have cited it.  You need not comment on the thoroughness of the submission’s citations.

Significance of contribution and benefit.  It’s particularly helpful to your co-chairs if you can assess the contribution objectively, preferably in terms of the past work you’ve identified .  Please don’t just give a short, unexplained answer or avoid this point altogether.

Coverage of all the criteria.  Your co-chairs will pull together all the reviewers’ comments on relevant past work, contribution’s significance, benefit, validity and originality.  You’re not obliged to cover every one of these, but please don’t devote the whole review to just one issue.  Equally, don’t cover all the criteria in a cursory manner without backing up each claim.

Reviewing ‘as is’.  The tight deadlines of CHI rule out any possibility of checking whether authors make changes demanded by the committee.  Therefore the decision on whether or not to accept must be made on the basis of what the authors submit for review.  Please do recommend improvements, but don’t require acceptance to hinge on making these changes.

Polite, temperate language.  However much you may dislike the submission, try to say so in a manner helpful to the authors and informative to the co-chairs.  Please don’t vent your anger or shower abuse.

Meeting the Deadline

The reviewing and selection of CHI submissions takes place against very tight deadlines.  After all the reviews are submitted, the co-chairs prepare meta-reviews summarizing their recommendations.  If reviews aren’t ready in time, meta-reviews can’t be written, and the committee won’t have sufficient information to make its decisions.  Your help is essential to ensuring these deadlines are met.  In particular, we rely on you to submit your reviews on time. If you cannot complete a thoughtful review by the review deadline, please contact your associate chair as soon as possible.