ISCA 2009 Reviewer Guidelines

Luiz André Barroso, Program Chair

As an ISCA reviewer you are in a position to make a positive impact on the health and vibrancy of our technical community. Here are some of the most important things to keep in mind as you read a submission and write your feedback:

A reviewer is a mentor

    Giving authors valuable feedback on their research is a key part of your service to our community. That is, it is as important as your role in helping us select the best papers for the conference. Write your review as if you were trying to give a friend-colleague candid, constructive advice about their project.

review is not about how smart you are. We know you are smart; that's why we asked you to be a reviewer. A review is about just two things: helping the program committee make sound decisions about the final program and helping the authors improve their work.

    Think first of the positive aspects of the work, and show the appropriate praise and enthusiasm for them. When you point out shortcomings, follow each of them up with constructive suggestions for improvement. Be critical of the work while being courteous and considerate to the authors. Remind yourself that the final program will be filled with great albeit imperfect articles.

Take confidentiality seriously

    You are being entrusted an unpublished manuscript to review. The efforts around those papers may not have been publicly disclosed yet, and the authors have not given us any permissions to do so on their behalf. You cannot distribute any manuscripts you are reviewing to others. If you feel that a colleague's opinion would be critical to the article, ask the Program Committee Chairman to invite your colleague as an additional reviewer.

Timeliness = Respect

    Authors have worked very hard to produce their submission, and they have made their deadline. Nothing shows more disrespect to their efforts than a late review or a rushed/cryptic one. Our selection process has historically benefited greatly from the authors' opportunity to rebut reviews before a final decision is reached. A late review deprives them of that opportunity. Please plan ahead your time budget for reviews, especially if you are reviewing more than one submission.

There's no single formula for a great paper

    One feature of a great program is balance between papers that make significant contributions to established areas and papers that venture into truly new territory. Both categories of papers are expected to display originality and technical soundness. It is generally easier to conduct rigorous quantitative evaluation methods when working within established areas, so we do expect to see excellent quantitative evaluations in those papers.
 However, if you are excited about an article which carries a bold new idea, you are allowed to recommend acceptance even if the evaluation methodology used has some shortcomings. Authors of those papers are still expected to make their best effort to objectively assess the value of their ideas.

ISCA should be a "Big Tent"

    Computer Science has grown tremendously in breadth of disciplines over the past 40 years, and is continuing to evolve. If we choose to stick to a static and narrow definition of what makes a legitimate ISCA paper, computer architecture is bound to decrease in relevance as a discipline over time. Instead, ISCA should be inclusive of work in areas that can broaden the boundaries of our field, so that it continues to have a great impact on our industry and on society at large.

    Although there will be cases where it will be fair to question whether a paper belongs in ISCA, think twice before using that argument to reject a submission. If we think our reviewers will have the expertise to appreciate the technical merits of a paper and our attendees will enjoy learning about it, we should consider accepting it.

Further reading:

If you haven't reviewed research papers in a while, I'd recommend reading Alan Jay Smith's The Task of the Referee (IEEE Computer, 4/90). Mark Hill has also written a great set of guidelines for reviewers of ISCA'05. Thanks to Mark, Kathryn McKinley, Steve Keckler and Brad Calder for letting me steal material from their guidelines.