"Shadow PC" Exercise for NSDI'04

This year, NSDI is running a "Shadow PC" exercise in which students can review NSDI submissions when consent is given by the authors and the reviewing occurs in a supervised setting. The aim of this exercise is to educate students in the art of reviewing by using a more realistic setting than has been previously available -- real submissions and real reviewing software. We believe that doing so will provide a long term benefit to the community as a whole.

The remainder of this page describes the process to be followed by consenting authors and Shadow PC chairs, along with its safeguards. The exercise an experiment, and we will solicit feedback after it is concluded to assess whether it should be run in future years. In the meantime, we welcome your feedback and thank the NSDI PC chairs for their support. We hope that you, as an author, will consent to the use of your paper in this exercise, and you, as faculty or student, will make best use of this opportunity.

Steps for Authors

1. The submission page includes a checkbox that allows you to "opt in" to the Shadow PC exercise at the time you submit and for each paper that you submit. Only papers for which the authors give explicit consent will be made available to Shadow PCs.
2. Shadow PC reviewing will occur in parallel and independent of the main review process on a blinded set of papers. The shadow and main reviews will not be mixed, and the shadow process will play no role in determining the outcome for your paper. The shadow review process will be held to the same standards of confidentiality and accountability as the main review process. In particular, the names of the Shadow PC chairs will be made public but the shadow review process itself will be double-blind, with neither Shadow PC members nor Shadow PC chairs knowing the identities of authors.
3. Shadow PC reviews will be returned to consenting authors as a benefit of participating by December 18, one week after the NSDI notification of acceptance. The exercise is then concluded. We hope that this additional feedback will prove useful to authors.

Steps for Shadow PC Chairs

1. We anticipate that Shadow PC chairs will make use of the exercise via a class or weekly reading group. In a typical setting, students will be assigned one or two submissions to review before class (using a similar Web setup and review form as the main conference). The class will then meet and function as its own PC by discussing the set of reviews and merits of each paper, including whether the submissions should be accepted. Shadow PC Chairs must accept responsibility for the following:
   • Organizing a Shadow PC made up of an accountable group of students.
   • Advising the students on the ethics of the reviewing process and its codes of confidentiality and accountability before the exercise commences.
   • Obtaining reviews from participating students before December 11, when NSDI authors receive notifications, to facilitate their (blind) return to the paper authors.
   • Safeguarding the confidentiality of submissions during and after the exercise.
   • Returning feedback gathered from the Shadow PC to the organizers at the conclusion of the exercise.
2. Any faculty member at a higher educational institution may request that a Shadow PC be set up on their behalf by sending mail to Tom Anderson (tom@cs.washington.edu) before September 22 (one week after the NSDI submission deadline).  Students, please find a faculty sponsor at your institution rather than mail us directly. The mail should describe the intended use of the submissions and indicate that the Shadow PC chair accepts all of the conditions above.
3. We will set up Shadow PC review sites based on the merit of the request and subject to our resource constraints. The reviewing sites will allow the Shadow PC chair to create their own PC and assign review work to the PC. We expect to undertake to:
   • Populate the review sites with a selection of anonymized submissions from consenting authors and NSDI review forms. This will occur shortly after the NSDI submission deadline.
   • Provide Shadow PC chairs with administrative access to the site to allow them to manage their own PC.
   • Return Shadow PC reviews to authors at the conclusion of the exercise. We expect this to occur shortly after the NSDI notifications.

Tom Anderson (tom@cs.washington.edu) and David Wetherall (djw@cs.washington.edu) 
University of Washington