Submittable Reviews

Update (27 Feb 2021): The changes outlined below are now in place!

One of the Review Module’s most useful features has always been the ability for reviewers to save incomplete or provisional reviews. It’s especially handy for reviewers who want to record their initial assessment of a submission then finalize it later, after they’ve analyzed their other assignments.

Up to now, though, there was no way for reviewers to “lock in” their reviews. Unfortunately this meant call admins and review chairs couldn’t be 100% certain a review was ever final, even after the review deadline had passed.

That’s about to change! Soon reviewers will have the option to submit completed reviews, marking them as final for call admins and review chairs. Reviewers will still be able to save their work and return to make changes later, but only until they submit a review. Once that happens the review will be locked, allowing the reviewer to view it but not to make further changes to it (unless a call admin unlocks it).

screenshots showing Tracker dataOn the admin side, values for reviewer progress and average score will no longer be calculated based on all reviews. Instead, the default will be to calculate those values using only submitted reviews. The Tracker will, however, have a handy toggle that will allow admins and chairs to include unsubmitted reviews. (See screenshots to the right showing information for the same two submissions. Each submission has been assigned to two reviewers. The reviewers have completed their reviews but have not submitted them. The top image shows the default view, which is based solely on submitted reviews. The bottom image shows the same view but with unsubmitted reviews included.)

Other new features for admins and chairs include the ability to submit reviews on behalf of reviewers and to unlock/return submitted reviews to make them editable. The data-export feature will also include the review status of each review in the output file.

Reviewers will see changes in two places: the main review form and the page that displays their review assignments.

On the main review form, reviewers will no longer have to click a button to save their work. Changes will instead be saved automatically. Because of that, the Save button will no longer be needed and will be replaced with a Submit button, which will be inactive until the review is eligible for submission (i.e. the reviewer has answered all of the required questions on the review form).

review-assignment page detailOver on the review-assignments page, reviewers will see two new columns in the assignments table: Review Status and Actions. (See the screenshot to the right.)

The Review Status column will display one of the following four values for each review assignment:

  • Not started = None of the review questions have been answered.
  • Incomplete = Some review questions have been answered but at least one required question is missing an answer.
  • Complete = All required review questions have been answered. The review is eligible for submission.
  • Submitted = The review has been submitted and may not be changed.

(BTW, these are the same values used in the data-export file.)

The Actions column will always display a button to edit the review (or to view it if it’s been submitted). If the review is complete but has not yet been submitted, the column will also display a button to submit the review.

I really hope these changes will improve the user experience for call admins and review chairs by making it possible for them to know for certain when reviewers are done with their work while not taking away any of the features and functionality already enjoyed by reviewers. As always, if you have any feedback about the changes or want to pass along ideas for other improvements, please let me know!

Published by

Dan

Dan is the founder and CEO of ProposalSpace. The buck stops with him.