Order of Review Assignments – New Feature?

I came across an interesting article by Maarten Boudry outlining his Seven Laws of Pessimism. In it, Boudry explains the various forces that lead people to believe a situation is getting worse when it is actually getting better.

One of his seven laws, the Law of Conservation of Outrage, proposes that the total amount of outrage remains constant despite the amount of progress. One passage in particular stood out for me:

When I review a stack of student essays, the ones that were submitted last (which tend to be rushed and therefore poorly written) are on top of the pile. So, the quality of the essays improves as I read through the pile. Without even realizing it, I start raising my bar as I mark, and I have to recalibrate my grades at the end, otherwise I grade the better essays too harshly (or the early ones too leniently).

This made me wonder if ProposalSpace should allow call organizers and/or reviewers to determine the order in which assignments are displayed. Currently, they are always displayed in the order in which the proposals were started. So it’s possible—if not likely—that the quality declines as a reviewer works through their assignments since the later submissions are ostensibly rushed. (This is actually the opposite of Boudry’s situation, where the quality improves, but the underlying effect should be the same. Namely, that reviewers unknowingly adjust the bar as the quality changes.)

I would love to hear feedback from call organizers regarding whether this would be a useful feature and if so, how it should be implemented. I can imagine a few possibilities:

  • Change the default behavior for the entire system so that assignments are always displayed randomly.
  • Add a checkbox in the call settings to allow admins to display assignments randomly.
  • Add a set of options in the call settings to allow admins to select whether to display assignments randomly, by when they were created, or by when they were submitted.
  • Add an option for each reviewer to control how their own assignments are ordered—randomly, by creation date, or by submission date.

Good idea? Bad idea? Worth a try? Let me know what you think!

Submittable Reviews

One of the Review Module’s most useful features has always been the ability for reviewers to save incomplete or provisional reviews. It’s especially handy for reviewers who want to record their initial assessment of a submission then finalize it later, after they’ve analyzed their other assignments.

Up to now, though, there was no way for reviewers to “lock in” their reviews. Unfortunately this meant call admins and review chairs couldn’t be 100% certain a review was ever final, even after the review deadline had passed.

That’s no longer the case! Reviewers now have the option to submit completed reviews, marking them as final for call admins and review chairs. Continue reading Submittable Reviews

Reviewer Conflicts of Interest

We’ve made a few changes to make it easier for reviewers, review chairs, and call admins to identify conflicts of interest.

Although conflicts are not very common (accounting for about 0.1% of reviews in ProposalSpace), we understand that it’s critical for reviewers to be able to flag them clearly as conflicts and for call admins and review chairs to be able to identify them easily.

Continue reading Reviewer Conflicts of Interest