I came across an interesting article by Maarten Boudry outlining his Seven Laws of Pessimism. In it, Boudry explains the various forces that lead people to believe a situation is getting worse when it is actually getting better.

One of his seven laws, the Law of Conservation of Outrage, proposes that the total amount of outrage remains constant despite the amount of progress. One passage in particular stood out for me:

When I review a stack of student essays, the ones that were submitted last (which tend to be rushed and therefore poorly written) are on top of the pile. So, the quality of the essays improves as I read through the pile. Without even realizing it, I start raising my bar as I mark, and I have to recalibrate my grades at the end, otherwise I grade the better essays too harshly (or the early ones too leniently).

This made me wonder if ProposalSpace should allow call organizers and/or reviewers to determine the order in which assignments are displayed. Currently, they are always displayed in the order in which the proposals were started. So it’s possible—if not likely—that the quality declines as a reviewer works through their assignments since the later submissions are ostensibly rushed. (This is actually the opposite of Boudry’s situation, where the quality improves, but the underlying effect should be the same. Namely, that reviewers unknowingly adjust the bar as the quality changes.)

I would love to hear feedback from call organizers regarding whether this would be a useful feature and if so, how it should be implemented. I can imagine a few possibilities:

• Change the default behavior for the entire system so that assignments are always displayed randomly.
• Add a checkbox in the call settings to allow admins to display assignments randomly.
• Add a set of options in the call settings to allow admins to select whether to display assignments randomly, by when they were created, or by when they were submitted.
• Add an option for each reviewer to control how their own assignments are ordered—randomly, by creation date, or by submission date.

Good idea? Bad idea? Worth a try? Let me know what you think!