Order of Review Assignments – New Feature?

I came across an interesting article by Maarten Boudry outlining his Seven Laws of Pessimism. In it, Boudry explains the various forces that lead people to believe a situation is getting worse when it is actually getting better.

One of his seven laws, the Law of Conservation of Outrage, proposes that the total amount of outrage remains constant despite the amount of progress. One passage in particular stood out for me:

When I review a stack of student essays, the ones that were submitted last (which tend to be rushed and therefore poorly written) are on top of the pile. So, the quality of the essays improves as I read through the pile. Without even realizing it, I start raising my bar as I mark, and I have to recalibrate my grades at the end, otherwise I grade the better essays too harshly (or the early ones too leniently).

This made me wonder if ProposalSpace should allow call organizers and/or reviewers to determine the order in which assignments are displayed. Currently, they are always displayed in the order in which the proposals were started. So it’s possible—if not likely—that the quality declines as a reviewer works through their assignments since the later submissions are ostensibly rushed. (This is actually the opposite of Boudry’s situation, where the quality improves, but the underlying effect should be the same. Namely, that reviewers unknowingly adjust the bar as the quality changes.)

I would love to hear feedback from call organizers regarding whether this would be a useful feature and if so, how it should be implemented. I can imagine a few possibilities:

  • Change the default behavior for the entire system so that assignments are always displayed randomly.
  • Add a checkbox in the call settings to allow admins to display assignments randomly.
  • Add a set of options in the call settings to allow admins to select whether to display assignments randomly, by when they were created, or by when they were submitted.
  • Add an option for each reviewer to control how their own assignments are ordered—randomly, by creation date, or by submission date.

Good idea? Bad idea? Worth a try? Let me know what you think!

Trend Watch: AI-generated Papers

It looks like one of the hottest topics—if not the hottest topic—for educational conferences in 2023 is going to be how to handle proposals generated by artificial intelligence (AI) tools, like ChatGPT.

While it is tempting for conference organizers to implement outright bans on AI-generated papers, such bans would not only be hard to enforce—especially as AI-generated text becomes increasingly difficult to detect—they might actually do more harm than good in some cases, like for authors who need to submit papers written in a non-familiar language.

James Vincent over at The Verge has written an excellent piece on the complicated issues facing conference organizers and bans on AI-generated content. (Interestingly, the article focuses on the decision by one of the most prestigious machine-learning conferences to ban all papers generated by machine-learning systems.)

Is your conference ready for AI-generated papers? Share your thoughts below!

UPDATE: So it looks like an industrious computer-science student has already developed software that can detect whether text is generated using ChatGPT. In addition, the company that developed ChatGPT, OpenAI, has said it is not only working on a feature that adds a watermark to the output but also its own tool for detecting ChatGPT-generated text.