Order of Review Assignments – New Feature?

I came across an interesting article by Maarten Boudry outlining his Seven Laws of Pessimism. In it, Boudry explains the various forces that lead people to believe a situation is getting worse when it is actually getting better.

One of his seven laws, the Law of Conservation of Outrage, proposes that the total amount of outrage remains constant despite the amount of progress. One passage in particular stood out for me:

When I review a stack of student essays, the ones that were submitted last (which tend to be rushed and therefore poorly written) are on top of the pile. So, the quality of the essays improves as I read through the pile. Without even realizing it, I start raising my bar as I mark, and I have to recalibrate my grades at the end, otherwise I grade the better essays too harshly (or the early ones too leniently).

This made me wonder if ProposalSpace should allow call organizers and/or reviewers to determine the order in which assignments are displayed. Currently, they are always displayed in the order in which the proposals were started. So it’s possible—if not likely—that the quality declines as a reviewer works through their assignments since the later submissions are ostensibly rushed. (This is actually the opposite of Boudry’s situation, where the quality improves, but the underlying effect should be the same. Namely, that reviewers unknowingly adjust the bar as the quality changes.)

I would love to hear feedback from call organizers regarding whether this would be a useful feature and if so, how it should be implemented. I can imagine a few possibilities:

  • Change the default behavior for the entire system so that assignments are always displayed randomly.
  • Add a checkbox in the call settings to allow admins to display assignments randomly.
  • Add a set of options in the call settings to allow admins to select whether to display assignments randomly, by when they were created, or by when they were submitted.
  • Add an option for each reviewer to control how their own assignments are ordered—randomly, by creation date, or by submission date.

Good idea? Bad idea? Worth a try? Let me know what you think!

Trend Watch: AI-generated Papers

It looks like one of the hottest topics—if not the hottest topic—for educational conferences in 2023 is going to be how to handle proposals generated by artificial intelligence (AI) tools, like ChatGPT.

While it is tempting for conference organizers to implement outright bans on AI-generated papers, such bans would not only be hard to enforce—especially as AI-generated text becomes increasingly difficult to detect—they might actually do more harm than good in some cases, like for authors who need to submit papers written in a non-familiar language.

James Vincent over at The Verge has written an excellent piece on the complicated issues facing conference organizers and bans on AI-generated content. (Interestingly, the article focuses on the decision by one of the most prestigious machine-learning conferences to ban all papers generated by machine-learning systems.)

Is your conference ready for AI-generated papers? Share your thoughts below!

UPDATE: So it looks like an industrious computer-science student has already developed software that can detect whether text is generated using ChatGPT. In addition, the company that developed ChatGPT, OpenAI, has said it is not only working on a feature that adds a watermark to the output but also its own tool for detecting ChatGPT-generated text.

Tips for Avoiding Malware in Submissions

We’ve all been cautioned for years never to open files from suspicious sources. And even if a file is from a trusted source it’s best not to open it until it’s been scanned with antivirus software. Otherwise you run the risk of installing malware, like a Trojan horse, spyware, or ransomware. Opening a file—or even clicking a link—can be one of the easiest ways for hackers to gain control of your computer.

But what if you’re a call admin or reviewer? It’s difficult to assess the quality of submissions without opening documents or following links. This is the security dilemma inherent to every call for proposals: You have to collect content but the only sure-fire way to avoid harm from that content is never to open it.

Unfortunately there is no bulletproof solution. There are, however, a few best practices call admins and reviewers can observe to help reduce their risk:
Continue reading Tips for Avoiding Malware in Submissions

Bulk File Downloads Are Here!

File-upload fields are one of the most convenient ways to collect information from authors. And in some cases, like for photos, there really is no better option.

But one downside to using file-upload fields has been that in order to collect the files, you previously had to pull up each submission and download the files individually.

Not any more! Now you can download files all at once using the Data-export tool.

Continue reading Bulk File Downloads Are Here!

New Feature: Upload Restrictions

Call admins now have the ability to restrict which types of files are allowed for uploads.

The new setting is in two places: the form builder and the Publishing Module. In both cases it is a safelist (as opposed to a blocklist) meaning authors are only allowed to upload files with extensions you specify. Or put another way, authors are prohibited from uploading files not in the list.

Continue reading New Feature: Upload Restrictions

PubMod Updates

The Publishing Module has a number of exciting improvements in place:

  • The module now uses the domain mtgsked.com instead of the main proposalspace.com domain.
  • Virtually all of the ProposalSpace-specific branding is gone, making the meeting-specific branding stand out more.
  • The meeting name, location, and dates can now be customized in the module’s settings.
  • Pages are responsive (i.e. they display nicely on mobile devices).

Continue reading PubMod Updates

New Feature: Admin Session Copies

Calls using the Advanced Scheduling Module now have a new option when creating admin sessions:

Previously all admin sessions had to be started from scratch. This posed a bit of a challenge, however, if someone wanted to combine multiple submissions into a single session (for a panel discussion, for example).

Now admins have the option of either starting an admin session from scratch or making a copy of a submission or another admin session. Hopefully this helps to minimize the need to copy and paste most, if not all, of the content into the new session.