Why you should reconsider appealing conference rejections.

An Appeal Not to Appeal

by Niklas Elmqvist, University of Maryland, College Park

This year, I served as papers co-chair for IEEE InfoVis for the second time (2016 and 2017), which was an excellent (if work-intensive at times) experience. I was honestly humbled to be given the honor and responsibility of this position, and I have thoroughly enjoyed giving back to the community in this way. However, one of the little negative things you have to deal with as papers chair, I have learned, is the angry emails and the appeals that you receive about paper decisions. The simple point that I will try to make in this blog post is this: don’t try to appeal, because it will basically never work (except in very specific circumstances) and is likely to have more negative than positive outcomes.

To be fair, this is only a minor concern for InfoVis, as we received a very small amount of such emails (I can count them on the fingers of one hand). I think a big part of the reason for this is the quality of reviews for InfoVis: while the conference is very selective (with an acceptance rate between 22-24%), the reviewers put a lot of time and thought into their reviews. In other words, while reviews are often strict, they are often fair and appropriate. This is a good characteristic to strive for, because it ultimately results in fewer disgruntled authors, and, in turn, fewer angry emails to the papers co-chairs. But they still happen.

The Case for Appealing

First of all, let me say that I get it. Firing off an angry email in response to what you perceive to be unfair reviews is only human. Many times, the papers chair is the only person named in the notification letter, and so becomes a natural target for the author’s ire. Even sending a more measured appeal after the worst of the anger has melted away is perfectly understandable. While these are less about the passion of the moment and more about righteous and sometimes indignant feelings about being misunderstood or treated unfairly, I absolutely understand the need for some authors to offer some kind of response, to set the record straight.

In fact, this has happened to me. Of course it has; it has happened to everyone. No researcher gets all their work accepted on the first try, no matter how talented they are and how solid the paper is. Peer review is an inexact science. Sometimes you get allocated a reviewer that doesn’t spend enough time reading our paper, misunderstands or misinterprets your findings, or simply has a bad day. Sometimes—perish the thought!—the problem actually lies with your paper. In my case, I once had a paper rejected, it felt like, because of a problem with a color scale choice (and no, it wasn’t a rainbow color map!), another time because the reviewer erroneously claimed that the paper had to be anonymized (it didn’t have to be; this was VIS). I have many other examples; my papers get rejected all the time!

I get it. I still say that you shouldn’t appeal.

The Case against Appealing

Of course, the reason I am advocating that you refrain from sending appeals—angry emails or measured appeals alike—is not that you should spare the feelings of the poor papers chairs. After all, papers chairs tend to have thick skin, or they wouldn’t be selected to serve in this capacity: they are normally relatively senior people who have a keen understanding of valid contributions to a research field, they are in established positions where their job security is in no danger due to disgruntled authors, and they are seasoned enough to be able to separate personal from professional attacks. I’m not saying these people can’t handle acerbic or even angry emails. In fact, I’m pretty sure that they can; the times it has happened to me, I’ve merely smiled to myself and put the email aside.

No, the reason that I advocate that you don’t try to appeal is twofold: first of all, because it is not going to work (because decisions are final, peer review is inexact, and it is not scalable), and second, because it almost always leads to a negative net outcome.

Decisions are (Most Often) Final

You have to understand that papers chairs make decisions while weighing a lot of different factors. For example, for the IEEE InfoVis conference, the papers chairs have to balance the feedback from all of the reviewers, the recommendations of the program committee members, and the guidelines and instructions from both the steering committee and IEEE TVCG, the latter which publishes the special issue of the journal where InfoVis papers appear. Actually coming to these decisions involves at least a month of peer review and discussion among reviewers followed by a weeks-long process of deliberation between the papers chairs, a part of which is negotiating with TVCG about the acceptance rate and how to handle borderline papers. By the time notifications go out, all of these individual decisions have been made and all of the questions have been ironed out. Emergency reviews have been procured for borderline papers. A tentative acceptance rate has been set. TVCG knows the page budget it needs for the InfoVis track. (Incidentally, I wrote an article with my two papers co-chairs last year about the InfoVis reviewing process.)

All of this contributes to making any changes in the paper accept/reject decisions very difficult. Unless there were genuine clerical, procedural, or ethical issues with the review process, InfoVis does not allow appeals, and instituting a formal mechanism would be very difficult given the limited time available in the schedule. Even if the authors bring up valid points, it is very difficult (if not impossible) to make a change after the notifications have gone out. Most likely, any changes (i.e. new papers to accept) would have to go to another IEEE TVCG issue, i.e. not the conference special issue.

It’s the Nature of Peer Review

It is also worth considering the most common objection that authors tend to raise in their appeals: that the reviewers somehow misunderstood their work. If the reviewers only had understood this critical part of the work, the argument goes, the outcome would have been very different. Often these authors request a new review or even a new set of reviewers.

My response to this argument is always the same: unfortunately, if reviewers consistently misunderstood your paper, it is plausible that at least a part of the problem lies with the paper itself. In the example above for my rejected paper, the color scale was admittedly questionable, and even if this fact did not have a significant impact on the visual representation, we could have made better design choices or we could haveĀ communicated better why the color scale had little impact on its utility. Furthermore, even if your paper is perfectly clear and understandable, bad or unfair reviews are part of the game. Everyone gets them.

Also, different people have different perceptions of what constitutes a significant contribution and a significant concern, respectively. This makes it hard to argue objectively that someone is incorrect. Perhaps they just think a weakness they identified that you think is minor actually is a fatal flaw? At the end of the day, peer review often comes down to subjective opinions.

Even if we entertain, for a moment, the possibility that a reviewer is wrong about a point they made in a review, it is impossible to institute a clean room approach to eliminating that factually wrong feedback and performing the review anew. In my experience, reviewers get a feeling for a specific paper from reading it, and the review itself is only the tip of the iceberg representing that feeling. Sometimes the feedback surfaced in those reviews are not truly representative of what the really reviewer feels is wrong about the work. While this is of course unfair, peer review is, as already stated, imperfect. Even if you were able to surgically remove a misconception or misunderstanding from a review, that is not a guarantee that the paper will fare any better.

Look at the Size of that Thing

Finally, consider the scale of what you are asking for. If we go by the most common and reasonable argument that a paper was rejected because the reviewers misunderstood the contribution, consider what would happen if every author of a rejected paper appealed their decision for this reason. Now let us assume that every such appeal would require a new review round, either with the same or with a new set of reviewers. Theoretically, every rejected paper that falls in the borderline area where a single reviewer can swing the balance, and no doubt several other papers as well, would exercise their right, causing a huge amount of new reviewing work and requiring a significant time allotment in the production schedule for the conference.

From the viewpoint of a papers co-chair, this is just not practical. In a way, in submitting to a conference such as IEEE InfoVis, I would argue that you implicitly comply with the understanding that to achieve the short turn-around afforded by the conference, you eschew the possibility to appeal decisions and accept them as final.

If you really do want a way to communicate with your reviewers, you should instead be submitting to a conference where this is explicitly built into the review process—like the rebuttal mechanism at ACM CHI, or the revise and resubmit at ACM CSCW—or to a proper journal. See more on this below.

It’s Bad Karma

Remember, like with anything, that angry emails sent in the heat of the moment in response to perceived unfair or incorrect reviews may negatively affect your reputation. It’s just bad karma. As I said, I personally understand why people get upset, and I won’t hold your factually correct and argumentative appeal against you. I still think it’s a waste of time, for all of the reasons outlined above, but I won’t hold a grudge. That wouldn’t be professional.

But papers chairs are people too, and if you send an angry, rude, and perhaps even insulting email attacking the integrity of the review process, the reviewers, or even my colleagues and I, you can be sure that I will remember that. Again, papers chairs are selected to be above this kind of thing, but of course they can’t help if nasty incidents stick in their memory. It’s human nature. For example, I’m sure a certain journal editor remembers the acerbic email that I sent a while back in response to a long string of rejects. At least I wasn’t rude, but I’m still kicking myself for that.

How to Deal with Unfair Reviews?

Let me conclude with a small discussion of appropriate ways to deal with unfair or incorrect reviews. As part of this, I will draw on some of my previous postings:

  • Multi-round reviews: One approach is to submit to journals (or certain conferences like ACM CSCW) that use a multi-round review process where the authors and the reviewers engage in a dialog on the contributions of the work. Here the feedback mechanism is formalized and part of the process, so it is less contentious. Of course, you should still be careful how you respond to feedback from reviewers; my guide on writing effective revision response letters may come in useful here.
  • Rebuttals: Some conferences, notably the ACM CHI conference, has a formal rebuttal mechanism where the initial reviews are tentative and give the authors an opportunity to respond in a so-called “rebuttal letter”. Again, like for revision letters, the rebuttal needs to be balanced and appropriate, but provides another formal feedback mechanism for authors to defend their work and make corrections if the reviewers misunderstood your their work. My guide on writing rebuttals can help with this.
  • Reviewer feedback: If you feel that you absolutely must do something about an unfair review, but at the same time you know that complaining is unlikely to change anything (i.e. you read this far in this document), then sending a simple letter of feedback to the papers chairs or editor may be one option. In such a letter, you would acknowledge that you in no way are asking for your paper to be reconsidered, but just that you would like to highlight a poor review. Perhaps this is a pattern from that same reviewer, and the papers co-chairs may not be aware of it until they receive enough such feedback (honestly, in my experience, we do have a good feeling about flaky or overly harsh reviewers, but that’s besides the point). Anyway, at the risk of opening the floodgates for such reviewer feedback emails, and pending a process where the authors get to rate their reviews, this is one example where emailing the papers co-chairs can be a good choice.
  • Dealing with rejection: If all else fails and no other option is open to you, consider just silently dealing with the rejection and chalking it up as the price we all pay to participate in the peer review process. This is not a perfect world: some injustices are never corrected, and you need to be okay with that. I wrote a guide on dealing with rejection some time ago as well; it explicitly mentions not emailing the chair or editor, and gives some fruitful alternatives instead (spoiler: the best advice from that article is to use the review feedback to revise your paper and resubmit it to the next deadline).

Hopefully this little post will dissuade at least a few authors from sending off an angry email in response to the next round of reviews they get. Not only is it a waste of your time, it is also likely going to result in negative karma for yourself. Save your powder and instead focus on something productive, like revising the work and resubmitting to the next deadline. In the long run, this will pay off much more than bashing reviewers, papers chairs, and editors.

Document History

  • June 30, 2017 – original post.
  • July 5, 2017 – added scalability discussion, added feedback letter.