Sunday, March 01, 2009

Double Blind Review, again...

Sorelle's excellent post and discussions got me thinking more about DBR, and then I read Michael's post, which captured an important part of what I wanted to distill out. So first, a summary of the summary, and then other important points.

There are a few serious reasons brought up against DBR. I should say up front that I don't view reasons of the form "We are unbiased/Theory papers can be reviewed with subjectivity/We are better than other fields" as serious, because there's ample empirical evidence for the existence of unconscious systematic biases in many fields (no, not in theory specifically, but we're are still human). And as Michael points out, there's sufficient evidence of actual bias (even if it's not considered malevolent)

The serious reasons against DBR are essentially three:
  • DBR places an unacceptable restriction on the author's ability to disseminate the work
  • There is no way a paper can get a fair review without the reviewers being able to google around to get a sense of the work being produced. Since such a process could easily reveal the names of the authors, this defeats the purpose of DBR, giving an illusion of fairness where none really exists
  • You can't do subrefereeing if you don't know who the authors are.
Michael addressed the first in his post, and the second points more to weakness in the review system than DBR itself. There are mechanisms for dealing with the third in conferences with DBR.

But most importantly, I think the objections are missing the point. If the claim was, "SBR is unfair, and here's a perfect system to replace it", then these objections would be reasonable counters to that claim. But in all that I've read, the reason for DBR is:
To replace the systematic biases associated with SBR by a 'equal playing field of problems' when it comes to paper review.
In other words, the point is not to be perfect, but to be imperfect in a fair way, that doesn't unfairly work against group identity that has nothing to do with the quality of the research.

I really think that perspective is important to keep in mind when discussing this. I have yet to hear any argument for significant harm to authors in going from SBR to DBR. Inconvenience yes, but direct harm, no. And any harm is spread "across the board": I am as likely to suffer in this system as a new grad student or a famous researcher. But the benefits are disproportionate, and correct for structural biases, and that's the point.

11 comments:

  1. If a paper takes over a year to get through the referee process (which is not at all an unreasonable estimate for many journals), then that means it's not on arxiv.org or ECCC for a year. This is not only bad for science, it disproportionately hurts the people who aren't at the center of things, and who don't hear about results through informal networks.

    ReplyDelete
  2. To replace the systematic biases associated with SBR by a 'equal playing field of problems'...

    As others have pointed out, the problem is that DBR does not achieve this. People who "play by the rules" (a bad phrase, since it is unclear what the rules are) will have their papers judged anonymously. People who post their work to an archive, give talks about their work, and send preprints to their friends will have the (supposed?) advantage of non-anonymous review.

    ReplyDelete
  3. Actually, maybe it would be a good thing. After some famous professor gets a few papers rejected they'll learn that what they need to do is put their papers on the arXiv as soon as they're ready.

    The next time the committee knows to give the paper the respect the professor deserves. People get to see preprints faster. DBR doesn't accomplish what it was supposed to but has a positive effect nevertheless.

    ReplyDelete
  4. The system is not broken, so dont fix it. --Sariel

    ReplyDelete
  5. It amazes me that people make the "smart enough to be unbiased" claim with a straight face. The same goes for the claims of being able to ignore the author order and weight all authors equally.

    ReplyDelete
  6. Sariel --

    You should clarify what you mean by "not broken".

    If your claim is that bias does not actually affect decisions being made on the floor, I'm very dubious of that claim. It's clear to me that bias enters the picture, as I've stated in my post on the subject. It's not clear to me what overall effect that bias has, but to suggest it has no eventual impact or even minimal impact without additional evidence seems presumptuous. Similarly, if your claim is such bias is actually helpful, you should present an argument as such.

    If your claim is that the costs of double-blind reviewing are sufficiently high that the apparent gains one might hope from it are not worth the cost, that seems to also be a worthwhile argument to try to make.

    But comments like "The system is not broken, so don't fix it" serve to convince me that the arguments on the other side that the system is broken can't be effectively answered, making me more in favor of trying double-blind admissions.

    ReplyDelete
  7. "but we're are still human"
    Is this an example of proof carrying code? :-)

    ReplyDelete
  8. The question is not if there is bias in the system. The question is if the effect of this bias causes enough damage to be worthy to do something about it. There is no proof provided so far in the discussion that such a significant damage exists. In particular, a *minor* lack of fairness does not disturb me - life is unfair, deal with it. No system will provide perfect fairness, or even better fairness.

    My second point, is that DBR channels energies from the important stuff - doing research, doing good job refereeing papers, just to create pretense of fairness.

    Thirdly, since I consider DBR to be a waste of my time, I will not submit/subreferee etc for a theory conference doing DBR.

    BTW, FOCS/STOC/SODA can benefit more from an improved refereeing process (like making PC biggers (and maybe have two layered PCs), making sure papers are refereed by the people, etc) than any improvement due to DBR.

    ReplyDelete
  9. BTW, I guess my negative response for the whole idea comes from horror stories I heard from people in networking, where if you fail to make your paper anonymous enough, it gets rejected from a conference. Similarly, biology has very weird refereeing process for Nature, where you have to keep your research secret till they decide if to accept it or reject it. (i.e., the chilling effect.) There might be a variant of DBR that avoids all these effects, but to me, it feels just like a step in the wrong direction.

    Let me offer a counter proposal (just to amuse people) - I call it the doubly openly open review system (DOOR) - all the papers submitted to a conference are immediately made available online on the conference webpage (with authors names, and their swiss bank accounts available to everybody). Any person that wants, can submit (not anonymously) comments to the PC about any paper they want to comment on. PC discussion is still confidential, naturally. One can even have an open (but moderated) discussion channel on each paper.

    ReplyDelete
  10. If a paper takes over a year to get through the referee process (which is not at all an unreasonable estimate for many journals), then that means it's not on arxiv.org or ECCC for a year.

    Why wait until a paper is accepted before submitting it to the arXiv?

    ReplyDelete
  11. Consider the following situation. One writes a report and submit it to a conference with double-blind review process. Shall these guys still submit the report to the arXiv before the conference paper gets accepted/rejected? Will these guys be unfair?

    ReplyDelete

Disqus for The Geomblog