Friday, June 25, 2010

Bad Research As Spam

Jon Katz and Matt Welsh have both written recently about the problems of dealing with crap papers, mainly the pain of having to review them. In an unrelated event, I got into a discussion with some colleagues about the problems of "crap research', and ended up formulating a theory: viz,
bad research is like spam email
in the sense that

  1. There seems to be no way to stop it, and many economic incentives to continue it
  2. You can't stop it by diktat
  3. The only way to deal with it appears to be effective filtering. Like spam, bad research has less of an effect when it gains no attention. 
There are other similarities:
  1. we can block spam by filtering certain domains. We also tend to ignore certain kinds of conferences
  2. we can block spam by blocking certain email addresses. We also might ignore certain researchers, or at least downweight their work after a series of bad experiences. 
  3. More explicit spam blocking policies create a false-negative problem. False-negatives are also a big problem in research.
But this analogy also suggests that we shouldn't be designing strategies to eliminate bad research. We should be designing better methods to focus attention on good research, via better filtering and highlighting mechanisms (social, authoritative or otherwise). 

Personally, I think bad research is less of a problem than lots of cargo-cult research, where it looks a lot like good research is being done, but when you look closely, you realize that nothing of value has been added to the universe. Sadly, a lot of funded research is also like this. 

PS: False negatives are a highly underrated problem in research. I think we vastly overestimate our ability to judge what kinds of research will have lasting value and what potential impact a paper can have. So it's important to err on the side of being more generous to papers, rather than less. 

4 comments:

  1. One difference (I haven't read your post yet): Bad research is often done by good, or at least well-meaning, people. Spammers may look superficially decent, but they generally lack any sense of morality.

    ReplyDelete
  2. What is crap research for you, might be good research for me. Didn't we have exactly this discussion about one of my papers? ;)

    Refereeing crappy papers is relatively easy and it is considerably less common than spam. However, it does support my opinion that only exceptionally good papers should be published in Journals. Unfortunately, I seem to be the only person to support this bizarre idea. --S

    ReplyDelete
  3. @S: indeed, and that's why it's important not to make strong judgements about papers. we just don't know enough !

    ReplyDelete
  4. All good points. We need to bring Clay Shirky in this discussion:

    It's Not Information Overload. It's Filter Failure.
    http://www.youtube.com/watch?v=LabqeJEOQyI


    I find that a lot of important papers have appeared in unknown journals and could pass as spam. Meanwhile, a lot of work appearing in the most prestigious venues goes on to mean nothing. The honest truth is that we are, collectively, really bad at predicting the future.

    ReplyDelete

Disqus for The Geomblog