On the Validation of Tool-Oriented Research

November 9, 2008

Or, Why the Little Computer Scientist Has Beef With “We Built Something Cool”-type Academic Papers.

Many papers describe a technology the authors built. I call these the “We Built Something Cool” papers. Most of the papers from my Ubiquitous Computing class fall into this category, along with a few Software Engineering reading group papers.

Here are some examples of papers in the category, of varying quality:

  • Fishpong: Encouraging Human-to-Human Interaction in Informal Social Environments by Yoon et al.
  • Uncle Roy All Around You: Implicating the City in a Location-Based Performance by Benford et al.
  • In Praise of Tweaking by Gulley
  • Digital Family Portraits: Supporting Peace of Mind for Extended Family Members by Mynatt et al.

There’s a common pattern: The researchers build a system, describe it, and publish. It’s getting old. 

Without some sort of validation of the utility of the system, there’s no answer to a claim of uselessness. We have a word for “useless things built for interest or fun”: Hobby. Go publish what you built on your blog. See if the world cares.

Here is the Little Computer Scientist’s Five-Step Plan for “We Built Something Cool” Papers That Don’t Suck:

  1. Name a clear problem. Quantitatively show that it exists and is worth fixing. (For those papers that don’t solve a problem but instead enhance the status quo, give a quantitative baseline to which you will compare your system.)
  2. Survey existing solutions for the problem. Explain their inadequacy.
  3. Describe the design of your solution. Describe the domain-specific lessons learned during the iterative development and testing of the solution.
  4. Show that your solution solves the problem better than anything else. Make sure that the testing generalizes to the target population. The results should be reproducible. I should be able to borrow or duplicate your system, evaluate it in the same way as you did, and get the same result.
  5. Describe how your solution has been made available to the public. Are you commercializing it? Is it available for download? Give some indication that it is on its way to enthusiastic adoption by those affected by the problem.

Some will argue that this bar is too high. There may be something to what they say. After all, it’s easy for me to say “do more.” I’m not writing papers yet; I’m no more than a practitioner-in-training and a cheap critic. It’s just that I’ve noticed a pattern in my reviews which is caused by a pattern in the papers I’ve reviewed.


3 Responses to “On the Validation of Tool-Oriented Research”

  1. Neil Says:

    I think there’s one case where someone can get away with this: if the thing they build is later widely adopted. Then, even though they haven’t done any of the points you mention, I think we can agree that it’s notable work. For example, TBL and the WWW.

    Most of the papers you mention certainly won’t be in that position, however.

  2. aran Says:

    Neil – I went looking for a “seminal” paper about the web. The best I can find is TBL 1996: “WWW: Past, Present and Future.” It easily meets the criteria above because in 1996 the Web was already validated by a public implementation in widespread use, with thousands of clients.

  3. Jorge Says:

    This is one of my pet peeves too, to the point where, to me, these papers Don’t Count.

    They make for great demos in conferences though.

Comments are closed.

%d bloggers like this: