Notes on “Heuristic Evaluation of Ambient Displays”

October 19, 2008

Heuristic Evaluation of Ambient Displays: Jennifer Mankoff, Anind K. Dey, Gary Hsieh, Julie Kientz, Scott Lederer, Morgan Ames.

Jakob Nielsen’s Ten Usability Heuristics are the gold standard of heuristic evaluation for computer interfaces. He has established their validity and utility for the improvement of software. In fact, he established heuristic evaluation itself as a legitimate technique.

His heuristics don’t intuitively apply to Ambient Displays, however, where the goals are different from traditional user interfaces. In a traditional user interface, the user’s full attention is focused on a task. An ambient display is meant to capture the user’s attention only when necessary and even then only for a brief moment.

The authors of this paper brainstormed new heuristics for the evaluation of Ambient Displays. They compared the new heuristics with Nielsen’s heuristics and concluded that the new heuristics were useful and superior to Nielsen’s.

The paper provided insight into how they went about developing their new heuristics. They also made a significant effort to attach a quantitative and empirical foundation for the new heuristics. This seems like a difficult obstacle to overcome given the fluid and subjective nature of heuristic evaluation in general.

I’m suspicious of their results. First, they failed to find an effect for the hypothesis that “the number of issues found in the ambient condition will be greater and the issues will be more severe than those found in the Nielsen condition.” Second, they eliminated an outlier to find an effect for the hypothesis “The ambient heuristics will be more useful to evaluators than Nielsen’s heuristics. A heuristic that finds many, severe problems is more useful than a heuristic that finds fewer problems with lower severity.” I don’t agree that the elimination of the “outlier” was appropriate or even that the data point they eliminated was an “outlier.”

The problem was that they weren’t making a valid comparison. Nielsen’s heuristics are inferior to any set of heuristics designed for ambient displays, because Nielsen’s heuristics aren’t about ambient displays. It’s like applying an evaluation chart for a beauty pageant to a dog show. I think a more appropriate experiment would be to instruct evaluators to find all the issues with an Ambient Display. In one condition, give them the heuristic list and in the control condition give them no list.

%d bloggers like this: