Notes on “On the Difficulty of Replicating Human Subjects Studies in Software Engineering”

May 7, 2009

On the Difficulty of Replicating Human Subjects Studies in Software Engineering, J. Lung, J. Aranda, S. M. Easterbrook, and G. Wilson

The authors replicated an unpublished study which claimed to be able to predict programmer aptitude before training. The study had some inevitable modifications and found a contradictory result. We still can’t predict programmer aptitude.

Perfect replication is hard for a variety of reasons. These include cost, environmental differences, ethics board differences, large gaps in skill between subjects and others.

This paper somehow turned around my view of Software Engineering research. It occurred to me that Empirical Software Engineering is more like sociology than psychology. Psychologists study intrinsic properties of people. Sociologists study the systems and large-scale interactions. Psychologists are able to establish valid reliable constructs and perform controlled laboratory experiments because the properties they study don’t require time. Sociologists are unable to make such crisp conclusions because they’re forced out into the broader environment.

This is a key difference between the Empirical Studies of Programmers and the newer Empirical Software Engineering. The Empirical Studies of Programmers were focused on the psychology of programmers and they were able to perform controlled studies to get at the results they wanted. Newer Empirical Software Engineering research is forced out into the wild and controlled studies become very difficult.

%d bloggers like this: