Notes on “A systematic review of theory use in software engineering experiments”

June 23, 2009

A systematic review of theory use in software engineering experiments: Hannay, J.E. et al.

Theories are for the generalization of knowledge. They are the engine of our deeper understanding of the world. Experiments provide descriptions of causality, but theories explain causality.

The article is an overview of theory for the purposes of software engineering.

Challenges for a software engineering researcher: Finding relevant theories, determining what counts as a theory (called ‘theory identity’), using theories explicitly.

There are five types of theory:

  1. Analysis: Descriptions of the world. It is controversial whether these count as theories. .
  2. Explanation: Explanations of the world lacking predictive power. A UML model can be viewed as a type 2 theory of the design of a software system
  3. Prediction: Providing predictions without explanations e.g. statistical models
  4. Explanation and prediction: Most commonly accepted kind of theories.
  5. Design and action: How-to instructions including design principles, usually with an implicit prediction of benefit.

The article contains a brief discussion of the epistemology of theory, which I don’t care about.

Theories are built of representations, constructs and relationships in a scope. These are conceptual-level concerns. Experiments operationalize these, e.g. experimental variables operationalize constructs, treatments operationalize causes, hypotheses operationalize predictions. .

Theories can be used in several roles within experimental studies. A theory can inform the design of an experiment, it can be used to explain the results of an experiment post hoc, it can be tested by (and perhaps modified subsequent to) an experiment, it can be proposed given the results of an experiment, or it can form the basis of other theories that take the other roles.

The authors filtered much of current ESE research to identify 103 article that mention theory use in one of the roles above. They performed some sophisticated analyses of theory use in these articles. Theories are rarely used more than once; they are most commonly used to inform research questions and the design of experiments; theories are more frequently proposed than tested. The take-home message seems to be that theory use in ESE research is weak, although it is acknowledged that better theory use would be a good thing.

I note that this paper’s research method did not operate using any theory of its own.

Despite the fact that Greg sent this one for us to read (5 weeks after it went into my reading queue, thankyouverymuch), this was a good paper. It would be fun to brainstorm some grand unified type-3 theories of software engineering.


6 Responses to “Notes on “A systematic review of theory use in software engineering experiments””

  1. Greg Wilson Says:

    “Despite the fact that Greg sent this one for us to read…this was a good paper.” Despite? *sigh*

  2. aran Says:

    @Greg: *grin* just checkin’ who’s paying attention 😉

  3. I’m up for the brainstorm.

  4. Jorge Says:

    I’m up for it too, but wouldn’t you rather have some Type-4 theories?

  5. aran Says:

    @Jorge: Dammit! Yes. When I typed Type-3, I meant Type-4.

  6. Well, I just figured in a proper brainstorm, nothing gets excluded, so I can even invent some type-6 theories too…

Comments are closed.

%d bloggers like this: