Notes on “A framework and methodology for studying the causes of software errors in programming systems”

June 27, 2009

“A framework and methodology for studying the causes of software errors in programming systems”: Andrew Ko and Brad Myers.

Six part paper. Expanded version of Ko 2003. 40+ pages.

Section 1: Software errors are common and expensive. Existing tools for describing programming system error-proneness are inadequate.

Section 2: Classifications of programming difficulties

Four salient aspects of software errors:

  1. Surface Qualities: Notational issues
  2. Cognitive Cause: Forgetting, misunderstanding, not knowing, etc.
  3. Programming Activity: Specification, algorithm design, etc.
  4. Type of Action leading to the error: Creating code, reusing code, modifying specifications or code, designing, etc.

James Reason’s Human Error gives a systemic view of failures and a catalog of common cognition failures.

A system has several layers, each with its own defenses. An error in one can lie dormant until it interacts with errors in other layers.

In software engineering, the layers could be viewed as:

  • Specifications (Ambiguous, incomplete, incorrect)
  • Programmer (Knowledge, attention, expertise)
  • Programming System (Compiler errors, static checkers, syntax highlighting, assertions, etc.)
  • Program (Software errors, possibly predisposed by one of the previous layers.)

An error at the programmer layer is called a cognitive breakdown.

Three types of cognitive activity:

  1. Skill-based activity: Routine actions. Doesn’t really require much attention. E.g. Opening a file. Fails due to inattention breakdowns (e.g. strong, wrong habit kicks in, routine action interrupted but not resumed.) or overattention
  2. Rule-based activity: Learned expertise. Fails because of broken rules or wrong rule choice. e.g. in C, to map an action on an array, use a for loop.
  3. Knowledge-based activity: Conscious, deliberate problem-solving. Fails due to cognitive limitations and biases inherent in cognition. e.g. satisficing incorrectly, confirmation bias + overconfidence.

Section 3: Framework

Programmers have three types of activities: Specification, Implementation, Runtime

Six actions: Design, create, reuse, modify, understand, explore

Three breakdowns: Skill, Rule, Knowledge

Cognitive breakdown has four parts:

(What kind of breakdown) during (what action) through (what interface) on (what information).

Chains of cognitive breakdowns lead to software errors.

Section 4: Empirical methodology for using the framework to study a programming system’s error-proneness

  1. Design a programming task
  2. Observe using think-aloud
  3. Work backward from software error to causes
  4. Analyze chains of breakdowns for patterns and relationships

Section 5: Case study of using framework on Alice

40 hours of analysis on 15 hours of video.

Most errors caused by incorrect boolean conditions, copy-paste with references offscreen. It seems fairly straightforward to map errors through the framework.

Section 6: Discussion

The framework allows empirical comparisons. The methodology is limited in that it requires a lot of work, it may be difficult to learn, and its replicability has not been tested.

%d bloggers like this: