Notes on the Empirical Studies of Programmers, First Workshop (and some papers cited therein)

April 24, 2009

Human Detection and Diagnosis of System Failures

J. Rasmussen

Symptomatic troubleshooting matches symptoms of a problem to memories of symptoms due to past problems.

Topographic troubleshooting narrows bug locations in the source code. Strategies include working backward from erroneous results, elimination of known good code and mentally simulating candidate bug locations.

Comprehension Differences in Debugging by Skilled and Novice Programmers

Experts debug better than novices because they form better hypotheses about bug possibilities.

Experts and novices perform qualitatively similar activities during debugging.

Troubleshooting by comprehension means mentally simulating the program and comparing actual results to desired.

Plans in Programming: Definition, Demonstration and Development

Robert S. Rist

Programs are plan trees designed to accomplish hierarchical goals.

Novices use plans less as difficulty goes up. Specific plans are learned through experience.

Towards a Theory of the Comprehension of Computer Programs

R. Brooks

Program comprehension is largely a search for beacons in the code confirming a hypotheses about program function.

Processes in Computer Program Comprehension

Susan Wiedenbeck

Experts recall beacons better than novices. Beacons are focal points for program comprehension by experts.

Cognitive Processes in Program Comprehension

Stanley Letovsky

Programmers form a mental model of a program by “combining information gathered through reading the code and documentation with knowledge from a knowledge base of programming expertise.”

During a program comprehension task, subjects make inquiries, that is, form conjectures and ask questions.

Questions are divided into why, how, what, whether and discrepancy questions.

Why questions ask about the purpose of some code or design. (Or, what is the goal?)

How questions ask how known goals are accomplished by the program.

What questions are about names. (What is foo?)

Whether questions match hypotheses against the program. Does the program behave in a certain way?

Discrepancy questions reflect inconsistencies in the mental model of the program.

Conjectures are hypothesized answers to the questions.

A mental model of a program includes the goals, functions and data structures of the program. It also includes how they relate to each other.

Programmers build the mental model both top-down and bottom-up.

Mental Models and Software Maintenance

David C. Littman, Jeannine Pinto, Stanley Letovsky and Elliot Soloway

There are two main strategies for learning about a program for a change task. A systematic strategy aims to study the whole program and understand it before making the change. An as-needed strategy aims to target the learning to the bare minimum needed to make a change.

On a program change task for a small program, experts were systematic and novices were not. Those who followed the systematic strategy were much more successful than those that didn’t.

The Effects of Program-Dependent and Program-Independent Deletions on Software Cloze Tests

Mark Thomas, Stuart Zweben

Cloze test is the fancy name for fill in the blanks. Cloze tests are an accepted measure in the psychology of reading.

If a blank can be completed regardless of the understanding of the problem in question, it is a program-independent term.

If a blank can only be completed with a good understanding of how the program works, it is a program-dependent-term.

Software engineering researchers creating cloze tests must keep this distinction in mind and control for it.

An Empirical Study of the Effects of Modularity on Program Modifiability

Timothy D. Korson, Vijay K. Vaishnavi

…a modular program is faster to modify than a non-modular, but otherwise equivalent version of the same program, when one of the following conditions hold:

  1. Modularity has been used to implement “information hiding” which localizes changes required by a modification.
  2. Existing modules in a program perform useful generic operations, some of which can be used in implementing a modification.
  3. A significant understanding of, and changes to, the existing code are required for performing a modification.

Without one of the above conditions, increased modularity doesn’t help.

This study had an experimental design which ruthlessly controlled for programmer ability.

Experiments on Slicing-based Debugging Aids

A program slice for a given value is the subset of the larger program that affects the value.

Automatic program slicing does not help in debugging, surprisingly.

Program dicing is program slicing, but with code that contributes to known correct values also removed.

Automatic program dicing does help in debugging.

Analyzing the High Frequency Bugs in Novice Programs

A few types of bugs account for most of the errors new programmers make; however, they rarely reflect misconceptions about programming language constructs.

Ideas Generated

Do a variety of cloze tests: Delete all variables. Delete all keywords. Delete all expressions. And so on.

Construct a modern program dicing tool that integrates with a modern unit test framework.

%d bloggers like this: