Idea: Programmer Productivity Measurement via Inspection of Version Control

November 2, 2008

Measuring programmer productivity is a hard problem. Programmers game the system to maximize their top line and it’s hard to agree on a sensible definition of the bottom line. I understand this. I don’t claim to have solved the problem of measuring programmer productivity, because that would be crazy. Regardless, I’d be curious to see the result of the following:

Take a software project that has:

  • A version control system
  • An automated test or benchmark suite
  • Multiple full-time contributors
  • A large code base (thus, )
  • A significant history measured in years (thus, )
  • Lots of commit data

Now, we use the following system on existing data:

  • A programmer get points for lines of code committed, if and only if that code is exercised by the automated test or benchmark suite. The programmer gets negative points for code that isn’t exercised by the automated tests. This aims to “reward” useful contributions.
  • Those points are made negative if later another programmer changes or deletes the line of code. We assume that code was buggy or needed cleanup, costing another programmer or future self productivity. This is meant to “penalize” prolific-but-bad coders, introducers of bugs, and code churners.

The overall point of the exercise is to see if we can detect programmers who are hidden net negative contributors who wouldn’t obviously be revealed by contribution lines-of-code counts or who seem to get lots done at cost of teammates’ productivity.

I’d be happy for other suggestions on how to go about detecting these.

Advertisements

3 Responses to “Idea: Programmer Productivity Measurement via Inspection of Version Control”

  1. Andrew Says:

    Interesting but I think the penalization could be hard to get right. Maybe tying it to a task/bug tracking system like Trac where developers use structured references to specific tasks/bugs in their commit comments could help you to better handle what to reward and what to penalize.

    I think you could take this idea in another direction though. Discard the goal of measuring productivity and instead cast the problem as one of making programming more fun via collaboration or competition. The points become part of a game where you compete to develop the fastest/smallest/best tested code. I’d enjoy competing and learning from other programmers in this way while at the same time creating useful code.

  2. aran Says:

    @Andrew: On your first point, I’m convinced that the structured systems don’t contain the right information. Lots of important code changes happen that aren’t formally tracked in bug trackers. Even in organizations with thorough requirements about checkins, programmers piggyback small informal bug fixes on top of bigger ones.

    On your game idea, you might be interested in one of the papers we read in Greg’s reading group: http://www.cs.toronto.edu/~gvwilson/reading/gulley-in-praise-of-tweaking.pdf

  3. Ian Says:

    I had to think about this one a bit. I’m not convience it is a mearsure of productivity, but more a measure of code quality. If an area of code is ‘hot’, and requires a lot of change, it is an indicator of a number of potential things: poor/incomplete design and/or requirements, poor implementation, poor inspections/reviews, poor testing.

    It seems to be a great way to identify pieces of code that require closer scrutiny, and either a re-design, re-coding, re-inspection, or re-testing.

    It certainly is a great use for the version control system’s information.


Comments are closed.

%d bloggers like this: