Notes on “Competencies of Exceptional and Non-Exceptional Software Engineers”

June 4, 2009

Competencies of Exceptional and Non-Exceptional Software Engineers: Richard T. Turley, James M. Bieman

The setting is a large company. There are twenty software developers, grouped by managers into “Exceptional (top 5%)” and “Non-exceptional” and matched on years of experience. There is no discernible difference between the MBTI personality types of the groups. Interviews conducted via critical incident analysis, e.g. “tell me about a time you were at your best.” The recorded results are coded for statements about the important qualities of software engineers. The qualities elicited include engineering habits like prototyping, test automation, code reuse, code reading; personality factors like sense of fun, lack of ego, perseverence; and attitudes like pride in quality, strength of convictions, bias for action. For each group, count the number of subjects that claim the quality, and find that no quality distinguishes between the groups except prototyping.

Now distribute surveys to a broader sample of engineers in the same company, separating performers ranked in the top 30% from the bottom 70%. The surveys force the subjects to rank their habits, thereby choosing the habits they feel most accurately describe their behaviour. The fact that it’s a ranking is important and is critical for understanding the results described below.

Prototyping as a significant difference vanishes. (Maybe because only top 5% can prototype, but not top 30%?) Instead, top performers report a higher priority on mastery of skills and techniques, help others, maintain a big picture view, have strong convictions, and take a pro-active role with management. (Which looks to me like they are both better team players and technical team leads, i.e. the expert on a product or module.) Meanwhile non-exceptional subjects are more likely to be driven by a desire to contribute, seek help from others, and be willing to confront others.(In my view, less familiar with a domain and weaker team players.)

Experience is the only significant biographical factor distinguishing the groups.

Despite the bad news on single predictors, a discriminant function of 10 variables is able to predict >80% of cases. The ten variables (+ for high performer, – for non-exceptional) are: +Years at company in software, +number of languages used in typical work, +math(!) degree held, -perseverance, +bias for action, +sense of mission, +strong convictions, -responding to schedule pressure by sacrificing design, +helping others and -being willing to confront others.


One Response to “Notes on “Competencies of Exceptional and Non-Exceptional Software Engineers””

  1. Neil Says:

    I think CS grads from Waterloo get a B.Math degree, which might be the case elsewhere. So the question is what type of math they did for their degree — non-Euclidean geometry? Hyper-dimensional brane analysis? Python?

Comments are closed.

%d bloggers like this: