On Non-Truthful Peer Evaluation Mechanisms

February 23, 2009

In September, I reviewed a paper called “Turning Student Groups into Effective Teams.”

In one of my classes, we do a large project in groups of four. Based on a suggestion by the paper, the professor has implemented a peer review system.

The peer review system compares an evaluation of me to the average evaluations for my group.

Here is the system:

  1. Every teammate gives a rating for every other teammate’s performance for each assignment. In a group of four, I make three ratings and I am rated three times.
  2. My overall rating is the average of the three ratings of me.
  3. The baseline rating is the average of all the overall ratings.
  4. The professor assigns a mark to the group.
  5. If my overall rating is higher than the baseline rating, my mark is the group mark plus an adjustment bonus.
  6. If my overall rating is lower than the baseline rating, my mark is the group mark minus an adjustment penalty.

Game theorists have a concept called “Truthful Mechanisms.” A truthful mechanism is a system in which every player’s best interest is to tell the truth. 

This is a non-truthful mechanism. Nobody should use it ever.

Imagine I give all my teammates a perfect rating. If any teammate gives me a less-than-perfect rating, my overall rating will be lower than otherwise and my mark goes down. By giving everyone a perfect rating, I raise the team average and hurt myself.

Imagine I give all my teammates a zero rating. If any teammate gives me a better-than-zero rating, my overall rating will be higher than otherwise and my mark goes up. By giving everyone a zero rating, I lower the team average and help myself.

The professor argues that, because we perform this evaluation rigmarole several times, unfair conduct will be weeded out. I say, garbage in, garbage out. The incentive becomes “mark your teammates as low as you can get away with.”

The system is malicious: It pits teammates against each other.

The system is also very dumb. It can punish individuals in perfectly functional teams!

Let’s say a team works really well together. Pete thinks, “Everything went very well. I’ll give everyone ‘Excellent.'” Everyone else thinks, “Everything went well. I’ll give everyone ‘Very Good’.” Result: Pete’s mark is adjusted downward. Everyone else is adjusted upward. Oops!

Please, do not use the system presented in the Oakley paper.


4 Responses to “On Non-Truthful Peer Evaluation Mechanisms”

  1. Ian Says:

    Through the past 20 years, I’ve seen peer evaluations drift in and out out vogue, and I’ve done a fair number of peer evaluations – and I agree – in general they don’t work. You’re evaluating your peers from the reference point of yourself … or more specifically, your impressions of yourself.

    Personally, I’ve found that the *only* aspect that seems to work is the strength/weaknesses feedback that is some times included in the survey/evaluation – and hopefully fed back to you. If there is strong correlation in the responses then it is useful. Getting team mate’s impressions of you work is far more telling and useful then relative rankings.

  2. Anonymous Says:

    Aran, design me a better system and I’ll use it.

    BTW your description of the algorithm is wrong – you are supposed to rate yourself too. That fixes many of the problems you describe, but not all of them.

    I do use the Oakley method for my course, but I never apply it mechanically – I look at each team’s overall pattern of responses, and read their comments on each other, and them make adjustments to the output of the algorithm. I’ve never seen anyone blatantly game the system, but I have seen some (what appear to be) accidental problems of the type you describe.

    A more important weakness is that if two teammates fall out, and rate each other zero, the rest of the team get a bonus. And unless there are any clues from the rest of the team about what happened, there is no way to come up with a justifiable allocation of marks.

  3. […] Better Peer Evaluation System March 26, 2009 I’ve ranted against a popular peer evaluation system. An anonymous commenter challenged me: “Make […]

Comments are closed.

%d bloggers like this: