On Non-Truthful Peer Evaluation Mechanisms

In September, I reviewed a paper called “Turning Student Groups into Effective Teams.”

In one of my classes, we do a large project in groups of four. Based on a suggestion by the paper, the professor has implemented a peer review system.

The peer review system compares an evaluation of me to the average evaluations for my group.

Here is the system:

1. Every teammate gives a rating for every other teammate’s performance for each assignment. In a group of four, I make three ratings and I am rated three times.
2. My overall rating is the average of the three ratings of me.
3. The baseline rating is the average of all the overall ratings.
4. The professor assigns a mark to the group.
5. If my overall rating is higher than the baseline rating, my mark is the group mark plus an adjustment bonus.
6. If my overall rating is lower than the baseline rating, my mark is the group mark minus an adjustment penalty.

Game theorists have a concept called “Truthful Mechanisms.” A truthful mechanism is a system in which every player’s best interest is to tell the truth. 

This is a non-truthful mechanism. Nobody should use it ever.

Imagine I give all my teammates a perfect rating. If any teammate gives me a less-than-perfect rating, my overall rating will be lower than otherwise and my mark goes down. By giving everyone a perfect rating, I raise the team average and hurt myself.

Imagine I give all my teammates a zero rating. If any teammate gives me a better-than-zero rating, my overall rating will be higher than otherwise and my mark goes up. By giving everyone a zero rating, I lower the team average and help myself.

The professor argues that, because we perform this evaluation rigmarole several times, unfair conduct will be weeded out. I say, garbage in, garbage out. The incentive becomes “mark your teammates as low as you can get away with.”

The system is malicious: It pits teammates against each other.

The system is also very dumb. It can punish individuals in perfectly functional teams!

Let’s say a team works really well together. Pete thinks, “Everything went very well. I’ll give everyone ‘Excellent.’” Everyone else thinks, “Everything went well. I’ll give everyone ‘Very Good’.” Result: Pete’s mark is adjusted downward. Everyone else is adjusted upward. Oops!

Please, do not use the system presented in the Oakley paper.

Code Sprint

We just had three days straight of coding on PyVCAL. Jobias flew in from Alberta to join us in person. There was coffee and fruit. There were donuts and subs and delicious (ginger!) homemade cookies. We went out for tapas.

We got stuff done. We can connect to perforce, subversion and git. We can get basic information out of each repository. Our test infrastructure is in place and our tests will prove our progress soon. We refined out some design errors in our API. We made a big design decision that clarifies the difference between our representation of a version control system and our representation of a specific repository.

Haz made an automated checklist for our progress. Derek will implement a mock version of the API. Shazow connected us with the developer of AnyVCS. SubvertPy will relicense under the LGPL so we can use it.

Hack Session 3

Andrew flew solo to put together a new, improved, final version of the Idea Hat.  

Our hack sessions are open to anyone interested in having some fun and writing some code.

Resources for User Interface Design

A close friend asks,

Can you recommend/lend a good book on software UI design? Managing a

redesign of UI? Rolling out new UI to existing users?

The Little Computer Scientist answers,

Sure…

I’ve got:

• Paper Prototyping by Carolyn Snyder: About designing and evaluating UIs using paper
• The Design of Everyday Things by Don Norman: High-level design principles
• Don’t Make Me Think by Steve Krug: Specifically about designing web interfaces. Very short.
• Human-Computer Interaction (textbook) by Dix, Finlay, Abowd & Beale: Fairly deep. More stuff on Ubiquitous/CSCW/Groupware-type stuff than most people care about. But good stuff about doing ethnographic studies.

I don’t own, but strongly recommend:

• User Interface Design for Programmers by Joel Spolsky: Takes a “bare minimum you need to know” approach targeted at programmers. Partially available online (http://www.joelonsoftware.com/uibook/fog0000000249.html). The full version is worthwhile.
• The entire archives of useit.com by Jakob Nielsen. Free, and easily digested in small chunks. I’m subscribed to the newsletter and read new articles as they come along.
• GUI Bloopers by Jeff Johnson: Learn by (often hilarious) counter-example. Super-readable.

Honestly, I just don’t think UI design work is that hard. Learn a few foundations like Hick’s Law and Fitt’s Law, learn a handful of heuristics like seeking consistency and simplicity. Then just prototype alternatives and test, test, test on real people.

On managing a redesign of a UI and rolling it out, I don’t have much. Jensen Harris managed the Microsoft Office 2007 redesign and rollout. He documented a lot of thinking and decisions at http://blogs.msdn.com/jensenh/. That’s as real an experience as it gets.

A List of Definitive Computing Books

There are many books on every niche of computing. A few of these books are definitive, in that there is wide agreement that one single book is the book on a subject.

Here I attempt to gather a list of these books.

Career Programming

The Pragmatic Programmer

Code Construction

Code Complete

Algorithms

Introduction to Algorithms. The popular alternative is The Art of Computer Programming, but as I understand it nobody actually reads it.

Artificial Intelligence

The Green Book

Compilers

The Dragon Book

JavaScript

JavaScript: The Definitive Guide

C

The C Programming Language

C++

The C++ Programming Language

Java

Effective Java

Erlang

Programming Erlang

Ruby

Programming Ruby

Design Patterns

GoF 

Refactoring

Refactoring

Design

The Design of Everyday Things

Regular Expressions

Mastering Regular Expressions

Ruby on Rails

Agile Web Development With Rails

Django

The Django Book

Everyone Should Read It Because

The Wizard Book 

What have I missed?

Fine print: Where possible, I’ve linked to the author’s homepage for the book. Otherwise I just picked a decent resource. There was no scientific method in the choice of books. I went on reputation and likely answer to “Hey, I want to get good at _____ What book should I get?” There was no scientific method in the choice of categories, though I had a mental notability filter.

Hack Session 2

Andrew Trusty and I sat down for another 3-hour hack session on Thursday. A new version of the idea hat is the result. Our source code is available, of course.

We’re below 12 animal-hours of work on this thing. Not bad!

Here’s the changelist:

• You can make your own independent hats.
• New totally awesome design!
• Submitted ideas are now anonymous.

Known bugs:

• Every homepage access results in a new hat. We intend to have one global, default hat.
• The ideas list superimposes all ideas.
• URLs are not discoverable.

Responsiveness really matters

In my last rant I accused WordPress of bad decision-making in its new UI, because they’ve traded away some responsiveness for more features. I ranted that responsiveness is the most important user-interface quality attribute. 

Some evidence that Google and Amazon agree.

SortHN: My first GreaseMonkey script

SortHN is my first GreaseMonkey script. It reorders the items on the Hacker News front page in an attempt to push better submissions to the top.

Google “Code Snippets” are Cool

From http://www.google.com/search?q=defaultdict:

Notes on “Expert Python Programming”

I bulled through “Expert Python Programming” by Tarek Ziadé this weekend. Mostly I hoped for some tips that might save some time and effort for my 2125 project. Here are some highlights.

 

• Use generators, itertools and coroutines where appropriate.
• Check out the Python Decorator Library.
• There’s a paper about Python’s Method Resolution Order.
• There’s a How-To Guide for Descriptors. Python getter/setters are called descriptors.
• There’s an introduction to metaprogramming in Python.
• Use PyLint to check Python style and CloneDigger to detect duplicate code.
• I’ll want to create a PEP 301 Trove classification for PyVCAL based on the official list of classifiers.
• Python build managers are zc.buildout, Paver, and AutomateIt. I don’t like what I’ve seen of zc.buildout so I’ll have to check out the others.
• Buildbot automates compiles and tests and should be plugged in as a post-commit hook in the version control repository.
• Trac has plugins for different version control systems. I need to look into this.
• Seven writing rules for technical documentation:
  • First write ideas, then focus on style
  • Tailor to the reader
  • Keep it simple
  • Introduce one concept at a time
  • Use realistic code examples
  • Keep it light, but make sure the documentation is sufficient
  • Use templates
• Use Paster for documentation templates
• Sphinx helps to make documentation.
• “Tests are the best place for a developer to learn how software works. They are the use cases the code was primarily created for. Reading them provides a quick and deep insight into how the code works. Sometimes, an example is worth a thousand words. The fact that these tests are always up to date with the codebase makes them the best developer documentation a piece of software can have. Tests don’t go stale in the same way documentation does, otherwise they would fail.”

  Whoa. This suggests two studies: Do developers use tests as documentation to understand a codebase? (intuition: No.) If we encourage them to use tests as documentation, do they understand the codebase faster than developers just using traditional documentation? (intuition: No.) Finally: Real projects often have tests that fail and are ignored. Tests do go stale.

• There’s a Python Testing Tools Taxonomy.
• Minimock is a good mocking library.
• PyMetrics implements some metrics on codebases.
• Eventlet and Twisted are cool concurrent network programming libraries.
• PyDispatch is a multi-consumer multi-producer dispatch mechanism and might be a good way of implementing the Observer pattern.
• A little bit of metaprogramming makes the Visitor pattern better. One example is in the compiler.visitor module.

Search engine optimization, Google and the Deep Web

On January 7, 2008, on an old blog, I wrote:

I noticed some odd behaviour from the Googlebot in my server logs today: It’s using my search feature on key words.

Best guess is that it uses the search as one way of determining my site’s relevance on these keywords.

Clever. SEO tip: If your site has a search feature, make sure it works well on the keywords. Google cares.

Today: How google crawls the deep web and a newly-discovered announcement from April 2008: Crawling through html forms

How well does your site’s search form work for the keywords you care about?