Notes on “A Mathematical Theory of Communication”

October 6, 2008

A Mathematical Theory of Communication: C. E. Shannon

The gist: This paper introduces information theory. This includes how we measure information, including the use of bits; the relationship of information, uncertainty, probability, statistics and Markoff processes; compression and redundancy; and error correction. Shannon demonstrates the importance and utility of the entropy equation symbolized by H and makes numerous proofs around it.

The good: This paper is crisp, novel and well-written. It is grounded in clear mathematics. It founded a whole new area of research! It deserves its place among the most famous papers of the 20th century.

The bad: This paper was not easy to read. I’m not sure what kind of intellectual foundation would be needed to properly understand and digest it. Whatever background is needed, I lack it. Brevity does not maximize comprehension. I was relieved to reach the beginning of the appendices.

Advertisements

4 Responses to “Notes on “A Mathematical Theory of Communication””

  1. Ian Says:

    Wow – that is quite the paper – it looks like the basis for at least 1 of my 4th year EE Telecommunications courses – either introduction to tele system or digital telecom systems, maybe both! The scary part is I recognize most of the terminology and math, and even scarier, I have used some of that in the last few years. You’re welcome to borrow the texts if you need more bedtime reading material :).

  2. George Says:

    I have a great Information Theory Text I have used a bit if you want to take a gander at it.

    You should read about Semantic Hashing next!

    Quote from the paper:
    “So, assuming we already have the 30-bit code for the query document, the time required to create this long shortlist is zero, which compares favorably with other methods.”

    Priceless.

    http://www.cs.utoronto.ca/~hinton/absps/sh.pdf

  3. George Says:

    I have Cover and Thomas’s Elements of Information Theory if you want to take a look at it. Great book.

    You should read Russ’s semantic hashing paper next!
    Priceless quote from the paper:
    “So, assuming we already have the 30-bit code for the query document, the time required to create this long shortlist is zero, which compares favorably with other methods.”

    http://www.cs.toronto.edu/~rsalakhu/papers/semantic_final.pdf

  4. George Says:

    Better pictures in these slides than in the paper I linked to.
    http://www.cs.toronto.edu/~rsalakhu/talks/ipam_dbn.pdf


Comments are closed.

%d bloggers like this: