Information Theory

This course is an introduction to information theory, which emphasizes fundamental concepts as well as analytical techniques. Specific topics include: Information Measures, The I-Measure, Zero-Error Data Compression, Weak Typicality, Strong Typicality, Discrete Memoryless Channels, Rate-Distortion Theory, The Blahut-Arimoto Algorithms, Differential Entropy, Continuous-Valued Channels.

Prof. Raymond W. Yeung
Choh-Ming Li Professor of Information Engineering

About the Course

The lectures are based on the first 11 chapters of Prof. Raymond Yeung’s textbook entitled Information Theory and Network Coding (Springer 2008).

This book and its predecessor, A First Course in Information Theory (Kluwer 2002, essentially the first edition of the 2008 book), have been adopted by over 80 universities around the world. The electronic version of the 2008 book can be downloaded for free (with institutional subscription).

At the completion of this course, the student should be able to:

  1. Demonstrate knowledge and understanding of the fundamentals of information theory.
  2. Appreciate the notion of fundamental limits in communication systems and more generally all systems.
  3. Develop deeper understanding of communication systems.
  4. Apply the concepts of information theory to various disciplines in information science.

Course Syllabus

The lectures are based on the first 11 chapters of Prof. Raymond Yeung’s textbook entitled Information Theory and Network Coding (Springer 2008) listed above.

  • Chapter 1 The Science of Information
  • Chapter 2 Information Measures
  • Chapter 3 The I-Measure
  • Chapter 4 Zero-Error Data Compression
  • Chapter 5 Weak Typicality
  • Chapter 6 Strong Typicality
  • Chapter 7 Discrete Memoryless Channels
  • Chapter 8 Rate-Distortion Theory
  • Chapter 9 The Blahut-Arimoto Algorithms
  • Chapter 10 Differential Entropy
  • Chapter 11 Continuous-Valued Channels

Reference

Other than the textbook for this course, an excellent reference is Elements of Information Theory, 2nd Edition by T. M. Cover and J. A. Thomas (Wiley 2006).

Format

15 weeks of study

6-8 hours/week of work

English

There will be 15 weeks of lectures, each composed of video clips with a total length equal to about 50 minutes (equivalent to about 2 to 3 hours of classroom teaching).

Prerequisite

A solid course in probability at the undergraduate level would suffice.