CATALOG DESCRIPTION:  An advanced seminar course in information theory that complements EECS 428 Information Theory. It covers selected topics in classical information theory as well as some recent advances in network information theory, information theoretic aspects of signal processing (e.g., the compressed sensing problem), and the application of information theory in economics. We learn by doing.

INSTRUCTOR: Prof. Dongning Guo; OFFICE: Tech L-393, Phone: 491-3056 OFFICE HOURS: By appointment.

BY COURSES: EECS 422 (required), EECS 428 (preferred)
BY TOPICS: Probability theory, random processes, basic calculus and linear algebra.

Syllabus (tentative, the order of the topics may also change) 

  • Topic 1 Entropy, divergence, typical sequences, and types.
  • Topic 2 Source coding, rate-distortion theory and multiple descriptions.
  • Topic 3 Capacity-cost theory.

The preceding topics overlap with EECS 428 Information Theory substantially. The topics in below are out of the scope of EECS 428.

  • Topic 4 Information-estimation relationships.
  • Topic 5 Directed information.
  • Topic 6 Rational inattention and the application of information theory in economics.
  • Topic 7 Entropy power inequalities.
  • Topic 8 Distributed source coding (the Slepian-Wolf problem).
  • Topic 9 The multiaccess channel.
  • Topic 10 Multiuser detection and CDMA.
  • Topic 11 Compressed sensing.
  • If time permits, we also cover
  • Topic 12 The broadcast channel.
  • Topic 13 The interference channel.



Materials are selected from but not limited to the following texts:

  • Csisz´ar & K¨orner, Information Theory: Coding Theorems for Discrete Memoryless Systems, Academic Press, 1981.
  • G. Kramer, Topics in Multi-User Information Theory, Foundations and Trends in Communications and Information Theory, Vol. 4, Nos. 4–5, 2007.
  • S. Verd´u, Multiuser Detection, Cambridge University Press, 1998.
  • Cover & Thomas, Elements of Information Theory, 2nd ed, Wiley, 2006.
  • R. G. Gallager, Information Theory and Reliable Communication, Wiley, 1968.
  • D. J. MacKay, Information Theory, Inference and Learning Algorithms, Cambridge, 2004

and the following papers:

  • D. Guo, S. Shamai, and S. Verd´u, “Mutual information and minimum mean-square error in Gaussian channels,” IEEE Trans. Inform. Theory, vol. 51, pp. 1261–1282, Apr. 2005.
  • D. Guo and S. Verd´u, “Randomly spread CDMA: Asymptotics via statistical physics,” IEEE Trans. Inform. Theory, vol. 51, pp.
  • 1982–2010, June 2005.
  • C. A. Sims, “Implications of rational inattention,” Journal of Monetary Economics, 50(3): 66590, 2003.
  • C. A. Sims. Rational inattention and monetary economics,” in Handbook of Monetary Policy, 2010.

PROBLEM SETS AND PROJECT: There will be some problem sets. Each student will need to do a research project during the second half of the quarter. Students are encouraged to come up with their own ideas for the project and formulate their own problems. The instructor will discuss his/her project with each student. The project is due by Friday of the final exam week.

EXAMS: There will be a take-home final exam before or during the exam week.


  • Problem sets: 20%
  • Take home final: 30%
  • Project: 50%