Description of Course

This seminar course will expand on the concepts introduced in CSC 535. The primary aim of this course is to explore advanced techniques in probabilistic graphical models (PGMs) and statistical machine learning (ML) more broadly. Students will develop the ability to apply these techniques to their own research. Students will learn to perform statistical inference and reasoning in complex probabilistic statistical models. The course will survey state-of-the-art ML research including: variational inference, advanced Markov chain Monte Carlo sampling, Bayesian nonparametrics, Bayesian optimization, and Bayesian Deep Learning. Upon conclusion of this course students will be capable of developing new methods and advancing the state-of-the-art in ML and PGM research.

Primary Resources

Multiple readings will be chosen from the texts below. I have listed the references and will refer to them by acronym in the schedule.
  • PRML : Bishop, C. "Pattern Recognition and Machine Learning." Springer, 2006
  • WJ : Wainwright, M. J., and Jordan, M. I. "Graphical models, exponential families, and variational inference." Foundations and Trends in Machine Learning, 2008

Course Management


Instructor and Contact Information:

Instructor: Jason Pacheco, GS 724, Email:
Office Hours (Zoom): Tuesdays 3-4:30pm, Thursdays 9:00am-10:30am
Instructor Homepage:

Date Topic Readings Presenter / Slides
8/22 Introduction + Course Overview (slides)
8/24 Probability and Statistics : A Review PRML : Sec. 1.2.1-1.2.4

Why Isn't Everyone a Bayesian?
Efron, B. 1986
Objections to Bayesian Statistics
Gelman, A. 2008

PRML : Sec. 2.1-2.3
8/29 Probability and Statistics : Graphical Models PRML : Sec. 8.1-8.3

WJ : Sec. 2.1 and 2.2
8/31 Probability and Statistics : Message Passing Inference PRML : Sec. 8.4

Factor Graphs and the Sum-Product Algorithm
Kschischang, et al. 2001

Example factor-to-variable message update (Jupyter Notebook)
9/05 Labor Day : No Classes
9/07 Probability and Statistics : Message Passing Inference (Cont'd) (slides)
9/12 Probability and Statistics : The Exponential Family PRML : Sec. 2.4

WJ : Sec. 3.1-3.3
9/14 Variational Inference Variational Inference: A Review for Statisticians
Blei, D., et al., J. Am. Stat. Assoc. 2017

PRML : Sec. 10.1-10.4
Eric Duong
9/19 Variational Inference : Mean Field Example Latent Dirichlet Allocation
Blei, D. M., et al. JMLR, 2003
Yang Hong
9/21 Variational Inference : Stochastic Mean Field Stochastic Variational Inference
Hoffman, M. D. et al. JMLR, 2013
Amir Mohammad Esmaieeli Sikaroudi
9/26 Variational Inference : Stochastic Mean Field (continued) Project Proposal
9/28 Variational Inference : Stein Variational Stein Variational Gradient Descent
Liu, Q. and Wang, D., NeurIPS. 2016
Alex Loomis
10/03 Monte Carlo Methods Introduction to Monte Carlo Methods
MacKay, D. J. C . Learning in Graphical Models. Springer, 1998
10/05 Monte Carlo Methods (continued) Jason
10/10 Monte Carlo Methods : Hamiltonian Monte Carlo MCMC Using Hamiltonian Dynamics
Neal, R. M., From: "The Handbook of MCMC.", Chapman & Hall / CRC Press, 2011
Read Sec. 1-4 (inclusive)

A Conceptual Introduction to Hamiltonian Monte Carlo
Betancourt, M. arXiv, 2017
Maryam Eskandari
(CSC669-1 slides)
10/12 Early Project Status Eric, Yang
10/17 Monte Carlo Methods : No U-Turn Sampler The No-U-Turn sampler: Adaptively Setting Path Lengths in HMC
Hoffman, M. D. and Gelman, A. JMLR, 2014
Project Status: Amir
Lecture: (slides)
10/19 Early Project Status Sammi, Alex
10/24 Early Project Status Moyeen, Tuan
10/26 Early Project Status Mary, Shanrui
10/31 Implicit Models Markov chain Monte Carlo Without Likelihoods
Marjoram, P. et al. PNAS, 2003
11/02 Implicit Models : Approximate Bayesian Computation Approximate Bayesian Computation (ABC)
Sunnaker, M. et al. PLoS Computational Biology, 2013
Md. Moyeen Uddin
11/07 Implicit Models : Neural Likelihood Free Inference Sequential Neural Likelihood: Fast Likelihood-Free Inference with Autoregressive Flows
Papamakarios, G. et al. AISTATS, 2019
Shanrui Zhang
11/09 Bayesian Deep Learning Recommended / Not Required
Hands-on Bayesian Neural Networks – A Tutorial for Deep Learning Users Jospin et al. ArXiv, 2022

Other Resources
Blog Post: Joris Baan (2021)
YouTube Playlist
NeurIPS BDL Workshop
11/14 Bayesian Deep Learning (Continued) Jason
11/16 Bayesian Deep Learning : Variational Autoencoder Auto-encoding Variational Bayes
Kingma, D. P. and Welling, M. arXiv, 2013

An Introduction to Variational Autoencoders
Kingma, D. P. and Welling, M. arXiv, 2019

Additional Resources:
From Autoencoder to Beta-VAE
Weng, L. Github, 2018
Tuan Nguyen
11/21 Bayesian Deep Learning : Monte Carlo Dropout Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
Gal, Y. and Ghahramani, Z. ICML, 2016
Sammi Abida Salma
11/23 Gaussian Processes CH 2 - through 2.2 (inclusive)
Gaussian Processes for Machine Learning
Rasmussen, C. MIT Press, 2006

11/28 Bayesian Optimization Taking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B. et al. Proceedings of the IEEE, 2015
Project Report (Due: 12/14)
Report Instructions

Slides from: Adams, R. NeurIPS. 2017
11/30 Course Wrap-Up First Tuan will give his ~15min project presentation
Second I will present:
      How to write a good CVPR submission
      Bill Freeman. MIT CSAIL. 2014
12/05 Project Presentations Yang, Amir, Shanrui, Sammi
12/07 Project Presentations Moyeen, Eric, Alex, Mary

© Jason Pacheco, 2022