This seminar course will expand on the concepts introduced in CSC 535. The primary aim of this course is to explore advanced techniques in probabilistic graphical models (PGMs) and statistical machine learning (ML) more broadly. Students will develop the ability to apply these techniques to their own research. Students will learn to perform statistical inference and reasoning in complex probabilistic statistical models. The course will survey stateoftheart ML research including: variational inference, advanced Markov chain Monte Carlo sampling, Bayesian nonparametrics, Bayesian optimization, and Bayesian Deep Learning. Upon conclusion of this course students will be capable of developing new methods and advancing the stateoftheart in ML and PGM research.
D2L: https://d2l.arizona.edu/d2l/home/1205997
Piazza: https://piazza.com/arizona/fall2022/csc696h1
Instructor: Jason Pacheco, GS 724, Email: pachecoj@cs.arizona.edu
Office Hours (Zoom): Tuesdays 34:30pm, Thursdays 9:00am10:30am
Instructor Homepage: http://www.pachecoj.com
Date  Topic  Readings  Presenter / Slides 

8/22  Introduction + Course Overview  (slides)  
8/24  Probability and Statistics : A Review 
PRML
: Sec. 1.2.11.2.4 Optional: Why Isn't Everyone a Bayesian? Efron, B. 1986 Objections to Bayesian Statistics Gelman, A. 2008 Reference: PRML : Sec. 2.12.3 
(slides) 
8/29  Probability and Statistics : Graphical Models 
PRML
: Sec. 8.18.3 Optional: WJ : Sec. 2.1 and 2.2 
(slides) 
8/31  Probability and Statistics : Message Passing Inference 
PRML
: Sec. 8.4 Optional: Factor Graphs and the SumProduct Algorithm Kschischang, et al. 2001 Reference: Example factortovariable message update (Jupyter Notebook) 
(slides) 
9/05  Labor Day : No Classes  
9/07  Probability and Statistics : Message Passing Inference (Cont'd)  (slides)  
9/12  Probability and Statistics : The Exponential Family 
PRML
: Sec. 2.4 Optional: WJ : Sec. 3.13.3 
(slides) 
9/14  Variational Inference 
Variational Inference: A Review for Statisticians Blei, D., et al., J. Am. Stat. Assoc. 2017 Optional: PRML : Sec. 10.110.4 
Eric Duong (slides) 
9/19  Variational Inference : Mean Field Example 
Latent Dirichlet Allocation Blei, D. M., et al. JMLR, 2003 
Yang Hong (slides) 
9/21  Variational Inference : Stochastic Mean Field 
Stochastic Variational Inference Hoffman, M. D. et al. JMLR, 2013 
Amir Mohammad Esmaieeli Sikaroudi (slides) 
9/26  Variational Inference : Stochastic Mean Field (continued) 
Project Proposal (slides) 

9/28  Variational Inference : Stein Variational 
Stein Variational Gradient Descent Liu, Q. and Wang, D., NeurIPS. 2016 
Alex Loomis (slides) 
10/03  Monte Carlo Methods 
Introduction to Monte Carlo Methods MacKay, D. J. C . Learning in Graphical Models. Springer, 1998 
Jason (slides) 
10/05  Monte Carlo Methods (continued) 
Jason (slides) 

10/10  Monte Carlo Methods : Hamiltonian Monte Carlo 
MCMC Using Hamiltonian Dynamics Neal, R. M., From: "The Handbook of MCMC.", Chapman & Hall / CRC Press, 2011 Read Sec. 14 (inclusive) Optional: A Conceptual Introduction to Hamiltonian Monte Carlo Betancourt, M. arXiv, 2017 
Maryam Eskandari (slides) (CSC6691 slides) 
10/12  Early Project Status  Eric, Yang  
10/17  Monte Carlo Methods : No UTurn Sampler 
The NoUTurn sampler: Adaptively Setting Path Lengths in HMC Hoffman, M. D. and Gelman, A. JMLR, 2014 
Project Status: Amir Lecture: (slides) 
10/19  Early Project Status  Sammi, Alex  
10/24  Early Project Status  Moyeen, Tuan  
10/26  Early Project Status  Mary, Shanrui  
10/31  Implicit Models 
Markov chain Monte Carlo Without Likelihoods Marjoram, P. et al. PNAS, 2003 
(slides) 
11/02  Implicit Models : Approximate Bayesian Computation 
Approximate Bayesian Computation (ABC) Sunnaker, M. et al. PLoS Computational Biology, 2013 
Md. Moyeen Uddin (slides) 
11/07  Implicit Models : Neural Likelihood Free Inference 
Sequential Neural Likelihood: Fast LikelihoodFree Inference with Autoregressive Flows Papamakarios, G. et al. AISTATS, 2019 
Shanrui Zhang (slides) 
11/09  Bayesian Deep Learning 
Recommended / Not Required Handson Bayesian Neural Networks – A Tutorial for Deep Learning Users Jospin et al. ArXiv, 2022 Other Resources Blog Post: Joris Baan (2021) YouTube Playlist NeurIPS BDL Workshop 
Jason (slides) 
11/14  Bayesian Deep Learning (Continued) 
Jason (slides) 

11/16  Bayesian Deep Learning : Variational Autoencoder 
Autoencoding Variational Bayes Kingma, D. P. and Welling, M. arXiv, 2013 Optional: An Introduction to Variational Autoencoders Kingma, D. P. and Welling, M. arXiv, 2019 Additional Resources: From Autoencoder to BetaVAE Weng, L. Github, 2018 
Tuan Nguyen (slides) 
11/21  Bayesian Deep Learning : Monte Carlo Dropout 
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning Gal, Y. and Ghahramani, Z. ICML, 2016 
Sammi Abida Salma (slides) 
11/23  Gaussian Processes 
CH 2  through 2.2 (inclusive) Gaussian Processes for Machine Learning Rasmussen, C. MIT Press, 2006 
(slides) 
11/28  Bayesian Optimization 
Taking the Human Out of the Loop: A Review of Bayesian Optimization Shahriari, B. et al. Proceedings of the IEEE, 2015 
Project Report (Due: 12/14) Report Instructions Slides from: Adams, R. NeurIPS. 2017 
11/30  Course WrapUp 
First Tuan will give his ~15min project presentation Second I will present: How to write a good CVPR submission Bill Freeman. MIT CSAIL. 2014 
Tuan 
12/05  Project Presentations  Yang, Amir, Shanrui, Sammi  
12/07  Project Presentations  Moyeen, Eric, Alex, Mary 