This seminar course will expand on the concepts introduced in CSC 535. The primary aim of this course is to explore advanced techniques in probabilistic graphical models (PGMs) and statistical machine learning (ML) more broadly. Students will develop the ability to apply these techniques to their own research. Students will learn to perform statistical inference and reasoning in complex probabilistic statistical models. The course will survey state-of-the-art ML research including: variational inference, advanced Markov chain Monte Carlo sampling, Bayesian nonparametrics, Bayesian optimization, and Bayesian Deep Learning. Upon conclusion of this course students will be capable of developing new methods and advancing the state-of-the-art in ML and PGM research.
D2L: https://d2l.arizona.edu/d2l/home/1205997
Piazza: https://piazza.com/arizona/fall2022/csc696h1
Instructor: Jason Pacheco, GS 724, Email: pachecoj@cs.arizona.edu Office Hours (Zoom): Tuesdays 3-4:30pm, Thursdays 9:00am-10:30am Instructor Homepage: http://www.pachecoj.com
| Date | Topic | Readings | Presenter / Slides |
|---|---|---|---|
| 8/22 | Introduction + Course Overview | (slides) | |
| 8/24 | Probability and Statistics : A Review |
PRML
: Sec. 1.2.1-1.2.4 Optional: Why Isn't Everyone a Bayesian? Efron, B. 1986 Objections to Bayesian Statistics Gelman, A. 2008 Reference: PRML : Sec. 2.1-2.3 |
(slides) |
| 8/29 | Probability and Statistics : Graphical Models |
PRML
: Sec. 8.1-8.3 Optional: WJ : Sec. 2.1 and 2.2 |
(slides) |
| 8/31 | Probability and Statistics : Message Passing Inference |
PRML
: Sec. 8.4 Optional: Factor Graphs and the Sum-Product Algorithm Kschischang, et al. 2001 Reference: Example factor-to-variable message update (Jupyter Notebook) |
(slides) |
| 9/05 | Labor Day : No Classes | ||
| 9/07 | Probability and Statistics : Message Passing Inference (Cont'd) | (slides) | |
| 9/12 | Probability and Statistics : The Exponential Family |
PRML
: Sec. 2.4 Optional: WJ : Sec. 3.1-3.3 |
(slides) |
| 9/14 | Variational Inference |
Variational Inference: A Review for Statisticians Blei, D., et al., J. Am. Stat. Assoc. 2017 Optional: PRML : Sec. 10.1-10.4 |
Eric Duong (slides) |
| 9/19 | Variational Inference : Mean Field Example |
Latent Dirichlet Allocation Blei, D. M., et al. JMLR, 2003 |
Yang Hong (slides) |
| 9/21 | Variational Inference : Stochastic Mean Field |
Stochastic Variational Inference Hoffman, M. D. et al. JMLR, 2013 |
Amir Mohammad Esmaieeli Sikaroudi (slides) |
| 9/26 | Variational Inference : Stochastic Mean Field (continued) |
Project Proposal (slides) |
|
| 9/28 | Variational Inference : Stein Variational |
Stein Variational Gradient Descent Liu, Q. and Wang, D., NeurIPS. 2016 |
Alex Loomis (slides) |
| 10/03 | Monte Carlo Methods |
Introduction to Monte Carlo Methods MacKay, D. J. C . Learning in Graphical Models. Springer, 1998 |
Jason (slides) |
| 10/05 | Monte Carlo Methods (continued) |
Jason (slides) |
|
| 10/10 | Monte Carlo Methods : Hamiltonian Monte Carlo |
MCMC Using Hamiltonian Dynamics Neal, R. M., From: "The Handbook of MCMC.", Chapman & Hall / CRC Press, 2011 Read Sec. 1-4 (inclusive) Optional: A Conceptual Introduction to Hamiltonian Monte Carlo Betancourt, M. arXiv, 2017 |
Maryam Eskandari (slides) (CSC669-1 slides) |
| 10/12 | Early Project Status | Eric, Yang | |
| 10/17 | Monte Carlo Methods : No U-Turn Sampler |
The No-U-Turn sampler: Adaptively Setting Path Lengths in HMC Hoffman, M. D. and Gelman, A. JMLR, 2014 |
Project Status: Amir Lecture: (slides) |
| 10/19 | Early Project Status | Sammi, Alex | |
| 10/24 | Early Project Status | Moyeen, Tuan | |
| 10/26 | Early Project Status | Mary, Shanrui | |
| 10/31 | Implicit Models |
Markov chain Monte Carlo Without Likelihoods Marjoram, P. et al. PNAS, 2003 |
(slides) |
| 11/02 | Implicit Models : Approximate Bayesian Computation |
Approximate Bayesian Computation (ABC) Sunnaker, M. et al. PLoS Computational Biology, 2013 |
Md. Moyeen Uddin (slides) |
| 11/07 | Implicit Models : Neural Likelihood Free Inference |
Sequential Neural Likelihood: Fast Likelihood-Free Inference with Autoregressive Flows Papamakarios, G. et al. AISTATS, 2019 |
Shanrui Zhang (slides) |
| 11/09 | Bayesian Deep Learning |
Recommended / Not Required Hands-on Bayesian Neural Networks – A Tutorial for Deep Learning Users Jospin et al. ArXiv, 2022 Other Resources Blog Post: Joris Baan (2021) YouTube Playlist NeurIPS BDL Workshop |
Jason (slides) |
| 11/14 | Bayesian Deep Learning (Continued) |
Jason (slides) |
|
| 11/16 | Bayesian Deep Learning : Variational Autoencoder |
Auto-encoding Variational Bayes Kingma, D. P. and Welling, M. arXiv, 2013 Optional: An Introduction to Variational Autoencoders Kingma, D. P. and Welling, M. arXiv, 2019 Additional Resources: From Autoencoder to Beta-VAE Weng, L. Github, 2018 |
Tuan Nguyen (slides) |
| 11/21 | Bayesian Deep Learning : Monte Carlo Dropout |
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning Gal, Y. and Ghahramani, Z. ICML, 2016 |
Sammi Abida Salma (slides) |
| 11/23 | Gaussian Processes |
CH 2 - through 2.2 (inclusive) Gaussian Processes for Machine Learning Rasmussen, C. MIT Press, 2006 |
(slides) |
| 11/28 | Bayesian Optimization |
Taking the Human Out of the Loop: A Review of Bayesian Optimization Shahriari, B. et al. Proceedings of the IEEE, 2015 |
Project Report (Due: 12/14) Report Instructions Slides from: Adams, R. NeurIPS. 2017 |
| 11/30 | Course Wrap-Up |
First Tuan will give his ~15min project presentation Second I will present: How to write a good CVPR submission Bill Freeman. MIT CSAIL. 2014 |
Tuan |
| 12/05 | Project Presentations | Yang, Amir, Shanrui, Sammi | |
| 12/07 | Project Presentations | Moyeen, Eric, Alex, Mary |