Description of Course

This seminar course will expand on the concepts introduced in CSC 535. The primary aim of this course is to explore advanced techniques in probabilistic graphical models (PGMs) and statistical machine learning (ML) more broadly. Students will develop the ability to apply these techniques to their own research. Students will learn to perform statistical inference and reasoning in complex probabilistic statistical models. The course will survey state-of-the-art ML research including: variational inference, advanced Markov chain Monte Carlo sampling, Bayesian nonparametrics, Bayesian optimization, and Bayesian Deep Learning. Upon conclusion of this course students will be capable of developing new methods and advancing the state-of-the-art in ML and PGM research.

Primary Resources

Multiple readings will be chosen from the texts below. I have listed the references and will refer to them by acronym in the schedule.
  • PRML : Bishop, C. "Pattern Recognition and Machine Learning." Springer, 2006
  • WJ : Wainwright, M. J., and Jordan, M. I. "Graphical models, exponential families, and variational inference." Foundations and Trends in Machine Learning, 2008

Course Management

D2L: https://d2l.arizona.edu/d2l/home/1205997
Piazza: https://piazza.com/arizona/fall2022/csc696h1

Instructor and Contact Information:

Instructor: Jason Pacheco, GS 724, Email: pachecoj@cs.arizona.edu
Office Hours (Zoom): Tuesdays 3-4:30pm, Thursdays 9:00am-10:30am
Instructor Homepage: http://www.pachecoj.com

Date Topic Readings Presenter / Slides
8/22 Introduction + Course Overview (slides)
8/24 Probability and Statistics : A Review PRML : Sec. 1.2.1-1.2.4

Optional:
Why Isn't Everyone a Bayesian?
Efron, B. 1986
Objections to Bayesian Statistics
Gelman, A. 2008

Reference:
PRML : Sec. 2.1-2.3
(slides)
8/29 Probability and Statistics : Graphical Models PRML : Sec. 8.1-8.3

Optional:
WJ : Sec. 2.1 and 2.2
(slides)
8/31 Probability and Statistics : Message Passing Inference PRML : Sec. 8.4

Optional:
Factor Graphs and the Sum-Product Algorithm
Kschischang, et al. 2001

Reference:
Example factor-to-variable message update (Jupyter Notebook)
(slides)
9/05 Labor Day : No Classes
9/07 Probability and Statistics : Message Passing Inference (Cont'd) (slides)
9/12 Probability and Statistics : The Exponential Family PRML : Sec. 2.4

Optional:
WJ : Sec. 3.1-3.3
(slides)
9/14 Variational Inference Variational Inference: A Review for Statisticians
Blei, D., et al., J. Am. Stat. Assoc. 2017

Optional:
PRML : Sec. 10.1-10.4
Eric Duong
(slides)
9/19 Variational Inference : Mean Field Example Latent Dirichlet Allocation
Blei, D. M., et al. JMLR, 2003
Yang Hong
(slides)
9/21 Variational Inference : Stochastic Mean Field Stochastic Variational Inference
Hoffman, M. D. et al. JMLR, 2013
Amir Mohammad Esmaieeli Sikaroudi
(slides)
9/26 Variational Inference : Stochastic Mean Field (continued) Project Proposal
(slides)
9/28 Variational Inference : Stein Variational Stein Variational Gradient Descent
Liu, Q. and Wang, D., NeurIPS. 2016
Alex Loomis
(slides)
10/03 Monte Carlo Methods Introduction to Monte Carlo Methods
MacKay, D. J. C . Learning in Graphical Models. Springer, 1998
Jason
(slides)
10/05 Monte Carlo Methods (continued) Jason
(slides)
10/10 Monte Carlo Methods : Hamiltonian Monte Carlo MCMC Using Hamiltonian Dynamics
Neal, R. M., From: "The Handbook of MCMC.", Chapman & Hall / CRC Press, 2011
Read Sec. 1-4 (inclusive)
Maryam Eskandari
10/12 Early Project Status (Phase I)
10/17 Monte Carlo Methods : No U-Turn Sampler The No-U-Turn sampler: Adaptively Setting Path Lengths in HMC
Hoffman, M. D. and Gelman, A. JMLR, 2014
10/19 Early Project Status (Phase II)
10/24 Implicit Models Markov chain Monte Carlo Without Likelihoods
Marjoram, P. et al. PNAS, 2003
Jason
10/26 Implicit Models : Approximate Bayesian Computation Approximate Bayesian Computation (ABC)
Sunnaker, M. et al. PLoS Computational Biology, 2013
Md. Moyeen Uddin
10/31 Implicit Models : Neural Likelihood Free Inference Sequential Neural Likelihood: Fast Likelihood-Free Inference with Autoregressive Flows
Papamakarios, G. et al. AISTATS, 2019
Shanrui Zhang
11/02 Bayesian Deep Learning Jason
11/07 Bayesian Deep Learning : Variational Autoencoder Auto-encoding Variational Bayes
Kingma, D. P. and Welling, M. arXiv, 2013
Tuan Nguyen
11/09 Bayesian Deep Learning : Dropout Monte Carlo Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
Gal, Y. and Ghahramani, Z. ICML, 2016
Sammi Abida Salma
11/14 Bayesian Deep Learning : TBD
11/16
11/21 Gaussian Processes CH 2 - 2.1.3 and 2.1.5 (inclusive)
Gaussian Processes for Machine Learning
Rasmussen, C. MIT Press, 2006

11/23 Bayesian Optimization Taking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B. et al. Proceedings of the IEEE, 2015
11/28 Bayesian Optimization (Continued) TBD
11/30 Course Wrap-up
12/05 Project Presentations
12/07 Project Presentations

© Jason Pacheco, 2020