Primary Resources

  • WJ : Wainwright, M. J., and Jordan, M. I. "Graphical models, exponential families, and variational inference." Foundations and Trends in Machine Learning, 2008
  • DB : Barber, D. "Bayesian Reasoning and Machine Learning." Cambridge University Press, 2012
  • PRML : Bishop, C. "Pattern Recognition and Machine Learning." Springer, 2006
  • JP : Pacheco, J. L. "Variational Approximations with Diverse Applications." Brown PhD Thesis, 2016
  • ES : Sudderth, E. B. "Graphical Models for Visual Object Recognition and Tracking." MIT PhD Thesis, 2006


Date Topic Assigned Reading Presenter Additional Resources
8/26 Introduction + Course Overview Thirteen Rules for Giving a Really Bad Talk
Efron, B. 2013
Jason Pacheco
8/28 Bayesian Inference Jason Pacheco
Why Isn't Everyone a Bayesian?
Efron, B. 1986
Objections to Bayesian Statistics
Gelman, A. 2008
9/2 Labor Day | No Class
9/4 Graphical Models WJ - Chapter 2 Jason Pacheco
JP - Sec. 2.1
ES - Sec. 2.2
9/9 Exponential Families WJ - Chapter 3 Jason Pacheco
JP - Sec. 2.2
ES - Sec. 2.1
PRML - Sec. 2.4
9/11 Variational Inference (VI) PRML - Sec. 10.1-10.4 Jason Pacheco
JP - Sec. 2.3
ES - Sec. 2.3
DB - Sec. 28.1-28.4

Variational Inference: A Review for Statisticians
Blei, D., et al., J. Am. Stat. Assoc. 2017
9/16 VI: Mean Field Variational Methods Variational Message Passing
Winn, J. and Bishop, C. JMLR, 2005
Simon Swenson
ES - Sec. 2.3.1
PRML - Sec. 10.1-10.6
WJ - Ch. 5

An Introduction to Variational Methods for Graphical Models
Jordan, M. I., et al. Machine Learning, 1999
Tutorial Variational Bayes and Beyond: "Bayesian inference for big data"
Broderick, T. 2019
9/18 VI: Mean Field Example Latent Dirichlet Allocation
Blei, D. M., et al. JMLR, 2003
Marium Yousuf
9/23 Research Roundtable (Part 1) Marina, Kairong
9/25 VI: Stochastic Variational Stochastic Variational Inference Hoffman, M. D. et al. JMLR, 2013 Kairong Jiang
9/30 Research Roundtable (Part 2) Marium, Loren, Simon
10/2 VI: Loopy Belief Propagation & Bethe Variational Methods Understanding Belief Propagation and its Generalizations
Yedidia, J. S., et al., IJCAI, 2001
Marina Kisley (notes) WJ - Sec. 4.1
ES - Sec. 2.3.2
JP - Sec. 2.3.1-2.3.2

Constructing Free Energy Approximations and Generalized Belief Propagation Algorithms
Yedidia, J. et al., IEEE Info. Theory, 2005
10/7 Research Roundtable (Part 2.5)
Open Topics Discussion: Plan to discuss any changes to scheduled topics
10/9 VI: Expectation Propagation PRML - Sec. 10.7
Jason Pacheco
Expectation Propagation Summary
Sudderth, E., MIT 6.975 Course Notes, 2002
Expectation Propagation for Approximate Bayesian Inference
Minka, T. P. UAI, 2001
10/14 Markov chain Monte Carlo (MCMC): Introduction Introduction to Monte Carlo Methods
MacKay, D. J. C . Learning in Graphical Models. Springer, 1998
Lecture: Jason Pacheco
10/16 MCMC: Hamiltonian Monte Carlo
Term Project Proposals: Due Fri 10/18
Details on the Resources page.
MCMC Using Hamiltonian Dynamics
Neal, R. M., From: "The Handbook of MCMC.", Chapman & Hall / CRC Press, 2011
Read Sec. 1-4 (inclusive)
Kairong Jiang
10/16 MCMC: Hamiltonian Monte Carlo (Cont'd) Typical Sets / Volume Preservation Jason Pacheco
A Conceptual Introduction to Hamiltonian Monte Carlo
Betancourt, M. arXiv, 2017
10/23 MCMC: No U-turn Sampler The No-U-Turn sampler: Adaptively Setting Path Lengths in HMC
Hoffman, M. D. and Gelman, A. JMLR, 2014
Simon Swenson
10/28 MCMC: Reversible Jump MCMC RJMCMC Computation and Bayesian Model Determination
Green, P. J. Biometrika, 1995
Loren Champlin
On Bayesian Analysis of Mixtures with an Unknown Number of Components
Richardson, S. and Green, P. J. J. R. Stat. Soc., 1997
Minus attached discussion
Reversible Jump MCMC
Green, P. J. and Hastie, D. I. Genetics, 2009
Trans-dimensional markov chain monte carlo
Green, P. J. Oxford Stat. Sci. Ser., 2003
10/30 Dynamical Systems: Introduction Lecture: Jason Pacheco
11/4 Dynamical Systems: Introduction (continued) Lecture: Jason Pacheco
11/6 DS: Particle Filter An Overview of Existing Methods and Recent Advances in Sequential Monte Carlo
Cappé, O. et al. Proc. IEEE. 2007
Read Sec. 1 & 2 (Inclusive)
Lecture: Jason Pacheco A Tutorial on Particle Filtering and Smoothing: Fifteen years later
Doucet, A. and Johansen, A. M. Handbook of Nonlinear Filtering. 2009
11/11 Veteran's Day | No Class
11/13 DS: Particle Filters (Cont'd) No Assigned Readig Lecture: Jason Pacheco
11/18 DS: HMM Topic Models Integrating Topics and Syntax
Griffiths, T. L. et al. NeurIPS (2005)
11/20 DS: Infinite HMM The Infinite Hidden Markov Model
Beal, M. et al. NeurIPS (2002)
Stochastic Variational Inference for Bayesian Time Series Models
Johnson, M. J. and Willsky, A. S. ICML (2014)
11/25 Bayesian Deep Learning: Variational Autoencoders Auto-encoding Variational Bayes Kingma, D. P. and Welling, M. ICLR, 2014 Marina Kisley Tutorial on Variational Autoencoders
Doersch, C. ArXiv, 2016
Stochastic Backpropagation and Approximate Inference in Deep Generative Models
Rezende, et al. ICML, 2014
11/27 BDL: Structured VAEs Composing graphical models with neural networks for structured representations and fast inference
Johnson, M. J., et al. NIPS, 2016
Jason Pacheco
12/02 BDL: Dropout Sampling Dropout as a Bayesian Approximation:Representing Model Uncertainty in Deep Learning
Gal, Y. and Ghahramani, Z. ICML, 2016
Loren Champlin Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Srivastava, et al. JMLR, 2014
Variational Dropout and the Local Reparameterization Trick
Kingma, et al. NIPS, 2015
Variational Dropout is not Bayesian
Hron, J. et al. ArXiv, 2017
12/04 BDL: Deep Exponential Families Deep Exponential Families
Ranganath, R. et al. AISTATS (2015)
Jason Pacheco
12/04 Course Wrap-Up / Future Directions Lecture: Jason Pacheco Last Regular Class!
12/9 Term Project Presentations (Group 1)
12/11 Term Project Presentations (Group 2)
Final Project Reports Due

© Jason Pacheco, 2019