Bayesian Methods for Machine Learning Zoubin Ghahramani * Tutorial Notes Now Available Here * Topic Many topics in Machine Learning (e.g. kernel methods, clustering, semi- supervised learning, feature selection, active learning, reinforcement learning) can be addressed within the framework of Bayesian statistics. While the proportion of work in machine learning based on statistical modelling has grown over the past few years, there remains a good degree of skepticism with respect to taking a fully Bayesian approach. This tutorial aims at introducing fundamantal topics in Bayesian statistics as they apply to machine learning problems, and addressing some misconceptions about Bayesian approaches. The tutorial will also attempt to present a balanced view of the limitations of Bayesian approaches. Finally the tutorial will delve into some of the practical issues in current Bayesian machine learning including the role of approximation algorithms, sampling methods, and nonparameterics. Intended audience The tutorial is intended for the broad ICML community. Little prior knowledge is assumed (other than basic probability theory). The participants will hopefully get both a big picture of Bayesian approaches to machine learning and some insight into specific state-of-the-art methods. Detailed Outline [total 180 minutes, outline subject to revision] 1. Three canonical problems [10 minutes] o Linear Classification o Coin Toss o Clustering with Gaussian Mixtures 2. Foundations [30 minutes] o Representing beliefs and the Cox Axioms o Dutch Book Theorem o Asymptotic Convergence and Consensus o Occam's Razor o Priors: Objective, Subjective, Hierarchical and Empirical Bayes o Exponential Family and Conjugate Priors o How to choose priors? 3. Intractability [10 minutes] o Bayesian inference in Gaussian mixtures and linear classifiers o Hidden variables, parameters and partition functions 4. Approximation Tools [40 minutes] o BIC o Laplace Approximation o Variational Approximations o MCMC o Exact Sampling break 5. Feature Selection, Model Selection and Bayesian Methods [20 minutes] o Do we need to select features? o Automatic Relevance Determination o Model selection criteria and model averaging 6. Bayesian Discriminative Modelling [20 minutes] o Myth: Bayesian methods = Generative models o Bayes Point Machines vs Support Vector Machines o Bayesian Neural Networks 7. From Parametric to Nonparametric Bayes [20 minutes] o Gaussian Processes o Dirichlet Processes and Infinite Mixtures o Other non-parametric Bayesian models 8. Further Topics [15 minutes] o Bayesian Active Learning and Bayesian Decision Theory o Bayesian Semi-supervised Learning o Reconciling Bayesian and Frequentist Views 9. Open Discussion of Limitations and Criticisms [10 minutes] o Philosophical o Practical o Computational 10. Other Questions from the Audience Format The format will be data-projected slides; I will also occasionally use the whiteboard and have some simple Matlab demos to illustrate some ideas. However, the focus won't be on algorithms but rather on concepts. Presenter Zoubin Ghahramani is a Reader in Machine Learning at the Gatsby Unit in London, and an Associate Research Professor at CALD at CMU. He has given tutorials at NIPS, ICANN, the Machine Learning Summer School in Canberra, and various other summer schools. He is interested in Bayesian machine learning, computational approaches to sensorimotor control, and applications of machine learning to bioinformatics. Zoubin Ghahramani Gatsby Computational Neuroscience Unit University College London 17 Queen Square, Room 403 London WC1N 3AR United Kingdom Tel +44 (0)20 7679 1199 Fax +44 (0)20 7679 1173 Email zoubin "AT" gatsby.ucl.ac.uk http://www.gatsby.ucl.ac.uk/~zoubin . Bayesian Methods for Machine Learning Zoubin Ghahramani * Tutorial Notes Now Available Here * Topic Many topics in Machine Learning (e.g. kernel methods, clustering, semi- supervised learning, . Other non-parametric Bayesian models 8. Further Topics [15 minutes] o Bayesian Active Learning and Bayesian Decision Theory o Bayesian Semi-supervised Learning o Reconciling Bayesian and Frequentist. learning, feature selection, active learning, reinforcement learning) can be addressed within the framework of Bayesian statistics. While the proportion of work in machine learning based on statistical