October 11th NP Bayes Meetup

Posted on October 11, 2012 by evan

In today’s NP Bayes discussion group we returned to the 2006 Hierarchical Dirichlet Process (HDP) paper by Teh et al to discuss sampling-based inference. We spent most of our time sorting through the notational soup needed to specify the HDP variables and their relationships between one another. This led to a brief discussion of implementation issues, and finally a description of the three Gibbs sampling techniques presented in the paper.

NP Bayes reading group (9/27): hierarchical DPs

Posted on October 3, 2012 by jpillow

Our second NPB reading group meeting took aim at the seminal 2006 paper (with >1000 citations!) by Teh, Jordan, Beal & Blei on Hierarchical Dirichlet Processes. We were joined by newcomers Piyush Rai (newly arrived SSC postdoc), and Ph.D. students Dan Garrette (CS) and Liang Sun (mathematics), both of whom have experience with natural language models.

We established a few basic properties of the hierarchical DP, such as the the fact that it involves creating dependencies between DPs by endowing them with a common base measure, which is itself sampled from a DP. That is:

$G_0 \sim DP(\gamma, H)$ (“global measure” sampled from DP with base measure $H$ and concentration $\gamma$ ).
$G_j|\alpha_0,G_0 \sim DP(\alpha_0,G_0)$ (sequence of conditionally independent random measures with common base measure $G_0$ , e.g., $G_j$ are distributions over clusters from data collected on different days)

Beyond this, we got bogged down in confusion over metaphors and interpretations, unclear whether $G_j$ ‘s were topics or documents or tables or restaurants or ethnicities, and were hampered by having two different version of the manuscript floating around with different page numbers and figures.
This week: we’ll take up where we left off, focusing on Section 4 (“Hierarchical Dicirhlet Processes”) with discussion led by Piyush. We’ll agree to show up with the same (“official journal”) version of the manuscript, available: here.

Time: 4:00 PM, Thursday, Oct 4.
Location: SEA 5.106
Please email pillow AT mail.utexas.edu if you’d like to be added to the announcement list.

Revivifying the NP Bayes Reading Group

Posted on September 23, 2012 by jpillow

After a nearly 1-year hiatus, we’ve restarted our reading group on non-parametric (NP) Bayesian methods, focused on models for discrete data based on generalizations of the Dirichlet and other stick-breaking processes.

Thursday (9/20) was our first meeting, and Karin led a discussion of:

Teh, Y. W. (2006). A hierarchical Bayesian language model based on Pitman-Yor
processes. Proceedings of the 21st International Conference on
Computational Linguistics and the 44th annual meeting of the
Association for Computational Linguistics. 985-992

In the first meeting, we made it only as far as describing the Pitman-Yor (PY) process, a stochastic process whose samples are random probability distributions, and two methods for sampling from it:

Chinese Restaurant sampling (aka “Blackwell-MacQueen urn scheme”), which directly provides samples $\{X_i\}$ from distribution $G \sim PY$ with G marginalized out.
Stick-breaking, which samples the distribution $G = \sum \pi_i \delta_{\phi_i}$ explicitly, using iid draws of Beta random variables to obtain stick weights $\pi_i$ .

We briefly discussed the intuition for the hierarchical PY process, which uses PY process as base measure for PY process priors at deeper levels of the hierarchy (applied here to develop an n-gram model for natural language).

Next week: We’ve decided to go a bit further back in time to read:

Teh, Y. W.; Jordan, M. I.; Beal, M. J. & Blei, D. M. (2006). Hierarchical dirichlet processes. Journal of the American Statistical Association 101:1566-1581.

Time: Thursday (9/27), 4:00pm.
Location: Pillow lab
Presenter: Karin

note: if you’d like to be added to the email announcement list for this group, please send email to pillow AT mail.utexas.edu.

NP Bayes Reading Group: 1st meeting

Posted on June 17, 2011 by jpillow

We’ve started a reading group to come to grips with some of the recent developments in non-parametric (NP) Bayesian modeling, in particular, hierarchical Bayesian models for discrete data. The defining characteristic of NP models are that the number of parameters scale with the amount of data (leading to an infinite number of parameters in the limit of infinite data). Although these have sparked a mini-revolution in cognitive psychology (e.g., Tenenbaum, Griffiths & Kemp 2006), they do not appear to have found much application to statistical analysis of neural data (with the exception of spike sorting — see, e.g. Wood & Black 2008).

Our first assignment is to go through the slides from Michael Jordan’s 2005 NIPS tutorial (slides.ps). Last week we began, and made it through slide #23, covering the basic ideas of non-parametric models, exchangeability, De Finetti’s theorem, conjugate priors, Gibbs sampling, graphical models, Dirichlet & Beta distributions.

A few issues set aside for further exploration: