Continuing from last week, we discussed the formulation of generative clustering (mixture model) with fixed number of clusters K using Dirichlet distribution as a prior for cluster size distribution following Jordan’s slides. The definition of Dirichlet process (DP) and its existence was briefly shown via Kolmogorov extension theorem. Following (Sethuraman, 1994), we discussed the stick breaking construction of DP. Stick breaking provides the sample-biased permutation of Poisson-Dirichlet distribution obtained by Kingman limit (Kingman, 1975). The following fun facts about (extended) Dirichlet distribution are from (Sethuraman, 1994).
Fun Fact 1 Let
be a n-dimensional vector consisting of 0’s, except having 1 at j-th index.
Then,
. Then,![\displaystyle \begin{array}{rcl} \sum_i \gamma_i Dir([\alpha \gamma_1, \ldots, \alpha \gamma_n] + e_j) &= Dir(\alpha \gamma_1, \ldots, \alpha \gamma_n). \end{array}](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle+%5Cbegin%7Barray%7D%7Brcl%7D+%5Csum_i+%5Cgamma_i+Dir%28%5B%5Calpha+%5Cgamma_1%2C+%5Cldots%2C+%5Calpha+%5Cgamma_n%5D+%2B+e_j%29+%26%3D+Dir%28%5Calpha+%5Cgamma_1%2C+%5Cldots%2C+%5Calpha+%5Cgamma_n%29.+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0 "\displaystyle \begin{array}{rcl} \sum_i \gamma_i Dir([\alpha \gamma_1, \ldots, \alpha \gamma_n] + e_j) &= Dir(\alpha \gamma_1, \ldots, \alpha \gamma_n). \end{array}")
Next week, we will continue on the discussion of DP as a prior for nonparameteric Bayesian clustering, posterior of DP and how to do inference with DP. (Jordan slide #45)
Possible further exploration:
- Sampling from Poisson-Dirichlet distribution (Donnelly-Tavaré-Griffiths sampling?)
- Proof of Lemma 3.2 from Sethuraman 1994
The number of tables distribution of a CRP has mean 
Pingback: NP Bayes Reading Group: 1st meeting | Pillow Lab Blog
I think we should also follow up on the three “fun facts” and proof that stick-breaking provides samples from a DP. Also, let’s revisit the Kingman limit; I’m still not entirely clear what limit that is, or what the other possibilities are.
I have edited the post to included the fun facts. The proof of stick breaking and contents of Kingman 1975 are coming soon, probably as a separate post. 🙂
Pingback: NP Bayes Reading Group: 3rd meeting | Pillow Lab Blog