infomax | Pillow Lab Blog

As the first paper in our summer reading series on information theoretic learning and synaptic plasticity of spiking neurons, we discussed one of the earliest papers:

Taro Toyoizumi, Jean-Pascal Pfister, Kazuyuki Aihara, Wulfram Gerstner. Generalized Bienenstock–Cooper–Munro rule for spiking neurons that maximizes information transmission. Proceedings of the National Academy of Sciences of the United States of America, Vol. 102, No. 14. (05 April 2005), pp. 5239-5244, doi:10.1073/pnas.0500495102

BCM rule is a rate-based synaptic plasticity rule which is a stable version of naive Hebbian learning rule $\Delta w \propto x y$ where $x$ and $y$ are rate of pre-synaptic neuron and post-synaptic neuron respectively. Hebbian learning rule has positive feedback; high firing rate makes the synapse stronger which in turn makes firing rate higher. BCM rule fixes this with a sliding threshold which is controlled by the output (post-synaptic) firing rate – higher firing rate makes the threshold get higher, which increases the range of depression.

This paper by Toyoizumi et al derives a learning rule from first principle of maximizing mutual information between input spike trains and the output spike train. This alone will prefer high firing rate, so they include a penalty for high firing rates. The cost function is:

$L = I(\mathbf{X};\mathbf{Y}) - \gamma D_{KL}(P(Y)||\tilde P(Y))$

where $I$ denotes mutual information, $D_{KL}$ is the Kullback-Leibler divergence, $\gamma$ is the trade-off between the two terms, and $\tilde P(Y)$ is the target spike rate distribution. The paper utilizes an escape rate approximation of IF neuron with extra spike-history dependent rate modulation term to define the conditional probability $P(Y|X)$ . Then, the synaptic plasticity rule is obtained by taking the derivative $\frac{\partial L}{\partial w_i}$ , then applying time average instead of expectation, and small step size approximation, they arrive at an online learning rule per synapse that resembles BCM in the special case of LNP neuron (no refractory). Next meeting (this Friday), we will dig deeper into the details of the derivation.