Relative entropy and learning rules

Abstract

The dynamics of a probabilistic neural network is characterized by the distribution ν(x’‖x) of successor states x’ of an arbitrary state x of the network. A prescribed memory or behavior pattern is represented in terms of an ordered sequence of network states ${\mathit{x}}^{\mathrm{}\left(1\right)\mathrm{}}$,${\mathit{x}}^{\mathrm{}\left(2\right)\mathrm{}}$,...,${\mathit{x}}^{\left(\mathit{l}\right)}$. A successful procedure for learning this pattern must modify the neuronal interactions in such a way that the dynamical successor of ${\mathit{x}}^{\left(\mathit{s}\right)}$ is likely to be ${\mathit{x}}^{(\mathit{s}+1)}$, with ${\mathit{x}}^{(\mathit{l}+1)}$=${\mathit{x}}^{\mathrm{}\left(1\right)\mathrm{}}$. The relative entropy G of the probability distribution ${\mathrm{\delta }}_{\mathit{x}}^{(\mathit{s}+1)}$,x’ concentrated at the desired successor state, evaluated with respect to the dynamical distribution ν(x’‖${\mathit{x}}^{\left(\mathit{s}\right)}$), is used to quantify this criterion, by providing a measure of the distance between actual and ideal probability distributions. Minimization of G subject to appropriate resource constraints leads to ‘‘optimal’’ learning rules for pairwise and higher-order neuronal interactions. The degree to which optimality is approached by simple learning rules in current use is considered, and it is found, in particular, that the algorithm adopted in the Hopfield model is more effective in minimizing G than the original Hebb law.