Efficient supervised learning in networks with binary synapses
Open Access
- 26 June 2007
- journal article
- Published by Proceedings of the National Academy of Sciences in Proceedings of the National Academy of Sciences of the United States of America
- Vol. 104 (26), 11079-11084
- https://doi.org/10.1073/pnas.0700324104
Abstract
Recent experimental studies indicate that synaptic changes induced by neuronal activity are discrete jumps between a small number of stable states. Learning in systems with discrete synapses is known to be a computationally hard problem. Here, we study a neurobiologically plausible on-line learning algorithm that derives from belief propagation algorithms. We show that it performs remarkably well in a model neuron with binary synapses, and a finite number of "hidden" states per synapse, that has to learn a random classification task. Such a system is able to learn a number of associations close to the theoretical limit in time that is sublinear in system size. This is to our knowledge the first on-line algorithm that is able to achieve efficiently a finite number of patterns learned per binary synapse. Furthermore, we show that performance is optimal for a finite number of hidden states that becomes very small for sparse coding. The algorithm is similar to the standard "perceptron" learning algorithm, with an additional rule for synaptic transitions that occur only if a currently presented pattern is "barely correct." In this case, the synaptic changes are metaplastic only (change in hidden states and not in actual synaptic state), stabilizing the synapse in its current state. Finally, we show that a system with two visible states and K hidden states is much more robust to noise than a system with K visible states. We suggest that this rule is sufficiently simple to be easily implemented by neurobiological systems or in hardware.Keywords
This publication has 27 references indexed in Scilit:
- Learning by Message Passing in Networks of Discrete SynapsesPhysical Review Letters, 2006
- Convergence of stochastic learning in perceptrons with binary synapsesPhysical Review E, 2005
- Survey propagation: An algorithm for satisfiabilityRandom Structures & Algorithms, 2005
- Emergent Properties of Networks of Biological Signaling PathwaysScience, 1999
- Slow stochastic Hebbian learning of classes of stimuli in a recurrent neural networkNetwork: Computation in Neural Systems, 1998
- Learning in Neural Networks with Material SynapsesNeural Computation, 1994
- Constraints on learning in dynamic synapsesNetwork: Computation in Neural Systems, 1992
- Capacity of neural networks with discrete synaptic couplingsJournal of Physics A: General Physics, 1990
- The interaction space of neural networks with sign-constrained synapsesJournal of Physics A: General Physics, 1989
- Critical storage capacity of the J = ± 1 neural networkJournal of Physics A: General Physics, 1989