The relative advantages of sparse versus distributed encoding for associative neuronal networks in the brain

Abstract

In some neuronal networks in the brain which are thought to operate as associative memories, a sparse coding of information can enhance the storage capacity. The extent to which this statement is valid in general is discussed here, by considering some simple formal models of associative memory which include different neurobiological constraints. In nets of linear neurons, trained with either a Hebbian (purely incremental) or a Stanton and Sejnowski learning rule, sparse coding increases the number of independent associations that can be stored. When neurons are nonlinear, for a diversity of learning rules, sparse coding may result in an increase in the number of patterns that can be discriminated. The analysis is then used to help interpret recent evidence on the encoding of information in the taste and visual systems, as obtained from recordings in primates. Following the taste pathway, it is found that the breadth of tuning of individual neurons becomes progressively finer, consistent with the idea that sparser representations become advantageous as the taste information is eventually associated with that coming from other sensory modalities. In the visual system, considering a population of neurons in the temporal cortex that respond preferentially to faces, it is argued that their breadth of tuning represents a compromise between fully distributed encoding, and a grandmother cell type of encoding, which would result in a given neuron responding only to an individual face.