Redundancy Reduction as a Strategy for Unsupervised Learning

Abstract

A redundancy reduction strategy, which can be applied in stages,is proposed as a way to learn as efficiently as possible thestatistical properties of an ensemble of sensory messages. Themethod works best for inputs consisting of strongly correlatedgroups, that is features, with weaker statistical dependencebetween different features. This is the case for localized objectsin an image or for words in a text. A local feature measuredetermining how much a single feature reduces the total redundancyis derived which turns out to depend only on the probability of thefeature and of its components, but not on the statisticalproperties of any other features. The locality of this measuremakes it ideal as the basis for a "neural" implementation ofredundancy reduction, and an example of a very simple non-Hebbianalgorithm is given. The effect of noise on learning redundancy isalso discussed.