Sparse Bayesian Learning for Basis Selection
Top Cited Papers
- 19 July 2004
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Signal Processing
- Vol. 52 (8), 2153-2164
- https://doi.org/10.1109/tsp.2004.831016
Abstract
Sparse Bayesian learning (SBL) and specifically relevance vector machines have received much attention in the machine learning literature as a means of achieving parsimonious representations in the context of regression and classification. The methodology relies on a parameterized prior that encourages models with few nonzero weights. In this paper, we adapt SBL to the signal processing problem of basis selection from overcomplete dictionaries, proving several results about the SBL cost function that elucidate its general behavior and provide solid theoretical justification for this application. Specifically, we have shown that SBL retains a desirable property of the /spl lscr//sub 0/-norm diversity measure (i.e., the global minimum is achieved at the maximally sparse solution) while often possessing a more limited constellation of local minima. We have also demonstrated that the local minima that do exist are achieved at sparse solutions. Later, we provide a novel interpretation of SBL that gives us valuable insight into why it is successful in producing sparse representations. Finally, we include simulation studies comparing sparse Bayesian learning with basis pursuit and the more recent FOCal Underdetermined System Solver (FOCUSS) class of basis selection algorithms. These results indicate that our theoretical insights translate directly into improved performance.Keywords
This publication has 27 references indexed in Scilit:
- Adaptive filtering algorithms for promoting sparsityPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2004
- Sparse inverse solution methods for signal and image processing applicationsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- A Variational Method for Learning Sparse and Overcomplete RepresentationsNeural Computation, 2001
- Proportionate normalized least-mean-squares adaptation in echo cancelersIEEE Transactions on Speech and Audio Processing, 2000
- An Introduction to Variational Methods for Graphical ModelsMachine Learning, 1999
- Fast orthogonal least squares algorithm for efficient subset model selectionIEEE Transactions on Signal Processing, 1995
- Sparse Approximate Solutions to Linear SystemsSIAM Journal on Computing, 1995
- Sparse approximate multiquadric interpolationComputers & Mathematics with Applications, 1994
- Restoration of blurred star field images by maximally sparse optimizationIEEE Transactions on Image Processing, 1993
- Statistical Decision Theory and Bayesian AnalysisPublished by Springer Nature ,1985