A count‐dependent filter for smoothing flow cytometric histograms

Abstract

An adaptive count‐dependent algorithm for smoothing statistically limited histograms has been developed. It considers both the spatial frequency limitations of the measurement system (described by the measurement system point spread function) and the reliability of the measured data (indicated by the effective number of counts influencing each channel of the histogram). Windows for smoothing flow cytometric histograms are derived from an assumed Gaussain‐shaped point spread function (PSF) with a constant coefficient of variation. The windows are developed by scaling the variances of the Gaussian functions inversely with the statistical reliability of the data contained in each channel of the measured histogram. The reliability of this data is determined by taking the square root of the number of counts influencing the value tabulated for each channel. Using the algorithm, a smoothed version of the measured histogram may be developed from a linear sum of the products of the individual scaled Gaussian functions and the original measured histogram. Data are presented demonstrating the advantages of count‐dependent smoothing over non‐count‐dependent smoothing using synthesized DNA histograms as a function of sample size.