The optical fractionator is a design‐based two‐stage systematic sampling method that is used to estimate the number of cells in a specified region of an organ when the population is too large to count exhaustively. The fractionator counts the cells found in optical disectors that have been systematically sampled in serial sections. Heretofore, evaluations of optical fractionator performance have been made by performing tests on actual tissue sections, but it is difficult to evaluate the coefficient of error (CE), i.e. the precision of a population size estimate, by using biological tissue samples because they do not permit a comparison of an estimated CE with the true CE. However, computer simulation does permit making such comparisons while avoiding the observational biases inherent in working with biological tissue. This study is the first instance in which computer simulation has been applied to population size estimation by the optical fractionator. We used computer simulation to evaluate the performance of three CE estimators. The estimated CEs were evaluated in tests of three types of non‐random cell population distribution and one random cell population distribution. The non‐random population distributions varied by differences in ‘intensity’, i.e. the expected cell counts per disector, according to both section and disector location within the section. Two distributions were sinusoidal and one was linearly increasing; in all three there was a six‐fold difference between the high and low intensities. The sinusoidal distributions produced either a peak or a depression of cell intensity at the centre of the simulated region. The linear cell intensity gradually increased from the beginning to the end of the region that contained the cells. The random population distribution had a constant intensity over the region. A ‘test condition’ was defined by its population distribution, the period between consecutive sampled sections and the spacing between consecutive sampled disectors. There were 1000 independently simulated cell populations for each test condition, and a ‘trial’ was conducted for each of these cell populations. In each trial we calculated the (unique) true CE of the population size estimate and the three CE estimates obtained by applying the Scheaffer–Mendenhall–Ott (SMO) and both Gundersen–Jensen (GJ) estimators. We compared the estimated CEs with the true CEs for each population distribution. We found that the CE estimates obtained by the SMO estimator were closer to the true CEs and had less scatter than those of the nugget‐modified GJ estimator. Both had small positive bias. The CE estimates obtained by the unmodified GJ estimator exhibited widely varying bias and large scatter. In all the population distributions we tested, the average true CE was very nearly proportional to 1/√Q¯T, where Q¯T is the average number of cells counted in the two‐stage systematic sample.