A simple method of removing the effect of a bottleneck and unequal population sizes on pairwise genetic distances

Abstract

In this paper, we derive the expectation of two popular genetic distances under a model of pure population fission allowing for unequal population sizes. Under the model, we show that conventional genetic distances are not proportional to the divergence time and generally overestimate it due to unequal genetic drift and to a bottleneck effect at the divergence time. This bias cannot be totally removed even if the present population sizes are known. Instead, we present a method to estimate the divergence times between populations which is based on the average number of nucleotide differences within and between populations. The method simultaneously estimates the divergence time, the ancestral population size and the relative sizes of the derived populations. A simulation study revealed that this method is essentially unbiased and that it leads to better estimates than traditional approaches for a very wide range of parameter values. Simulations also indicated that moderate population growth after divergence has little effect on the estimates of all three estimated parameters. An application of our method to a comparison of humans and chimpanzee mitochondrial DNA diversity revealed that common chimpanzees have a significantly larger female population size than humans.