Quantitative linguistics and complex system studies*

Abstract

Linguistic discourses treated as maximum entropy systems of words according to prescriptions of algorithmic information theory (Kolmogorov, Chaitin, & Zurek) are shown to give a natural explanation of Zipf's law with quantitative rigor. The pattern of word frequencies in discourse naturally leads to a distinction between two classes of words: content words (c‐words) and service words (s‐words). A unified entropy model for the two classes of words leads to word frequency distribution functions in accordance with data. The model draws on principles of classical and quantum statistical mechanics and emphasises general principles of classifying, counting and optimising their related costs for coding of sequential symbols, under certain obvious constraints; hence it is likely to be valid for diverse complex systems of nature. Unlike other models of Zipf s law, which require exponential distribution of word lengths, entropy models based on words as primary symbols do not restrict the word length distribution. It is shown that language exhibits the characteristics of complex adaptive systems (Gell‐Mann, 1994), in which the complexity measure is maximal for a system of intermediate algorithmic entropy, between totally ordered and disordered systems. A complexity function ‐ a higher order entropy ‐is defined for linguistic discourse which has the above properties. Natural discourses indeed seem to have the right mix of order and randomness and a complexity close to maximal.