Consider two observers who separately count the different species of plants or animals in a given area. If they each come back having found 100 species but only 5 particular species are found by both observers, then each observer clearly missed at least 95 species. Thus, we know that both observers miss a lot. On the other hand, if 99 of the 100 species each observer found had been found by both, it is fair to expect that they have found a far higher percentage of the total species that are there to find. The same reasoning applies to mark and recapture. If some animals in a given area are caught and marked, and later a second round of captures is done: the number of marked animals found in the second round can be used to generate an estimate of the total population. Another example arises in computational linguistics for estimating the total vocabulary of a language. Given two independent samples, the overlap between their vocabularies enables a useful estimate of how many more vocabulary items exist but did not happen to show up in either sample. A similar example involves estimating the number of typographical errors remaining in a text, from two proofreaders' counts.
Formulation
The Lincoln Index formalizes this phenomenon. If E1 and E2 are the number of species observed by two independent methods, and S is the number of observations in common, then the Lincoln Index is simply For values of S < 10, this estimate is rough, and becomes extremely rough for values of S < 5. In the case where S = 0 the Lincoln Index is formally undefined. This can arise if the observers only find a small percentage of the actual species, if the observers are using methods that are not statistically independent, or in other circumstances.
Limitations
The Lincoln Index is merely an estimate. For example, the species in a given area could tend to be either very common or very rare, or tend to be either very hard or very easy to see. Then it would be likely that both observers would find a large share of the common species, and that both observers would miss a large share of the rare ones. Such distributions would throw off the consequent estimate. However, such distributions are unusual for natural phenomena, as suggested by Zipf's Law). T. J. Gaskell and B. J. George propose an enhancement of the Lincoln Index that claims to reduce bias.
