[(E1 + 1)(E2 + 1) / (S + 1)] - 1

is preferred.

The simplicity of the arithmetic hides the complications of doing data collection well. The Petersen and Seber estimators assume (1) that the population is closed (e.g. the number of errors is constant) - this can also mean that the population is both closed and “well defined” (e.g. the two proof-readers have the same version of the text), (2) all errors are equally catchable by both proof-readers, (3) detected errors can be matched, and (4) detection of an error by one proof-reader is not influenced by the detection of the same error buy the other proof-reader. This is, I think, more stringent than two simple random samples mentioned above (that would cover (2) and (4) only). It may be easy (I have my doubts) to meet these assumptions in the context of proof-reading but is very difficult in epidemiological applications. In my experience, capture-recapture studies are often flawed (all of mine have been!). It is then a matter of identifying violations of assumptions and their likely effect in terms of the magnitude and direction on the final estimate.

There are several methods that use multiple testers / multiple lists. John might be interested in:

Schnabel, Z. E. (1938), “The Estimation of the Total Fish Population of a Lake”, American Mathematical Monthly, 45, 348–352.

which proposes an extension to the Petersen method using cumulative marking using a single mark (i.e. no need for multiple marking or different marks for different testers).

Good summaries of abundance estimators for capture-recapture data can be found in many textbooks. Krebs’ “Ecological Methodology” is well regarded.

Regards,
Sachin.

Our conversation then turned to how I couldn’t have found the answer faster had he handed me a book and told me what page to look on, let alone if we had needed to go to a library or arrange for a reference to be sent from another library!

- - - - -
Suppose that two editors, Jack and Jill, independently read a long newspaper article supplied by one of their journalists. Jack finds A typing errors, whilst Jill finds B typing errors. They compare copies and discover that they both found the same error on C occasions. How many errors do you expect to remain, unfound, in the article?

Let us suppose that the total number of errors in the article is E. This means that the number that have yet to be found equals E-A-B+C. The last factor of +C is so that we don’t double count the errors that Jack and Jill both found. Now, if the probability that Jack spots an error is p, and the probability that Jill spots an error is q, then we expect that A=pE, B=qE, and C=pqE, because they search independently. So, AB=pqE x E; hence AB=CE. Now we have the answer: the number of unfound errors equals E-A-B+C=AB/C - A - B + C, where we have replaced the unknown quantity, E, by AB/C. rearranging our formula, we have shown that the number of unfound errors is equal to (A-C)(B-C)/C; that is,

Number of unfound errors=(Number found only by Jack) x (Number found only by Jill)/(Number found by both Jack and Jill)
- - - - -

I wonder how well this kind of thing could apply to software quality results

That’s the question raised in the blog posting by Mat Roberts that launched all this.

or adolescent European Plaice are susceptible to conformal peer pressure…

Of course! Piercings. How could I have missed the obvious?

@Greg: If both readers find all the errors, then E1=E2=S, and also E1+E2-S = E1E2/S. But the situation we’re interested in is one where we don’t know how many errors really exist; all we know is how many errors the two readers found, and how many of those were found by both. Your formula assumes that there are no other errors. The Lincoln/Peterson/Laplace formula assumes that both readers have gathered a random sample of errors from the unknown population. By the way, a corner case is when the two error sets have no overlap at all — i.e., S = 0. One response to this situation is to say that with no overlap we have no upper bound on the total population. But it’s also possible to take a less dire view, and use a formula along the lines of:
(E1+1)(E2+1)/(s+1) - 1.

@John Cowan: The whole world is counting on you to fill in that last missing piece of the community memory bank!

I don’t think so, because testing for vulnerabilities is generally not a simple random sampling mechanism. Much depends on the method used for testing - static analysis versus fuzzing versus manual inspection or grey-box testing for example.

Certain testing methods will typically only find certain classes of vulnerabilities for example, and with varying degrees of effectiveness. Fuzzing (pseudo-random input with varying degrees of structure) might be very good at uncovering buffer overflows and other parsing or data-plane/semantic-boundary transition problems, but it typically doesn’t do as well at identifying privilege or permission bypass issues.

On the other hand, if you had two different teams fuzzing your firewall for example, and they came up with different results with some subset of identical results, it might make sense to apply the Lincoln index and estimate the total number of defects that could eventually be discovered by fuzzing, since fuzzing is essentially a random selection process.

The other interesting thing about fuzzing is that it tends to be subject to diminishing returns, so you might also be able to estimate the total “findable” number from the decline in the rate of identification. So perhaps the most useful upper bound would be to take E1 and E2 to be the rate-adjusted total-findable estimates (instead of the currently identified count) for the two different teams.

Another case where the selection process might not have been random is Petersen’s fish experiment. If 1/7th of a fish population had holes punched in their dorsal fins, and 1/5th of later catches were found with punched dorsal fins, it kind of suggests that punching holes in dorsal fins might make fish easier to catch. Either that, or adolescent European Plaice are susceptible to conformal peer pressure…