Initially prepared and edited by Don Hansen and Christine Rossiter of the AABP Food Safety Committee and the National Johne’s Working Group

For our purpose, accuracy is defined as the ability of a test to correctly identify the disease status of the animal tested. With a test that is 100% accurate, a non-diseased animal will always have a negative test result and a diseased animal will always have a positive test result. If we place a group of animals into categories by true disease or infection status and diagnostic test results, they will fall into one of four groups: (Figure 1)

1. diseased and test positive;         (True Positive)
2. non-diseased and test positive    (False Positive)
3. diseased and test negative;        (False Negative)
4. non-diseased and test negative   (True Negative)

Figure 1:

With a perfectly accurate test, all animals would be placed into either category A or D. However, few diagnostic tests are 100% accurate. Sometimes the infected animal is shedding too few organisms for the culture techniques to detect and the test gives a false negative result. Salmonellosis and Staph aureus mastitis are such examples. At other times a non-pathogenic "look alike" organism may be mistaken for the pathogen such as may happen with trichomonad identification in a case of suspected trichomoniasis. The test result would be a false positive.

Diagnostic test accuracy is based on two measures of performance, the sensitivity and specificity of the test. Together, they describe how well tests detect the true disease or infection status of animals being tested. Sensitivity is the ability of a test to detect infected animals correctly with a positive result. Specificity is the test’s ability to detect non-infected animals correctly with a negative result. The better the sensitivity and specificity of the test, the greater the accuracy.

In the process of test validation, sensitivity and specificity values are determined by evaluating the test independently in two separate groups of animals. All animals in one group are confirmed to be infected using a "gold standard" test or combination of tests. All animals in the other group are confirmed to be free of infection. Both should otherwise represent the general population of interest where the test will be used, i.e., (U.S.) cattle population.

The larger and more representative each group is, the better the validation and the more confidant we can be that the values describe how we expect the test to work in infected and non-infected individuals.

Sensitivity values for Johne’s tests are established by using the test in a well described representative population of infected cattle. Their infected status must be confirmed by the best "gold standard" test (s) for Mycobacterium avium subspecies paratuberculosis (Map) available. Preferably they also represent the various stages of Johne’s infection. Because tests are unable to detect Stage I infection, these animals are often not represented in validation populations.

Sensitivity is the percent of all infected animals in the validation sample that have a positive test result (A/A+C, from Figure 1). Therefore, sensitivity (SE) becomes the expected probability or chance of getting a positive test result in an infected animal. The higher the sensitivity, the fewer false-negative test results are expected to occur.

See the 2X2 table in Figure 2 for an example calculation of sensitivity for an ELISA in a validation group of 153 infected cows which yields a SE = 25%.

Figure 2:

Specificity values for Johne’s tests are established by testing a representative population of cattle known to be free of infection with Map. Prior to validation, all cattle in this group must be confirmed by history and negative herd tests (the best "gold standard" test) to not to be infected.

Specificity of the test is the percent of the non-infected animals that correctly have a negative test result (D/B+D). Therefore, the specificity (SP) is the expected probability, or likelihood of the test giving a negative result in infection-free animals. The higher the specificity, the fewer false-positive test results are expected to occur. See the example specificity calculation for an ELISA test in a group of 357 infection-free cows in Figure 2, which yields a SP = 98.5%.

The intracellular and slowly progressive nature of Map infection is largely responsible for the relatively low sensitivity of most Johne’s tests. Recall the four stages of Johne’s infection. Current tests generally cannot detect early Stage I infection and they miss many of the subclinically infected animals in Stage II.

However, tests will detect many infected cattle that are approaching (pre-clinical) Stage III, and nearly 95% of clinical stage IV cattle will be positive on most tests. Thus, false negative test results are common for cattle in the early stages of Johne’s infection, which includes most immature cattle.

The specificity values for ELISA and fecal culture are relatively high, meaning that false positive results are relatively uncommon (0% to 3% of non-infected cattle). However, even though the number is low, it is a quandary to know how to interpret positive test results when they do occur in unknown or low risk situations. Are they right or wrong? Most false positive results come from interference by cross-reacting organisms to which the animal has been exposed, i.e., other Mycobacteria, Corynebacteria, etc.

The population, distribution of stages of infection, and the environment from which the tested animals come may influence the sensitivity and specificity of a test in that group of animals. This is true for the published estimates and how they may apply when the test is used in clients’ herds. If we test only animals in the early stages of infection (immature cattle), the sensitivity of Johne’s tests will be low (<20%). On the other hand, if the animals were all in Stage IV, the sensitivity will be high (>90%). Published sensitivity values represent a population of infected animals in various stages.

Accurately defined Johne’s disease populations are difficult to find. Therefore, it is important to recognize that sensitivity and specificity values for Johne’s diagnostic tests are estimates that have been determined from finite populations. They do not necessarily represent all of the cattle in the US, nor all geographic regions of the country.

In the process of validating diagnostic tests that are measured on a continuous scale, e.g. antibody titers, a cut-off point is established and used to discriminate a "positive" from a "negative" result with the greatest accuracy possible. In deciding where a cut-off should be there is an inverse trade-off at any particular cut-off point between sensitivity and specificity.

For example, an ELISA test produces a range of values (derived from Optical Density readings reflecting antibody levels) from low to high. At some point along this continuum a cut-off is determined, above which the result values are considered to be positive. OD values below the cut-off point are considered negative. If the cut-off point is moved to a lower value on the scale, the sensitivity will increase and more infected animals will be called positive. However, the specificity at the new cut-off will decrease, and more non-infected animals will also be called positive (Figure 3).

Raising the cut-off point on the scale will do the opposite. The specificity will increase since only animals with high values will be called positive. Non-infected animals with high enough values to be called positive (a false positive result) will be rare. But the sensitivity will decrease at the same time. More infected animals with elevated values in an intermediate range will be called negative.

Although serology tests usually report results as positive or negative based on a single cutoff point, more information can be gained by looking at how the actual values are distributed around the cut-off value. It is useful to consider how the sensitivity, specificity and resulting decisions based on test results change when the cut-off is raised or lowered. Some test interpretations provide multiple cut-offs and a probability of infection associated with each one. In practice we also often develop our own cut-offs for making different decisions depending how accurate we think our test is, the benefit or return if we are right, and the cost if we are wrong (Figure 3).

Figure 3. Relative frequency of hypothetical titer values is graphed for two separate populations, diseased and non-diseased. The critical cut off point represents an optimal value for the diagnostic test.

Even though the current Johne’s tests are not 100% accurate, they are still useful tools for herd monitoring and disease control programs.  However, when a positive test result comes back from a lab, the question should arise, “Is this right?”  It would be helpful to know how much confidence we can have that the positive test result is correct.

One useful way to assess the chance or likelihood that a test-positive animal is truly infected, or a test-negative animal is truly not infected is to consider the predictive value of the positive (PPV) or negative (NPV) test result. Predictive value can be calculated or figured in a 2X2 table. However, it is actually an intuitive concept that takes into account:

1.  the sensitivity (SE) of the test (% of infected cattle expected to test positive)

2.  the specificity (SP) of the test (% of non-infected cattle expected to test negative)

3. the prevalence or number of infected animals expected to be in the population being tested (before testing).

Before proceeding further with predictive value, we need a definition of prevalence. 

The PPV is the percent of test-positive animals for which the result is correct, i.e., truly infected, (A/(A+B)).  It is affected by the disease prevalence in the herd, or the pre-test estimate of disease in the individual that is tested, i.e., a presumptive diagnosis or pre-test expected herd prevalence. The PPV is the probability that a positive test result agrees with true infection status.  Its calculation requires estimates of SE and SP for the test and herd prevalence or  probability of infection in an individual (P).

Prevalence and probability estimates are  determined from existing information.  Some options for making these estimates are listed in order of increasing accuracy.

1. Use an estimated prevalence based on herd history and risk assessment as estimated from pages A-2 through B7 of the Manual for Veterinarians, Johne’s disease Control Plan Guidelines or the NYSCHAP Johne’s Disease Risk Assessment.

2. Use the presumptive diagnosis, i.e., an  estimate of the probability that the animal may have Johne’s disease versus other disease(s).

3. Use a test prevalence estimate based on herd test results and calculate an estimate of the true prevalence (P).

The formula for calculating the predictive value of a positive test is: 

 PPV = SE x P / (SE x P)+[(1-SP) x (1-P)]. 

 For example, if SE =.25, SP=.985, P=.30, then PPV = 88%.  

 At 30% prevalence, a positive test result has about an 88% chance of being correct. Another approach which is more intuitive for some is illustrated in Figure 4, using numbers in the original 2X2 table. 

Figure 4:

		DISEASE STATUS			Calculated Predictive Values
		Present	Absent
TEST RESULT	Positive	38 A	5 B	A+B = 43	PPV = A/ (A+B) = 38/43 = 88%
	Negative	C 115	D 352	C+D = 467	NPV = D/ (C+D) = 352/467 = 75%
		A+C=153 SE = A/(A+C) = 38 / 153 = 25%	B+D=357 SP = D/(B+D) = 352 / (5+352) = 98.5%	A+B+C+D = 510 P = 153 / 510 = 30%

As no commercial test for Johne’s disease can detect early-stage infections, it is not possible for a single negative test result to assure that an animal is not infected.   Nevertheless, we can calculate a probability that a negative test result is correct called the negative predictive value (NPV) of the test result.  For Johne’s tests, NPV values usually overestimate the true probability of being uninfected (related to the fact that the SE of Johne’s tests are often overestimates and there are more infected but test-negative animals than we think). However, they are still very useful for individual animal and herd sample screening tests.

The NPV is figured as the percent of animals that have a negative test result, for which the negative result is correct.  It also represents an estimate of the probability or chance that a negative test result in an  individual animal correctly indicates it is not infected. Like the PPV, the NPV value for a negative test result will vary depending on the prevalence or pre-test estimate of infection in the individual.  

 The NPV can be calculated using the SE and SP of the test and a pre-test estimate of the  prevalence or probability of infection in the following formula: 

 NPV =SP x (1-P) / [SP x (1-P)]+[(1-SE)x P].  

 For example, if SE =.25, SP=.985, P=.30, then NPV=74%.  At 30% prevalence, the negative test result has a 74% chance of being correct.

 See Figure 4 above for the more intuitive calculation example using numbers in a 2x2 table. 

Table 1 has predictive values for several levels of prevalence and two reported estimates of SE for Johne's ELISAs. Use the table to find the predictive values (PPV or NPV) for the ELISA or fecal culture tests based on your estimate of the prevalence in the animal(s) tested.

	ELISA25		ELISA60		CULTURE
Estimated Prevalence	PPV	NPV	PPV	NPV	PPV	NPV
1%	15%	99%	16%	99%	99%	99%
5%	48%	96%	51%	97%	100%	97%
10%	67%	92%	69%	95%	100%	94%
20%	82%	84%	83%	91%	100%	87%
30%	88%	75%	90%	85%	100%	80%
40%	92%	66%	93%	78%	100%	71%
50%	95%	57%	95%	71%	100%	63%

Table 1.  For the generic Johne’s ELISA or fecal culture, the predictive value varies most dramatically with the prevalence (Estimated Prevalence) of infection in the herd. Next it is influenced by the SP of the test, and generally least by the test’s SE. 

 SE and SP values of 25% and 98% were used in the column headed ELISA25.  Those values have been recommended by the National Johne’s Working Group for assessing herds participating in the Voluntary Johne’s Disease Herd Status Program (VJDHSP).  

 SE and SP values of 60% and 97% were used in the column labeled ELISA60. These values are reported by a commercial manufacturer of ELISA tests.  When thinking about the performance of the ELISA across the ‘general’ population of infected cattle, these values should be considered an overestimate of the SE.

 For fecal culture (CULTURE), SE and SP values of 40% and 99.9% were used, also recommended by the National Johne’s Working Group for the VJDHSP.

Note that there are two important relationships between predictive values and disease prevalence (or probability estimates) that may be used in practice. 

 For tests less than 100% specific, the predictive value of a positive test result decreases as the estimates of prevalence or probability of infection decreases. In other words, the chance that a positive result is correct gets smaller, the fewer infected animals there are. 

 Because all Johne’s tests are less than 100% sensitive, the predictive value of a negative test result gets lower as the estimated herd prevalence or probability of infection increases.  The more infected animals there are, the greater the number of negative test results that will be false-negative results.

Understanding these relationships will help when interpreting positive and negative test results and assessing their potential impact for client’s decisions.

Understanding of the importance of the predictive value of test results is an incentive to veterinarians to do a Johne’s risk-assessment with their clients. A working pre-test estimate of the herd prevalence or probability of infection is an important piece of information to better judge the likely accuracy of individual test results and better help clients control Johne’s disease.  A good history and risk assessment combined with clinical evaluation of individual cows helps establish these estimates.

Veterinary Clinical Epidemiology by Ronald D. Smith, CRC Press, 1995 is a good reference for more discussion of predictive values and another predictor value called the Likelihood Ratio.

[INCOMPLETE - pending availability of AABP version on-line]