Home              College of Veterinary Medicine             Washington State University                 WSU Faculty &Staff Page

                    John Gay, DVM PhD DACVPM               AAHP                  FDIU             VCS


Evidence-based Medicine Literature Evaluation Questions

Updated December 07, 2006

Section Contents:

Attribution and Introduction:
The following are selected and adapted from the original materials produced by the Evidence-Based Medicine Working Group, McMaster University Medical School, Hamilton, Ontario, where the EBM movement originated. These have been under development for many years (see CMAJ (1981) 124:555-558, 703-710, 869-872, 985-990, 1156-1162, (1982) 130:377-381). Questions are also adapted from pgs. 267-270 in Fletcher RH et al. (1996) Clinical Epidemiology: The Essentials 3rd ed.

The original pre-publication documents and more reside on the Evidence Based Medicine Users' Guides to the Medical Literature website at (http://www.cche.net/usersguides/main.asp). The site contains background information on why each of these questions are important and a list of references. These have been updated with more information and on-line tools on the following subscription site:

Users' Guides Interactive: An online tool to guide clinicians in the appraisal and application of evidence into their everyday practice

Rather than being targeted at specific study designs, these questions are from the perspective of the clinical aspect (diagnosis, prognosis, therapy, prevention, risk factors) to which the clinician would apply the study information.

   [Return to Section Contents]                          [Return to Contents List]


Using a Primary Article About THERAPY or PREVENTION:

(Modified from JAMA 270:2598-2601 (1993), 271:59-63(1994), online version at http://www.cche.net/usersguides/therapy.asp, which includes explanation).

How valid are the study results? (or could the presence of bias (systematic error) better explain the conclusion)

Primary Questions: (If the answer to any of these is "no", one probably should stop reading this paper and go to the next.)

  • Was the assignment of subjects to treatments randomized (vs. haphazard or "experiment of nature")?
  • Were all the subjects who entered the trial properly accounted for at its conclusion?
    • Was follow-up complete (all subjects accounted for)? If all those lost to follow-up from one group experienced a good outcome and all lost from the other experienced a bad one, or visa versa, would the study conclusions be markedly different?
  • Were subjects analyzed in the group to which they were allocated even if they didn’t complete the treatment?
  • Were subjects analyzed as a group if they were randomly allocated or handled as a group (avoiding the error of inflated precision or pseudoreplication)?

Secondary Questions: (If the answer to any of these is "no", one should retain some skepticism.)

  • Were clinicians, clients, and study personnel blind (masked) to the treatment allocation?
    • Was the evaluator of subject eligibility blinded to the assignment order?
  • Were the groups similar in clinically important factors at the start of the study?
  • Aside from the experimental intervention, were the groups treated equally (i.e., was symmetry maintained)?
What were the results?
  • How large was the estimate of the treatment effect (absolute or relative)? Is it clinically significant?
  • How precise (95% CI or SE) was the estimate of the treatment effect? Is the difference statistically significant? What high and low differences are consistent with this estimate?
Will the results help me in caring for my patients?
  • Can the results be applied to my patient care? Would my patients have been eligible for the study? Are there any reasons why the results should not be applied to my patient? (Study generalizability or external validity)
  • Were all clinically important outcomes considered?
  • Are the likely treatment benefits worth the potential harm and costs?

On-line Materials:

McMaster Size of Treatment Effect Calculator (downloadable Excel spreadsheet)

   [Return to Section Contents]                          [Return to Contents List]


Using a Primary Article About a DIAGNOSTIC TEST:

(Modified from JAMA 271:389-391, 271:703-707 (1994), online version at http://www.cche.net/usersguides/diagnosis.asp, which includes explanation)

Are the results of the study valid?

Primary Questions: (If the answer to any of these is "no", one probably should stop reading this paper and go to the next.)

  • Was an independent, blind comparison with a reasonable reference ("gold") standard (typically autopsy, biopsy, surgery, long-term follow up) applied to every subject in both groups (diseased, non-diseased) to establish the true presence or absence of the target disorder?
  • Were the observers doing subjective evaluations blind to the other test results?
  • Did the study include an appropriate spectrum of patients, both those with the target disorder and those with competing disorders, on who the test will be applied in clinical practice (clinical vs. subclinical, acute vs. chronic, mild vs. severe, primary care vs. tertiary hospital)?

Secondary Questions: (If the answer to any of these is "no", one should retain some skepticism.)

  • Did the results of the test being evaluated influence the decision to perform the reference standard? If so, how severely could this bias the estimates of test performance?
  • Were the methods for performing the test described in sufficient detail to that the test is repeatable and reliable?

What are the results?

  • Are the diagnostic sensitivity and specificity of the test provided (these are valid only if an appropriate reference standard was used)?
  • Are the confidence intervals provided for each of these? What high and low values are consistent with this estimate?
  • If a test has a range of values and a cutoff is used to establish test results (e.g., positive, negative, suspect), such as an ELISA or biochemical test, are the effects of moving the cutoff described?
  • If predictive values positive and negative are reported, are the prevalences used relevant to my clinical situation?

Will the results help me in caring for my patients?

  • Will the reproducibility of the test result and its interpretation be satisfactory in my setting? Does the test have low laboratory variation? If subjective interpretation is required, is it well described and straight forward?
  • Are the results applicable to my patients? Do my patients have a similar mix of disease severity and competing conditions?
  • Will the results change my case management? Will the information gain be sufficient to change a clinical decision (rule-in or rule-out)?
  • Will patients be better off as a result of performing the test?

On-line Materials:

Bayes Formula Calculator (Medical College of Wisconsin - one of many)

   [Return to Section Contents]                          [Return to Contents List]


Using a Primary Article About RISK FACTORS or HARM:

(Modified from JAMA 271:1615-1619 (1994), on-line version at http://www.cche.net/usersguides/harm.asp, which includes explanation)

Are the results of the study valid?

Primary Questions: (If the answer to any of these is "no", one probably should stop reading this paper and go to the next.)

  • Were clearly identified comparison groups established that were similar with respect to important determinants of outcome (confounders) other than the one of interest?
  • What was the inherent strength of the study design used (RBCT, cohort, or case-control)? Was the risk factor randomly or haphazardly assigned or was it intrinsic, self-selected, or an "experiment of nature"?
  • Were the exposures and outcomes measured in the same way in the groups being compared (e.g., was symmetry maximized)?
  • Was follow-up sufficiently long and complete? If loss to follow up is greater from one group than another, could the association have been markedly affected?

Secondary Questions: (If the answer to any of these is "no", one should retain some skepticism.)

  • Is the temporal relationship correct? Does the putative cause precede the outcome, fulfilling one of the Hill-Evans Postulates?
  • Is there a dose-response gradient? Does increasing exposure increase the risk of the outcome, fulfilling another of the Hill-Evans Postulates?

What are the results?

  • How strong is the association between exposure and outcome? How large is the relative risk (risk or incidence in exposed group / risk or incidence in the unexposed group) or the odds ratio (odds of exposure in the cases / odds of exposure in the controls)?
  • Is the magnitude of the association large enough that it is likely biologically and clinically significant? Is the association unlikely to be due to bias in the study design (RBCT vs. cohort vs. case-control)?
  • How precise is the estimate of risk (how narrow is the confidence interval)? What high and low values are consistent with this estimate?

What are the implications for my practice?

  • Are the results applicable to my practice? Are my patients sufficiently similar to those in the study in terms of age, breed, co-morbidity, and other clinically important factors?
  • What is the magnitude of the risk?
  • Should I recommend to clients that they stop the exposure? Do the benefits of removing the exposure sufficiently outweigh the costs of doing so?

[Return to Section Contents]                [Return to Contents List]


Using a Primary Article About PROGNOSIS:

(modified from JAMA 272:234-237 (1994), online version at http://www.cche.net/usersguides/prognosis.asp, which includes explanation)

Are the results of the study valid?

Primary Questions: (If the answer to any of these is "no", one probably should stop reading this paper and go to the next.)

  • Was there a representative and well-defined sample of patients at a similar point in the course of the disease when follow-up began? Was this point well described?
  • Was follow-up sufficiently long and complete for the outcomes of interest? Would those lost to follow-up markedly change the conclusions if their outcomes were all good or all bad?

Secondary Questions: (If the answer to any of these is "no", one should retain some skepticism.)

  • Were the outcome criteria reasonably objective and unbiased? Were subjective outcomes well described and were they assessed by blinded observers?
  • Was there adjustment for known important prognostic factors, such as age and co-morbidity?

What are the results?

  • How large is the likelihood of the outcome event(s) in a specified period of time?
  • How precise (confidence interval widths) are the estimates of outcome likelihood?

Will the results help me in caring for my patients?

  • Were the clinical characteristics of the study patients well described? If so, are they similar to my own patients?
  • Will the results lead directly to selecting or avoiding therapy?
  • Are the results useful for reassuring or counseling clients in making decisions?

   [Return to Section Contents]                          [Return to Contents List]


Using an OVERVIEW (from the Evidence-Based Medicine Series):

(modified from JAMA 272:1367-1371(1994), on-line version at http://www.cche.net/usersguides/overview.asp, which includes explanation )

Are the results of the study valid?

Primary Questions: (If the answer to any of these is "no", one probably should stop reading this paper and go to the next.)

  • Did the overview address a focused clinical question?
  • Were the criteria used to select articles for inclusion stated? Were they appropriate?

Secondary Questions: (If the answer to any of these is "no", one should retain some skepticism.)

  • Is it unlikely that important relevant studies were missed? Is the search strategy given?
  • Was the validity of the included studies appraised? If so, are the criteria stated? How many studies failed the criteria and why?
  • Were the assessments of study validity reproducible? Was the assessment done by more than one person?
  • Were the study results similar from study to study? If not, why not? Are the studies too dissimilar for their results to be combined?

What are the results?

  • What are the overall results of the overview?
  • How precise were the results?

Will the results help me in caring for my patients?

  • Can the results be applied to my patient care?
  • Were all clinically important outcomes considered?
  • Are the benefits worth the harms and costs?

  [Return to Section Contents]                          [Return to Contents List]


Using a CLINICAL PRACTICE GUIDELINE

(Based on the on-line version at http://www.cche.net/usersguides/guideline.asp , which includes explanation )

Are the results of the overview valid?

Primary Questions: (If the answer to any of these is "no", one probably should stop reading this paper and go to the next.)

  • Are all clinically important options and outcomes clearly specified?
  • Was the process used to identify, select and combine evidence explicit? Were the procedures for evaluating evidence from the primary literature appropriate?

Secondary Questions: (If the answer to any of these is "no", one should retain some skepticism.)

  • Was an explicit and reasonable process used to compare the alternatives?
  • Are important recent advances accounted for? Are relevant pending studies that may change the guide noted?
  • Was the guideline peer reviewed? Has it been tested in the clinical setting?

What are the results?

  • Are practical, clinically important recommendations made?
  • How strong are the recommendations
    • A level - Based on randomized controlled trials that have homogeneous results
    • B level - Based on randomized controlled trials but that have heterogeneous results
    • C level - Based on observational studies
  • Is the  uncertainty associated with the evidence on which the guidelines are based clearly stated? What is its impact??

Will the results help me in caring for my patients?

  • Can the results be applied to my patient care?
  • Are the primary objectives of the guideline consistent with your clinical objectives?

     [Return to Section Contents]                          [Return to Contents List]

URL of this page: http://www.vetmed.wsu.edu/courses-jmgay/EvalEBM.htm


Return to:  John Gay's Homepage         College of Veterinary Medicine Homepage        Washington State University Homepage

Copyright 1998-2009 John Gay, Washington State University, All rights reserved
WSU Copyright Policy                WSU Disclaimer and Freedom of Expression Policy
Homepage URL: http://www.vetmed.wsu.edu/courses-jmgay/
Site Comments to webmaster@vetmed.wsu.edu