|
Guidelines for Assessing Professional Information
Updated
December 07, 2006
Contents:
Detailed Questions for Primary Papers:
(19K)
Evidence-based Medicine Literature Evaluation
Questions: (23K)
Note: For definitions of clinical epidemiology and evidence-based medicine terms,
see the accompanying Clinical Epidemiology &
Evidence-Based Medicine Glossary.
Introduction and Background:
This is several sets of questions to assist the
clinician in determining whether or not to incorporate the information contained in a
paper, in a conference presentation or on the Internet into their professional knowledge
base for application in the user’s clinical practice. This list is in addition to a
general background knowledge about clinical studies such as the forms of bias, the types
of study designs, and their inherent strength of evidence when properly executed. As most
information sources contain flaws of varying importance and as this assessment process is
subjective, the user must make their own decision about the strength of empirical evidence
that the source contains. First some definitions.
Primary Source: A primary scientific paper is defined
as the first publication of original research results that is in a form with sufficient
detail whereby the author’s peers can critically evaluate the research process and
could repeat the study to test the conclusions. The strongest form of primary publication
is a widely available, peer-reviewed scientific journal. By convention, most primary
scientific papers have six distinct sections: abstract, introduction, materials and
methods, results, discussion and conclusions, and references. Although not all primary
sources are structured in this fashion, their presence is the hallmark of a primary
scientific paper, the only exception being systematic reviews or meta-analyses.
Each section has a distinct purpose. The abstract concisely defines
what is in the paper by stating in 250 to 400 words the study’s principle objective
and the methodology used, summarizing the results, and stating the principle conclusions.
The introduction includes a brief review of pertinent literature that provides a rationale
for the study and ends by clearly stating the problem being investigated. The materials
and methods section provides either sufficient detail or citations about the research
methods so that a competent investigator can evaluate and could repeat the work. The
results are just that. The discussion contains the author’s interpretation of the
generalization and significance of the study including how the results are consistent or
inconsistent with previously published work.
Secondary Source: A secondary source is an information
source that does not have as a major component the description of formal observations or
experiments but rather is synthesized from some combination of primary sources,
experience, or authoritative belief (dogma). The primary literature used may have been
selected in a biased or incomplete fashion and may have been used without comprehensive
critical appraisal to establish the relative strength of evidence in each source. Examples
of secondary sources are review articles, journals specializing in practitioner-oriented
reviews, most practitioner-oriented conference proceedings, trade publications, most
e-mail conversations, and authorities presenting information without supporting evidence
in whatever format (lectures, CE meetings, e-mail forums).
Tertiary Source: A tertiary source is a compilation of
information for application across a broad spectrum, typically represented by class notes
and textbooks intended for use in core courses. The information is often presented in a
dogmatic, authoritative fashion as a sequence of facts and interpretations of their
meaning that the reader is expected to believe without reservation or evaluation. The
strength of the underlying evidence is not indicated, any current controversy between
researchers in the area is not addressed and areas where further scientific evidence is
needed are not indicated. The bibliography is usually predominately of secondary
literature and is usually intended to provide the interested reader with entry points to
the underlying primary literature. Much of the evidence-based information contained in
textbooks is filtered sufficiently that it is accepted by most all of the experts in the
field, much of it is unlikely to change in the future, and most of the changes will be
minor. However, depending on the field, textbooks contain a varying amount of dogma and
interpretations of facts that will change with the progress of research in the area,
sometimes significantly. Between the publication of significant new research results in
the primary literature to their integrated into tertiary sources is often a considerable
time lag. Because clinical experience is often not examined critically, clinical textbooks
tend to contain a larger proportion of dogma than do basic science textbooks. Class notes
usually contain less information than do textbooks and do not undergo the auditing process
as part of publication as textbooks do.
[Return to Contents List]
10 Quick
Questions to Apply to Primary Papers:
Does the introduction section clearly state the problem being investigated and provide a
conceptual justification for the study supported by a review of the relevant literature?
What is the specific question that the study was intended to answer? Is it clinically
relevant?
Does the material and methods section clearly state how the problem was studied in
sufficient detail that a person familiar with the area could repeat the study? Were the
study design and methods current and appropriate to answer the question?
Were the subjects selected, the research variables measured, and the outcomes
established appropriate for the question? Are they clearly relevant to your clinical
situation?
Was symmetry maintained at every opportunity (e.g., randomization, blinding, . . .) in
the design, execution and interpretation of the study?
Does the results section clearly state the findings?
How likely are the results due to the presence of systematic bias (selection,
measurement, confounding bias) rather than the effect investigated?
Are the effects of chance on the study assessed by presenting p-values or confidence
intervals?
Are the effects sufficiently large to be biologically or clinically significant? If the
effects are not statistically significant (a negative study), is the power of the study to
detect a minimum difference given and is this difference a reasonable minimum for
biologically or clinically significance?
Does the discussion section clearly state what the findings mean in the context of the
study and discuss their clinical relevance? Are areas needing further study indicated?
Are the citations current, relevant, and mostly primary papers in refereed scientific
journals?
[Return to Contents List]
10 Quick Questions to Apply to Reviews
(modified from CMAJ 138:697-703 (1988) and J Clin Epidemiol 44:91-98
(1991))
- What are the academic (M.S., PhD degrees) and clinical (specialty boards) credentials of
the authors in the area being reviewed? Do they likely have the expertise to find and to
critically evaluate the relevant literature?
- Was the purpose of the review and the specific question(s) being addressed clearly
stated?
- Were the sources and methods of locating relevant studies stated?
- Were explicit guidelines and inclusion and exclusion criteria used to determine which to
include in the review stated? Or does the bibliography reflect primarily secondary and
tertiary sources, implying the authors relied on others for critical evaluation?
- Was selection bias minimized (e.g., not selecting only studies with positive findings)?
- Was the methodological validity of primary studies assessed and were the validity
criteria reported?
- Was the assessment of validity reproducible and was it done by more than one person and
in a blinded fashion with minimal opportunity for bias?
- Was the information systematically integrated with explanation of data limitations and
inconsistencies?
- Were clinically important areas lacking support in the primary literature identified or
were specific needs for further research identified?
- Are you aware of important and relevant primary papers that should have been included
but were not?
[Return to Contents List]
Guidelines for Attending
Professional Meetings: Pointers for the Evidence Wary
- Beware of the insidious, ever present "Dr. Fox effect".
-
"Dr. Fox" was an actor who bamboozled an audience
of professionals after being given some buzzwords from the profession to string together
for his presentation.
-
Study of the effect in a medical school class: Ware, JE, RG
Williams (1975). The Dr. Fox effect: A study of lecturer effectiveness and
ratings of instruction. J Med Educ 50:149-156.
- Lesson: Evaluate the evidence in the message and not the graphics and the
delivery (for the travel weary, flashy and polished we hope but don’t be seduced).
The message, not the messenger, has the evidence!
- Remember that numerous studies have shown that only 1/2 of new information presented at
scientific meetings will eventually be published in scientific, peer-reviewed journals.
Of the new findings being presented, many are preliminary. Further work
often does not bear them out or the study is too flawed for acceptance by a scientific
refereed journal.
- Evaluate presentations for content and strength of primary evidence.
If the presentation is of new primary evidence, evaluate it in the same
fashion as a primary evidence paper in a scientific journal.
- Inherent strength of study type? (Case series vs. RBCT) Were the key elements of design
stated and executed correctly?
- John’s Law
- If they don’t state it, they likely didn’t know they
should do it (or know how to do it right) so only by blind luck will they have done it
right.
- If they don’t state it, stand up and ask.
- Was the study of sufficient size to detect clinically important differences?
- Were the results evaluated statistically (error bars on plots, p-values)? If not, could
the outcome have been due to chance?
- Remember that approximately 350 animals per group are required to have a reasonable
chance of detecting a 10% difference in a yes / no outcome between groups.
- Remember the "3 / N Rule" – Even if all N were done with great
success, this is still consistent with a failure rate of 3 / N.
- Integration of relevant primary literature, particularly the conflicting? Does this
person keep in touch with the current scientific evidence and do they know how to evaluate
it properly? What is the conceptual and theoretical basis of this study?
- If not, are citations to the relevant primary literature appearing in scientific
journals given in the proceedings paper or is this presentation likely strictly authority
(dogma) based?
- What proportion of citations are primary rather than secondary? Is the presenter relying
on the information synthesis of others rather than doing the critical evaluation himself
or herself?
- What is the indication that the presenter keeps up with the current scientific evidence
and knows how to evaluate it?
- Rub shoulders with lots of people, listen with an open but sufficiently skeptical mind,
take good notes, and ask good questions to fill in the gaps.
Often, much useful information is exchanged between colleagues meeting
in the halls and over meals outside of the formal meetings.
- Have a Great meeting, an excellent means of recharging your batteries for
clinical practice.
[Return to Contents List]
Guidelines for
Evaluating Internet Materials (Links):
(Original set from Bibliografia
sulla qualità delle informazioni mediche)
[Return to Contents List]
|