How to Know You Can Believe What You Read A brief description of the fundamentals of scientific literature review with links to on-line resources

Kirsti Bocskay, PhD, MPH Epidemiologist IV Office of Epidemiology Chicago Department of Public Health     Evidence-Based Medicine Requires New Skills   Evidence-based medicine (EBM) is when health practitioners treat patients using their own clinical experience and the careful, specific and prudent application of knowledge obtained from review of published research. The Journal of the American Medical Association (see below) and the British Medical Journal (see below) have both published a series of articles outlining best practices for reviewing research papers for use in clinical practice. Many online resources have summarized and/or adapted these articles, principles and guides [see On-Line Citations below].   Three Guiding Principles EBM has generated considerable interest among health care professionals, and consequently the how-tos of critically evaluating scientific papers has become a hot topic. This article will briefly review some of the basic fundamentals you need to know when evaluating scientific literature for your practice and will also provide references and links when available to more detailed articles and examples. Regardless of the type of study (clinical trial, intervention, case-control, etc.), three guiding principles for critiquing a paper are validity, reliability and applicability.   Validity Validity addresses the research methods used by a study's authors and whether or not their conclusions are supported by the study findings rather than by chance or a flaw in the study design or analysis or some other unmeasured variable or factor. In order to evaluate the validity of a study, you should look for several key pieces of information within the article. Authors should clearly state the aims of the research and hypothesis that their research will prove or disprove, along with the gaps in knowledge that their study will fill. A concise literature review highlighting all current and pertinent prior research is also imperative. Next, think about the choice of study design and whether it is suitable for the question or questions being asked by the authors. For instance, if the aim of a study is to test the efficacy of a new smoking cessation drug, a randomized clinical trial would be more appropriate than a cross-sectional study design using a survey to measure smoking cessation drug use and smoking habits. Another consideration is the study population, and specifically the comparison groups, case-control, exposed vs. unexposed, treatment vs. placebo, etc. The comparison groups should be the same except for the disease, exposure or treatment under investigation. If the group being treated with the new smoking cessation drug is composed of middle-aged, white males and the placebo group is adolescent, Hispanic females, the study findings could be biased as older persons may be more likely to want to quit smoking than teenagers, regardless of medication use. What if the drug works differently in men than women? It could have no measurable effect in men but might be very helpful in women. Given the comparison groups chosen by the authors, this outcome would not be detected. Additionally, the same criteria must be used for both groups in regards to inclusion and exclusion to the study. All study participants should also go through the same selection process, e.g. the placebo group shouldn't be volunteers and the treatment group handpicked from a sample of hospital patients. If the study required follow-up, the authors should discuss the number of participants who dropped out and if they differed from the persons that completed the study. Well documented studies will also have information on "power calculations" or the statistical analysis performed which provides an estimate of the number of study participants needed to see a significant difference in outcomes between groups. Generally, the larger the sample size, the smaller the effect size that can be detected. If the sample size is not justified but 95% confidence intervals are published in the results section, use them as an indicator. Confidence intervals that are too wide may indicate the sample size was not large enough. Validity can also be compromised if the outcomes being measured do not match up with the study question. For example, if the study was examining the efficacy of a new smoking cessation drug and its affect on long term quitting. The outcome variable of interest may be smoking status at one year following treatment, not smoking status at 1 month following treatment or a measure of the drug concentration excreted in urine. The authors should also give enough detail in their description of the methods utilized so the study can be reproduced by you and other researchers. As a final point, the authors should consider other variables or factors that could be associated with the disease, exposure or treatment and account for them in their study design through sample design and/or statistical analysis. Key words to look for include confounding, interaction, randomization, matching and adjustment (see inset). Assessing Validity of a Study What is the aim of the research or the question the researchers are asking? Was the study design appropriate for the question asked? How was the sample chosen? What is the sample size, and is it large enough to detect a difference?  What criteria were used to include and exclude potential subjects? Are the comparison groups similar in all areas except for the exposure, treatment, etc. of interest? Have all other explanations for the results been considered? How is the effect of exposure/treatment /etc. being measured? Reliability The second guiding principle for critical literature review is reliability, which focuses on the results of the study. One important consideration is the strength of the association between the exposure or treatment and the outcome of interest or the size of the difference in outcomes between the two groups. In his 1965 seminal article, Sir Austin Bradford Hill (Ref. #51) described nine causal criteria. First on his list was strength of association. Typically, the stronger the association between exposure/treatment and outcome the more likely there exists a "real" causal relationship. Measures of association should be published to quantify this causal relationship between exposure/disease, treatment/outcome. Depending on the type of study, this could be an odds ratio (OR), risk ratio (RR), etc. If any adjustments were made in the analysis due to other factors like race, gender, etc., observe if the measures of association changed. For instance, if the association dramatically decreased or disappeared entirely (OR went from 2.6 to 1.1), then there may not be a "real" difference between the two groups and the treatment/exposure did not provide any benefit or harm. The clinical significance of the difference between the two groups should also be taken into account. For example, if there is only a half day difference between the average length of cessation (time before starting smoking again) following two different smoking cessation interventions, it may be a statistically significant difference but probably not clinically meaningful. The study's authors should also discuss other comparable studies and why or why not they had different outcomes. If prior research has shown no association between exposure/treatment and outcome, and the current study did, the authors need to offer an explanation for the difference between study results. Authors also need to publish the level of significance and uncertainty of their statistical test results. Two common measures are p-values and 95% confidence intervals. For instance, if two groups are participating in different smoking cessation interventions and the researchers are measuring the effect each intervention has on the average length of smoking cessation, then a p-value reported as 0.01 indicates that the difference in average length of smoking cessation times between the two interventions has a 1% chance of resulting from something other than the different interventions themselves. Another way to think about it is that there is a 99% chance that the measured difference is due to the interventions. The standard p-value is generally specified as being equal to or less than 0.05. Anything greater than 0.05 is considered statistically insignificant. Confidence intervals provide a range of possible values. For example, if a study was conducted to examine the effect of hypnosis on smokers who were trying to quit smoking and the outcome was an average increase in quit time by 30 days for the hypnotized participants. However, the 95% confidence interval calculated by the authors ranged from 0-180 days, indicating that the true increase in cessation length falls somewhere between zero and a maximum effect of 180 days. Therefore, the authors should conclude that there was no significant increase in quit time for the hypnotized participants compared to the other unhypnotized patients. Generally, 95% confidence intervals that include the null are considered statistically insignificant because there is a chance that there is no effect of treatment/exposure. Assessing Reliability of a Study What is the strength of association between the exposure/treatment and outcome? What is the clinical significance of an association? How precise were the measurements? Do the authors consider other evidence or explanations for their findings? Applicability Last but certainly not least, you need to reflect on the applicability of the reviewed study findings and your patients. Think about how the study question relates to your work. Consider if the study population and your patient population are similar enough for the results to be useful to you i.e., share similar gender, age, racial/ethnic and socioeconomic distributions. If the study population is middle-aged, white men from the suburbs and your patient population is mostly African-American teenage girls from an urban center, the study's intervention technique may not be appropriate for your patients. Findings from studies that have many exclusionary criteria may be so specific to a particular population that the findings are inapplicable to the general population, or more importantly, not relevant to your own patients (see inset). Also, think about what incentives were used to recruit participants, persons who volunteer may be different from the population you work with. Finally, keep in min how was the study population was chosen. The gold standard is a random sample but more likely a convenience sample, such as hospital patients or volunteers, was used, which may lead to findings that are not as relevant to the general population or to your own patients. You should also consider the impact of the reported benefits and harms in your patients. If your patients are mostly uninsured, a smoking cessation drug that costs $500 and would not be a practical option. These considerations are only a brief introduction to what one needs to think about when reading a research article. Please use the list of citations included below for more in-depth information on critical reviewing. Thoughtful assessment of scientific literature is a skill that is learned and practiced over time. The more you read, think and question, the better! Assessing Applicability of a Study What are the similarities and differences between the study population and my patients? What are the benefits and harms? On-Line Citations Critical Appraisal Skills Programme at the Public Health Resource Unit. England, 2006. http://www.phru.nhs.uk/Pages/PHD/resources.htm (accessed April 2008). Barber G. Critically Reviewing the Literature. University of Plymouth, School of Health Professions, Millbrook House, Exeter, 2004. http://www2.plymouth.ac.uk/millbrook/rsources/sealit/critical.htm (accessed April 2008). Lunsford TH, Lunsford BR. How to Critically Read a Journal Research Article. JPO. 1996;8:24-31. Simon S. How to read a medical journal article. Kansas City, MO, 1997. http://www.childrensmercy.org/stats/journal/jour2003-07.htm (accessed April 2008). University of Sheffield, School of Health and Related Research. Critical appraisal and using the literature. http://www.shef.ac.uk/scharr/ir/units/critapp/resources.htm (accessed April 2008). Article and Book Citations The Journal of the American Medical Association Barratt A, Irwig L, Glasziou P, Cumming RG, Raffle A, Hicks N, Gray JA, Guyatt GH. Users' guides to the medical literature. XVII. How to use guidelines and recommendations about screening. JAMA 1999;281:2029-34. Bucher HC, Guyatt GH, Cook DJ, Holbrook A, McAlister FA. Users' guides to the medical literature. XIX. Applying clinical trial results. A. How to use an article measuring the effect of an intervention on surrogate end points. JAMA 1999;282:771-8. Dans AL, Dans LF, Guyatt GH, Richardson S. Users' guides to the medical literature. XIV. How to decide on the applicability of clinical trial results to your patient. JAMA 1998;279:545-9. Drummond MF, Richardson WS, O'Brien BJ, Levine M, Heyland D. Users' guides to the medical literature. XIII. How to use an article on economic analysis of clinical practice. A. Are the results of the study valid? JAMA 1997;277:1552-7. Giacomini MK, Cook DJ. Users' guides to the medical literature. XXIII. Qualitative research in health care A. Are the results of the study valid? JAMA 2000;284:357-62. Giacomini MK, Cook DJ. Users' guides to the medical literature. XXIII. Qualitative research in health care B. What are the results and how do they help me care for my patients? JAMA 2000;284:478-82. Guyatt GH, Haynes RB, Jaeschke RZ, Cook DJ, Green L, Naylor CD, Wilson MC, Richardson WS. Users' guides to the medical literature. XXV. Evidence-based medicine; principles for applying the Users' Guides to patient care. JAMA 2000;284:1290-6. Guyatt GH, Naylor CD, Juniper E, Heyland Dk, Jaeschke R, Cook DJ. Users' guides to the medical literature. XII. How to use articles about health-related quality of life. JAMA 1997;277:1232-7. Guyatt GH, Rennie D. Users' guide to the medical literature. JAMA. 1993;270:2096-7. Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? JAMA 1993;270:2598-601. Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? JAMA 1994;271:59-63. Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DJ, Cook RJ. Users' guides to the medical literature. IX. A method for grading health care recommendations. JAMA 1995;274:1800-4. Guyatt GH, Sinclair J, Cook DJ, Glasziou P. Users' guides to the medical literature. XVI. How to use a treatment recommendation. JAMA 1999;281:1836-43. Hayward RSA, Wilson MC, Tunis SR, Bass EB, Guyatt G. Users' guides to the medical literature. VIII. How to use clinical practice guidelines. A. Are the recommendations valid? JAMA 1995;274:570-4. Hunt DL, Jaeschke R, McKibbon KA. Users' guides to the medical literature. XXI. Using electronic health information resources in evidence-based practice. JAMA 2000;283:1875-9. Jaeschke R, Guyatt G, Sackett DL. Users' guides to the medical literature. III. How to use an article about a diagnostic test. A. Are the results of the study valid? JAMA 1994;271:389-91. Jaeschke R, Guyatt G, Sackett DL. Users' guides to the medical literature. III. How to use an article about a diagnostic test. B. What were the results and will they help me in caring for my patients? JAMA 1994;271:703-7. Laupacis A, Wells G, Richardson WS, Tugwell P. Users' guides to the medical literature. V. How to use an article about prognosis. JAMA 1994;271:234-7. Levine M, Walter S, Lee H, Haynes T, Holbrook A, Moyer V. Users' guides to the medical literature. IV. How to use an article about harm. JAMA 1994;271:1615-9. McAlister FA, Laupacis A, Wells GA, Sackett DL. Users' guides to the medical literature. XIX. Applying clinical trial results. B. Guidelines for determining whether a drug is exerting (more than) a class effect. JAMA 1999;282:1371-7. McAlister FA, Straus SE, Guyatt GH, Haynes RB. Users' guides to the medical literature. XX. Integrating research evidence with the care of the individual patient. JAMA 2000;283:2829-36. McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS. Users' guides to the medical literature. XXII. How to use articles about clinical decision rules. JAMA 2000;284:79-84. Naylor CD, Guyatt GH. Users' guides to the medical literature. X. How to use an article reporting variations in the outcomes of health services. JAMA 1996;275:554-8. Naylor CD, Guyatt GH. Users' guides to the medical literature. XI. How to use an article about a clinical utilization review. JAMA 1996;275:1435-9. O'Brien BJ, Heyland D, Richardson WS, Levine M, Drummond MF. Users' guides to the medical literature. XIII. How to use an article on economic analysis of clinical practice. B. What are the results and will they help me in caring for my patients? JAMA 1997;277:1802-6.  Oxman AD, Cook DJ, Guyatt GH. Users' guides to the medical literature. VI. How to use an overview. JAMA 1994;272:1367-71. Oxman AD, Sackett DS, Guyatt GH. Users' guides to the medical literature. I. How to get started. JAMA 1993;270:2093-5. Randolph AG, Haynes RB, Wyatt JC, Cook DJ, Guyatt GH. Users' guides to the medical literature. XVIII. How to use an article evaluating the clinical impact of a computer-based clinical decision support system. JAMA 1999;282:67-74. Richardson WS, Detsky AS. Users' guides to the medical literature. VII. How to use a clinical decision analysis. A. Are the results of the study valid? JAMA 1995;273:1292-5. Richardson WS, Detsky AS. Users' guides to the medical literature. VII. How to use a clinical decision analysis. B. What are the results and will they help me in caring for my patients? JAMA 1995;273:1610-3. Richardson WS, Wilson MC, Guyatt GH, Cook DJ, Nishikawa J. Users' guides to the medical literature. XV. How to use an article about disease probability for differential diagnosis. JAMA 1999;281:1214-9. Richardson WS, Wilson MC, Williams JW Jr, Moyer VA, Naylor CD. Users' guides to the medical literature. XXIV. How to use an article on the clinical manifestations of disease. JAMA 2000;284:869-75. Wilson MC, Hayward RS, Tunis SR, Bass EB, Guyatt G. Users' guides to the medical literature. VIII. How to use clinical practice guidelines. B. Will the recommendations help me in caring for my patients? JAMA 1995;274:1630-2. British Medical Journal Greenhalgh T, Taylor R. How to read a paper: Papers that go beyond numbers (qualitative research). BMJ. 1997;315:740-3. Greenhalgh T. How to read a paper: Assessing the methodological quality of published papers. BMJ. 1997;315:305-8. Greenhalgh T. How to read a paper: Getting your bearings (deciding what the paper is about). BMJ. 1997;315:243-6. Greenhalgh T. How to read a paper: Papers that report diagnostic or screening tests. BMJ. 1997;315:540-3. Greenhalgh T. How to read a paper: Papers that report drug trials. BMJ. 1997;315:480-3. Greenhalgh T. How to read a paper: Papers that summarise other papers (systematic reviews and meta-analyses). BMJ. 1997;315:672-5. Greenhalgh T. How to read a paper: Papers that tell you what things cost (economic analyses). BMJ. 1997;315:596-9. Greenhalgh T. How to read a paper: Statistics for the non-statistician. I: Different types of data need different statistical tests. BMJ. 1997;315:364-6. Greenhalgh T. How to read a paper: Statistics for the non-statistician. II: "Significant" relations and their pitfalls. BMJ. 1997;315:422-5. Greenhalgh T. How to read a paper: The Medline database. BMJ. 1997;315:180-3. Other Citations Bradford Hill A, The Environment and Disease: Association or Causation? Proceedings of the Royal Society of Medicine. 1965;58:295-300. Greenhalgh T. How to Read a Paper: The Basics of Evidence-based Medicine, 3rd edition. London, UK: Blackwell BMJ Books, 2006. Hennekens CH, Buring JE. Epidemiology in Medicine. Boston/Toronto: Little, Brown and Company, 1987. Kelsey JL, Whittemore AS, Evans AS, Thompson WD. Methods in Observational Epidemiology, 2nd edition. New York, NY: Oxford University Press, Inc., 1996. Last, JM (ed.). A Dictionary of Epidemiology, 4th edition. New York, NY: Oxford University Press, Inc., 2001. Ritchey J, Goldstein M, Dobbs D, Thomas S. The Epidemiology of Cigarette Smoking in Chicago. Chicago, Illinois; Chicago Department of Public Health Epidemiology Program. 2004. Rothman KJ, Greenland S. Modern Epidemiology, 2nd edition. Philadelphia, PA: Lippincott-Raven Publishers, 1998. Sackett DL, Rosenberg WM, Gray JAM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312:71-2.