Chapter 2. Decision Making: Use of Evidence-Based Medicine

Making thoughtful decisions about patient care is at the core of a physician’s responsibilities. Multiple factors feed into clinical decision making: a physician’s experience, the values and preferences of the patient and patient’s family, socioeconomic factors, resources available in the community, and the best evidence that exists at the time. Physicians may have an understandable tendency to be influenced by a past missed diagnosis or an emotional case or to ignore information that conflicts with preconceived theories. While recent emphasis is on the importance of peer-reviewed evidence, a management plan based solely on evidence from the literature may fail if not accepted by the family. Peer-reviewed evidence is therefore a necessary but insufficient basis for clinical decision making.

Satisfying the need for evidence typically comes in three forms: keeping up to date, building expertise in a specific field, or answering questions related to the care of specific patients. Patient-centered evidence, the focus of this chapter, demands a streamlined approach. For most clinicians, it is not practical to spend hours reading dozens of studies to answer every clinical question that arises. The clinician must prioritize the most important patient-centered questions. A reasonable approach is as follows, using the mnemonic PARCA1:

1. 1. Prioritize the most important patient-centered questions.

   2. Assemble the four elements of an answerable question (PICO, described shortly).

   3. Retrieve the evidence.

   4. Critically appraise the evidence.

   5. Apply the evidence to the particular patient.

The following vignette demonstrates the application of this process:

The parents of a 4-month-old are worried about their infant’s frequent emesis after feedings. They tried positioning her upright after feedings, but she has continued to vomit, and her weight gain has been inadequate. Would a trial of a thickened formula help?

The physician must first decide whether it is important enough to warrant the time and effort of a literature review. If the question pertains to the mechanism of action of medications or the pathophysiology or signs and symptoms of a disease (a background question), the answers are likely found in textbooks or review articles. If the question aims to ascertain which therapy is most effective or whether a diagnostic test is useful (a foreground question), the answer is likely found in systematic reviews or original journal articles.

For many foreground questions, clinical guidelines, predigested evidence, or advice from colleagues, including subspecialists, is sufficient. For such questions, the next steps are formulating a specific clinical question (step 2) and applying the answer to the particular patient (step 5). For other questions, proceeding through all the PARCA steps is warranted.

The preliminary question (Will thickened formula help?) must be converted into a more complete, answerable format. One method, from Sackett,2 is summarized by PICO:patient or problem, intervention, comparison, outcome. The time invested in framing clinical questions in such a way is rewarded in efficiency by weeding out inapplicable evidence sources, irrelevant comparisons, and clinically unimportant outcomes. The focus should be on patient-centered outcomes (eg, hospitalization rate, length of stay, decrease in pain); studies with disease-oriented outcomes (improvement in laboratory values) can be a guide to future investigation but may have limited application to patient care. Reframing the preliminary question in PICO format produces the following: In children with gastroesophageal reflux and poor weight gain (P), does thickened formula (I), compared with positioning alone (C), result in improved weight gain (O)?

Once the question is clearly stated, a practical, focused, and thorough search strategy begins by seeking systematic reviews, such as in the Cochrane Library. In addition to saving time, well-done systematic reviews can find statistical significance in the aggregate that was not reached in any individual study because they pool the results of multiple studies. A search can then be conducted for original studies published since the review was completed. The National Library of Medicine, on its PubMed Web site, has a portal (http://pubmedhh.nlm.nih.gov/nlm/pico) that permits clinicians to directly enter each part of the PICO question. Specifying a 4-month-old infant with reflux as the patient and problem and “thickened formula” as the intervention yields more than 2 dozen articles. To efficiently and effectively determine which articles are worth reading, clinicians can survey the article titles (and occasionally the abstracts) and compare them to the PICO question. Among the retrieved titles are the following:

1. 1. Effect of cereal-thickened formula and upright positioning on regurgitation, gastric emptying, and weight gain in infants with regurgitation.

   2. Efficacy of a pre-thickened infant formula: a multicenter, double-blind, randomized, placebo-controlled parallel group trial in 104 infants with symptomatic gastroesophageal reflux.

   3. Effects of thickened feeding on gastroesophageal reflux in infants: a placebo-controlled crossover study using intraluminal impedance.

   4. Smaller volume, thickened formulas in the management of gastroesophageal reflux in thriving infants.

Since the patient population of interest is refluxing infants with poor weight gain, article 4 (study of thriving infants) can be skipped. The desired outcome measure is weight gain, so studies that use disease-oriented outcomes, such as article 3 (intraluminal impedance), can also be skipped. Using this strategy, the number of studies to be appraised is reduced to a manageable number.

The “best” study type depends on the question. Questions about therapy are best addressed in systematic reviews and randomized control trials. To determine the efficacy of a diagnostic test, a prospective, blinded comparison trial is most useful. Questions about prognosis are best addressed in cohort studies, followed, in order of value, by case-control studies and case series; randomized control trials about prognosis or harm can be difficult to conduct, may have ethical problems, or may not even be feasible if the harm event is rare.3

In cohort studies (eFig. 2.1), investigators start by finding a group of children with a particular exposure and a group without such exposure. The groups are then followed to see whether disease develops. If a greater proportion of exposed than unexposed children develop disease, an association between the exposure and disease exists. Because the individuals in the groups are not randomly assigned, cohort trials are not as useful as randomized control trials in determining causation. The strength of associations in cohort trials is limited by the possibility of confounders or extraneous variables that may influence the association being studied.

In case-control studies (eFig. 2.2), investigators begin with a group of children with a disease (the cases) and then match them as closely as possible to individuals who do not have the disease (the controls). The proportion of each group with the exposure of interest is determined. If there is a significant difference (eg, if 90% of the case group and only 20% of the control group were exposed), then there may be an association between the exposure and the disease. Case-control studies are practical for rare diseases (Imagine the size of a cohort study needed for a disease with an incidence of 1 per 1 million.). The disadvantage of case-control studies is the problem of recall bias (past exposures must be remembered) and the inability to calculate incidence or prevalence of disease.

A third type of observational study is the cross-sectional study (eFig. 2.3). A population is surveyed at one moment in time for both the presence of disease and the presence of exposure. The prevalence of disease in exposed children is then compared with that of nonexposed children. Cross-sectional studies are easier to perform than cohort or case-control studies. While they cannot establish causation, they can give useful information about prevalence and can suggest associations that can then be studied in depth using a randomized control trial or stronger observational study.

The fundamental determination is whether a study is valid (ie, what is claimed as true is actually true). Seven questions aid in determining the validity of a study:

1. 1. Were the patients randomized, and was the randomization concealed? Randomization is the only way to equalize study groups for known and unknown confounding factors. A cohort study with carefully matched groups might achieve the former but not the latter.

   2. Were treatment and control patients similar at the start of the study (ie, did randomization work)? Most studies provide a table of demographic data and other potentially relevant characteristics of the individuals in the study groups. Comparability of the groups is necessary to assure that randomization was effective.

   3. Were patients analyzed in the groups to which they were randomized (intent to treat)? What should be done with participants who do not follow the assigned protocol? If 10% of children who were supposed to get thickened formula received regular formula instead, should they be counted in the treatment group? In the control group? Dropped from the study? The third option would provide a solution except for two problems. In real practice, noncompliance is common. The clinically important result is how a potential treatment performs in real-world use. Moving nonadherent participants into a different group might reduce the benefit of randomization, since a factor relevant to the study’s outcome also might affect compliance with the intervention (eg, children with the worst reflux may have given up on the thickened formula and tried something else).

   4. Were clinicians, patients, and study personnel blinded to the group assignment? If clinicians or patients know which intervention is being received, they might introduce bias into treatment plans or assessments and either exaggerate or minimize the true treatment effect.

   5. How many patients were lost to follow-up? Was the length of follow-up reasonable? If a large number of patients drop out of a study, it is difficult to assess the outcome in each group fairly (especially if, for instance, the patients dropped out because the new therapy was not working or had intolerable side effects). Some loss to follow-up is unavoidable. One way of assessing whether the loss is acceptable is by using a worst-case scenario analysis. The authors could assume the worst case, that all of the children lost to follow-up did poorly, and recalculate the results using that assumption. If the results still indicate a benefit, the conclusion is strengthened. As a rule of thumb, loss to follow-up of more than 20% significantly reduces the validity of the study.

   6. Is it possible that the results could have been due to chance (P > .05)? Another threat to the validity of a study is that the difference between groups is not real but is due simply to chance (a type I error). The probability of this error is expressed as P. By convention, up to a 5% likelihood of the results occurring by chance (P ≤ .05) is accepted.

   7. Is it possible that results could have been skewed by a small sample size? (When does n matter?) In a “positive” study (ie, a difference is demonstrated between intervention and control groups), statistical significance (P < .05) assures that the sample size (n) was adequate. In a “negative” study (ie, no statistically significant difference between groups), it is possible that a real difference between the study groups was missed (a type II error, or β). To decide if the sample size was large enough to avoid this error, the clinician must examine the methods section of the study for an explanation of how the sample size was determined. The authors should specify their sample size calculation based on the difference in response they want to detect, the estimated success rate in the control group, and the minimum acceptable probability of correctly finding a difference (1 – β, also known as power and generally set at 80% by convention). Using these assumptions, the authors can calculate—and should state—the sample size needed to demonstrate a difference. Additional measures are often useful beyond statistical significance: relative risk, odds ratio, absolute risk reduction, and the number needed to treat (the inverse of absolute risk reduction).

Many published studies provide an incorrect statistic (eg, relative risk is given in a study in which it is not valid to calculate), so it is important that clinicians aspiring to practice evidence-based medicine understand which statistics can be calculated from which types of studies and how to interpret those statistics (eTable 2.1).

The relative risk helps clinicians decide the degree to which a new treatment shows benefit (or harm). It is a comparison of the risk of disease in the experimental group (or, in a cohort trial, the exposed children) with the risk in the control group (or nonexposed). Consider a fictitious randomized control trial of a new vaccine for the prevention of malaria in sub-Saharan Africa.

The risk of malaria in the experimental group is 400/1220 = 33%. The risk in the control group is 800/1160 = 69%. The relative risk is thus 33%/69% = 0.48%. If the relative risk is less than 1, the new treatment is beneficial; if it is greater than 1, the new treatment causes harm. A relative risk of 1 indicates that the treatment makes no difference. If the example had been a cohort study instead of a randomized control trial, “exposed” and “nonexposed” would be substituted for “treatment group” and “control group;” the relative risk would indicate whether exposure increased the risk of disease. Relative risk should not be used for cohort or cross-sectional (prevalence) studies because it requires knowing the number of new cases (incidence) in each of the two groups. In our example, the children in the malaria study started disease free and were followed to determine how many contracted malaria. In a case-control study, however, the starting point would be a group that already has the disease and another that does not; because children enter the study already having the disease, the rate of new cases cannot be determined.

To approximate relative risk in case-control trials, the odds ratio is used. Consider the malaria study done as a case-control study, starting with a group of children with malaria and a matched group that is malaria free.

The proportion of each group that received the vaccine is measured. We must then convert from probability to odds. The probability (P) that a child with malaria (a case) received the vaccine is

P/(1 – P) = (400 ÷ 1200)/(1 – P) = 33%/67% = 0.5.

The odds that a malaria-free child (control) received the vaccine are

P/(1 – P) = (820 ÷ 1180)/(1 – P) = 70%/30% = 2.3.

The odds ratio is

Odds that a case got vaccine/Odds that a control got vaccine = 0.5/2.3 = 0.22.

Note that the relative risk (0.48) and the odds ratio (0.22) are quite different. The odds ratio is a good estimate of the relative risk only when the disease is a rare one. The high prevalence of malaria in the example explains the divergence between the two statistics and is a reminder to interpret odds ratio with caution.

There are instances in which the relative risk can also be misleading. Here, absolute risk reduction and number needed to treat become valuable. Consider the malaria study done in a much lower prevalence population (say, South America):

The relative risk is (4 ÷ 1220)/(8 ÷ 1160) = 0.48, identical to our African study. The study authors could still make the claim that the vaccine reduces the risk of malaria by 52%. It is clear, however, that the decision about whether to vaccinate is very different for South American doctors than for their African counterparts. The absolute risk reduction and the number needed to treat highlight this difference. The absolute risk reduction is the risk of disease in the control group minus the risk in the treatment group.

In the South American study, the absolute risk reduction is 8/1160 – 4/1220 = 0.69% – 0.33% = 0.36%.

The number needed to treat is the reciprocal (inverse) of the absolute risk reduction, so the number needed to treat is 1/0.36% = 278.

The number needed to treat indicates that 278 children would need to be vaccinated to prevent one case of malaria.

In the African study, the absolute risk reduction is 800/1160 – 400/1160 = 69% – 33% = 36%. The number needed to treat is 1/36% = 2.78. While the relative risk is the same, the number of children who would have to be vaccinated to prevent 1 case of malaria is much higher in South America, so a consideration of risks and costs is very different. Thus, unless the side-effect profile is intolerable, the malaria vaccine makes sense in Africa, but the decision is not as clear in South America. Reliance solely on relative risk would have masked this clinically important distinction.

Note also the odds ratio in the low-incidence South American trial is a good estimate of relative risk. If done as a case control trial, the odds that a case received the vaccine (P) are

P/(1 – P) = (4 ÷ 12)/(1 – P) = 33%/67% = 0.5.

The odds that a malaria-free child (control) received the vaccine are

P/(1 – P) = (1216 ÷ 2368)/(1 – P) = 51%/49% = 1.04.

The odds ratio is identical to the calculated relative risk:

Odds that a case got vaccine/Odds that a control got vaccine = 0.5/1.04 = 0.48.

If we decide the study results are valid, we must decide whether the results are applicable to our patient. Even if the outcome is relevant and statistically significant, it may still not be clinically meaningful. If, for example, a 1-oz. weight gain is demonstrated to be statistically significant in a very large study of infants with reflux treated with thickened formula, the clinician and family must decide whether such a modest gain is meaningful and whether it is worth the costs and risks of the intervention.

Proceeding through the PARCA steps to determine the quality and applicability of the evidence is not the end of the process. The physician must return to the decision-making philosophy discussed at the beginning of this chapter. The evidence is only as good as the criteria used in the PICO formulation: In this particular case, how certain is the physician that reflux is the cause of this infant’s poor weight gain? Current best evidence provides one piece of the ultimate management plan for the patient. Clinical judgment remains essential.