Health Affairs, 25, no. 4 (2006): w272-w275 (Published online 6 June 2006) doi: 10.1377/hlthaff.25.w272 © 2006 by Project HOPE	New Online   House Health Reform Bill   Paying for Reform   Vetting AHIP's Report   HIV/AIDS Costs   Brief: Insurance Reform   HA Blog Top 10

This Article Abstract Reprint (PDF) Submit a response to this article Alert me when this article is cited Alert me when Comments are posted Alert me if a correction is posted Services E-mail this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to My Personal Archive Download to Citation Manager Reprints & Permissions Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Gibson, M. Articles by Santa, J. Search for Related Content PubMed PubMed Citation Articles by Gibson, M. Articles by Santa, J. Related Collections Access To Care Health Reform Insurance Coverage Managed Care - Medicaid Medicaid Pharmaceuticals State/Local Issues Consumer Issues Evidence-Based Medicine

Web Exclusives

PERSPECTIVE

The Drug Effectiveness Review Project: An Important Step Forward

Abstract

 
Peter Neumann’s paper on the Drug Effectiveness Review Project (DERP) is a constructive if incomplete point of departure for discussing the work done by the project and the use of that work by decisionmakers in states and elsewhere. This Perspective attempts to establish the proper context for judging the DERP by comparing its product and processes with those commonly produced and used by industry and other parties. It also provides a direct response to the criticisms of the project noted by Neumann.

An old tactic for opposing meritorious change without explicitly defending the status quo is to compare the initiative in question to a hypothetical ideal rather than current conditions. This tactic is used extensively by the DERP’s detractors. The DERP’s supporters argue that although not perfect, its approach is much superior to the industry’s preferred process for dispensing information on drugs’ effectiveness and safety. At its core, the debate over the DERP centers on the difference between an approach that relies on physicians and patients attempting to unravel mounds of marketing data selected specifically to increase the market share of individual drugs versus one in which physicians, consumers, and policymakers alike have access to a systematic assessment of effectiveness and safety of the drugs in question, based on the best-quality evidence available. The former selects evidence that supports a foregone conclusion (you should buy my drug) while ignoring or concealing evidence that does not.² The latter asks what the existing research evidence shows when comparing one drug with another, selects evidence based on its objectivity and quality, and analyzes that evidence in a transparent process based on international consensus.

Criticisms of the DERP. Criticism: DERP reviewers assume that drugs within a class are equivalent and that no evidence of a difference is evidence of equality. Response: This is untrue, and detractors have not cited a single instance in DERP reports where this is the case. The DERP reviewers carefully report only what the evidence shows or doesn’t show; they draw no conclusions beyond that. If evidence of equivalence or difference exists, the reviewers make clear what that evidence is. If no such evidence exists, then the DERP simply reports this lack of evidence. By comparison, drug marketing successfully promotes substantial price differences among medications without credible evidence of superiority.

Criticism: The DERP favors randomized controlled trials (RCTs) over observational studies and other data sources. Response: This is true. In the science of evidence reviews, well-done RCTs and systematic reviews of RCTs are seen as the gold standard of clinical studies.³ While the project does use observational studies to evaluate the safety of medications because RCTs are often too short to adequately assess side effects that develop during long-term use, a well-done RCT is considered the most objective and reliable form of evidence. We return to this point later.

Criticism: The DERP doesn’t use the proper analytical tools for comparing disparate studies of drugs within classes. Response: DERP reports use the statistical methods generally used by systematic reviewers. The DERP and its researchers continue to participate in ongoing discussions designed to further refine the process for systematic reviews of drug classes and other subjects. Its sources and analyses are fully disclosed. This openness contrasts favorably to commercial models that are often not sufficiently transparent for others to critique their conclusions.

Criticism: The DERP does not promote the true value of drugs because it doesn’t integrate cost-effectiveness analyses. Response: The DERP compares drugs within classes. It assumes that drugs have value, and it seeks to provide only information that will help determine if one is a better value than another. The DERP leaves consideration of cost to others, but its reports can serve as a highly credible baseline for cost-effectiveness analysis and compare favorably to industry analyses that might use only one study of questionable quality to establish effectiveness or whose methods and data are proprietary and not disclosed.

Criticism: The DERP should be more transparent. Response: The DERP is among the most transparent research products available today; it compares favorably to an industry that is notorious for being unwilling to prospectively disclose and describe the research it is initiating and conducting, and then releases only selected results from those studies. The DERP solicits and considers public comment on its research questions before beginning work on a report. It solicits, considers, and discloses on request any evidence received from drug manufacturers. It posts draft reports for public comment and makes those comments available on request. Comparing draft reports with the final versions makes clear how the comments were addressed. Its reports list all studies found to be potentially relevant to the questions under review, and they identify studies included and excluded and the reasons for doing so. Almost all of its participants make decisions in public settings, with public input encouraged.⁴

Criticism: The DERP doesn’t submit its work to peer-reviewed journals for further scrutiny and publication. Response: DERP reviewers based in the Evidence-based Practice Centers (EPCs) have already published four peer-reviewed papers based on DERP reports, and several more are in process.⁵ The DERP has no control over the content of any publication from the EPCs.

Criticism: The DERP doesn’t sufficiently involve doctors with relevant expertise and patient advocates, and it shouldn’t exclude experts with industry ties. Response: The DERP does not apologize for having a conflict-of-interest policy that forbids its reviewers from having a financial relationship with the companies whose products they are evaluating. However, the DERP solicits and considers input from researchers and clinicians with relevant expertise and from patient advocates by requesting comments on its research questions and through its peer review of and comment solicitation on draft reports. The DERP routinely asks advocacy organizations to recommend peer reviewers (including those with industry ties) to critique its reports. The DERP is convinced that this is the proper balance for creating objective and useful reports. Growing concerns over conflicts of interest in the current widely accepted intermingling of researchers, clinicians, and industry have led to calls for reform, most recently for academic medical centers to change their current relationships with drug companies and other health product manufacturers.⁶

Criticism: Although DERP reports are of high quality, they can be misused to restrict access to medications. Response: There is an interesting irony in complimenting the DERP on the quality of its information, then criticizing it because the information could be misused. This position seems to imply that inferior information is less likely than high-quality information to be misused. Moreover, even though this fear is widely cited, Neumann was unable to document where this has actually happened. Given the fact that DERP reports have been extensively used by states during the past three years, it appears that this fear might be overstated. Moreover, DERP reports have helped restrict inappropriate access to drugs. Two years before Vioxx was removed from the market, many states kept it off of their preferred drug lists (PDLs) because the DERP reported evidence of its significant cardiac risks. DERP reports also provide a constructive alternative to inappropriate off-label use promotion, such as the Neurontin case, which resulted in a substantial legal settlement between industry and the state attorneys general.⁷

Questions of method and scope. Neumann touches briefly on disagreements related to methodology. If anything, this topic could use more exploration and will no doubt be the subject of continuing give and take among researchers, industry representatives, and decisionmakers. Among the most challenging of these issues is determining how to judge the quality and usefulness of studies other than RCTs. While RCTs, and systematic reviews of RCTs, are still seen as the most reliable of methods for judging the benefits of specific interventions, there is growing interest in the potential role of other forms of research, such as observational studies, case series, or even patients’ preferences. DERP participants have invested resources in exploring whether or not available observational studies can add value to their assessments of effectiveness. Space does not allow detailed coverage of this debate, but the DERP is committed to using its conclusions to continually refine and improve its reports.

Finally, Neumann devotes substantial time asserting the need for DERP reports to "consider economic evidence more forthrightly and holistically" to reduce mistrust of its reports and an undue focus on cost if cost and effectiveness are considered in sequence as opposed to simultaneously. Even though cost-effectiveness analyses have their place in the world, there are good reasons that the DERP does not delve into them. Chief among these reasons is the methodological uncertainty of cost-effectiveness studies. There is an ongoing discussion related to the methodology of systematic reviews, but that discussion is focused on the margins of those methods and builds upon an international consensus of what makes a good systematic review. Cost-effectiveness analyses can boast no such widely recognized consensus on method, and until one emerges, DERP participants are hesitant to invest their limited resources in something as yet so poorly defined.⁸ The DERP’s goal is modest: Get the first step right. Once the evidence step is complete, credible, and transparent, it will be easier to add economic analysis.

It is also important to remember that DERP reports compare drugs within a class, not drugs with other treatments. In this case, cost-effectiveness is easily calculated when medications are similar because the less costly drug is by definition the most cost-effective. When there is no evidence of a difference between drugs, cost-effectiveness analyses are worthless because there is no effect difference to calculate. When there are substantial differences among drugs, DERP participants have elected not to have researchers make a determination that they believe is more appropriately considered by local decision-making processes, which can account for unique realities in various locales. The reasons for this include the fact that drug pricing, benefit designs, and eligibility standards vary widely from state to state. Hence, the effectiveness trade-offs to these important policy considerations vary greatly, and an aggregate cost-benefit analysis would not be feasible.

Finally, Neumann’s assertion that the DERP’s decision not to include cost-effectiveness analyses in its studies "reveals a society still unable to consider economic factors openly in evidence reviews" seems an unduly harsh indictment, given that most of the members of the DERP consider cost in their public decision-making processes and are prevented from being even more explicit by federal law that prohibits disclosure of drug prices paid by Medicaid. Rather than arguing over whether cost should be considered simultaneously or in sequence, a more effective starting point would be to increase the transparency of economic factors within Medicaid and to increase disclosure of research that is now shrouded in claims of proprietary privilege, so that more-informed calculations of cost and benefit might be made, regardless of when they are factored into a decision.

IT IS REASONABLE FOR THE DERP to have supporters and detractors. Its participants have made the decision to use their scarce resources to improve the quality of evidence available and to factor that evidence into their decision making. In doing so, they hold themselves to a high methodological standard and hope that their efforts continue to encourage an open and substantive debate of how best to use evidence in the ongoing discussions required to continually improve both practice and policy. Clearly Neumann’s paper has helped further that debate.

Editor's Notes

 
Mark Gibson (gibsomar{at}ohsu.edu) is deputy director of the Center for Evidence-based Policy, Oregon Health and Science University, in Portland. John Santa is the medical director of the Drug Effectiveness Review Project (DERP) there.

NOTES

Top NOTES

 

1. P.J. Neumann, "Emerging Lessons from the Drug Effectiveness Review Project," Health Affairs 25 (2006): w262–w271 (published online 6 June 2006; 10.1377/hlthaff.25.w262).[CrossRef][Medline]
2. P. Juni, A.W. Rutjes, and P.A. Dieppe, "Are Selective COX 2 Inhibitors Superior to Traditional Non Steroidal Anti-inflammatory Drugs?" British Medical Journal 324, no. 7349 (2002): 1287–1288.[Free Full Text]
3. D.L. Sackett et al., "Evidence Based Medicine: What It Is and What It Isn’t," British Medical Journal 312, no. 7023 (1996): 71–72.[Free Full Text]
4. Oregon Health and Science University, "Drug Effectiveness Review Project: Description," http://www.ohsu.edu/drugeffectiveness/description/index.htm (accessed 12 May 2006).
5. S. Carson, M.S. McDonagh, and K. Peterson, "A Systematic Review of the Efficacy and Safety of Atypical Antipsychotics in Patients with Psychological and Behavioral Symptoms of Dementia," Journal of the American Geriatrics Society 54, no. 2 (2006): 354–361[CrossRef][Web of Science][Medline]; R. Chou, K. Peterson, and M. Helfand, "Comparative Efficacy and Safety of Skeletal Muscle Relaxants for Spasticity and Musculoskeletal Conditions: A Systematic Review," Journal of Pain and Symptom Management 28, no. 2 (2004): 140–175[CrossRef][Web of Science][Medline]; G. Gartlehner et al., "Discontinuation Rates for Selective Serotonin Reuptake Inhibitors and Other Second-Generation Antidepressants in Outpatients with Major Depressive Disorder: A Systematic Review and Meta-analysis," International Clinical Psychopharmacology 20, no. 2 (2005): 59–69[CrossRef][Web of Science][Medline]; and R.A. Hansen et al., "Efficacy and Safety of Second-Generation Antidepressants in the Treatment of Major Depressive Disorder," Annals of Internal Medicine 143, no. 6 (2005): 415–426.[Abstract/Free Full Text]
6. T.A. Brennan et al., "Health Industry Practices That Create Conflicts of Interest: A Policy Proposal for Academic Medical Centers," Journal of the American Medical Association 295, no. 4 (2006): 429–433.[Abstract/Free Full Text]
7. See In the Circuit Court of the State of Oregon (for the County of Marion), IN THE MATTER OF WARNER LAMBERT COMPANY, LLC, Case no. 04C14403, 13 May 2004; and F. Goodman et al., Drug Class Review on Antiepileptic Drugs in Bipolar Mood Disorder and Neuropathic Pain, Final Report, December 2004, http://www.ohsu.edu/drugeffectiveness/reports/documents/Antiepileptic%20Drugs%20Final%20Report.pdf (accessed 16 March 2006).
8. C.M. Bell et al., "Bias in Published Cost Effectiveness Studies: Systematic Review," British Medical Journal 332, no. 7543 (2006): 699–703.[Abstract/Free Full Text]

                      What's this?