Health Affairs, 24, no. 1 (2005): 163-173 doi: 10.1377/hlthaff.24.1.163 © 2005 by Project HOPE	New Online   Pay Cuts For Medicare Docs   Access To Care Woes   Public Coverage More Efficient   Empowering Consumers

This Article Abstract Reprint (PDF) Submit a response to this article Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services E-mail this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Add to My Personal Archive Download to Citation Manager Reprints & Permissions Citing Articles Citing Articles via HighWire Citing Articles via ISI Web of Science (11) Citing Articles via Google Scholar Google Scholar Articles by Tanenbaum, S. J. Search for Related Content PubMed PubMed Citation Articles by Tanenbaum, S. J. Related Collections Evidence-Based Medicine Health Reform Mental Health/Substance Abuse Physicians Quality Of Care

Implementing Evidence

Evidence-Based Practice As Mental Health Policy: Three Controversies And A Caveat

Abstract

 
Evidence-based practice (EBP) is the subject of vigorous controversy in the field of mental health. In this paper I discuss three distinct but interrelated controversies: how inclusive the mental health evidence base should be; whether mental health practice is a variety of applied science; and when and how the effectiveness goal in mental health is defined. I provide examples of evidence-based policy in mental health. These controversies pertain as well to general medicine. To the extent that they remain unresolved, evidence-based policy making may lead to ineffective and limited care.

This paper focuses mostly on psychologists—who share the field of mental health with psychiatrists, counselors, social workers, psychiatric nurses, and so forth—but not only as a concession to space. Rather, psychology is a primary locus of the EBP debate, and psychologists share many concerns with other professionals in the field. In psychology, EBP influences the research priorities of funding sources, the editorial policies of scholarly journals, the program agendas of scholarly conferences, the content of approved treatment lists from professional organizations and public agencies, and the tenor of intraprofessional discourse. EBP has been described as the cause of "psychological warfare between therapists and scientists."⁵ Whereas one prominent psychologist issued a "Manifesto for a Science of Clinical Psychology," the president-elect of the American Psychological Association (APA) considers it "fundamentally insane" to require the use of scientific treatment manuals in psychotherapy.⁶

The EBP debate is far ranging, and the various issues tend to run together. It is possible, however, to identify three distinct but interrelated and policy-relevant controversies. This paper depicts these three areas and offers a caveat about policy making in the face of these controversies not only in mental health but also in medicine. EBP sets methodological standards that may delegitimize effective treatments, and when those are incorporated into health policy making, patients and the polity may be adversely affected.

Controversy 1: Defining ‘Evidence’

Top Controversy 1: Defining... Controversy 2: Applying Research... Controversy 3: What Is... Caveat: The EBP Policy... Editor's Notes NOTES

 
The first controversy is, How restrictive should the definition of "evidence" be; that is, does the dominant definition inappropriately privilege some kinds of treatment over others? Although a number of evidence hierarchies—for example, that of David Sackett and colleagues—consider randomized controlled trials (RCTs) to be the gold standard for clinical decision making, a recent reformulation of the need for evidence calls for "practical clinical trials" (PCTs), which "select clinically relevant interventions to compare, include a diverse population of study participants, recruit participants from a variety of practice settings, and collect data on a broad range of health outcomes."⁷ This proposal notes the limitations of various alternatives: Traditional RCTs measure efficacy but not effectiveness, and nonexperimental research methods suffer from selection bias and confounding.

List of validated treatments. In psychology, perhaps the most influential evidence hierarchy has been the one adopted by the APA’s Division 12 (Clinical Psychology). In the early 1990s Division 12 charged the Task Force on the Promotion and Dissemination of Psychological Procedures with creating a list of Empirically Validated Treatments (EVTs) for dissemination to practitioners and educators. These were treatments (more recently called "empirically supported" and then "evidence based") for which there existed sufficiently rigorous evidence of efficacy—at least two RCTs or ten single-case experimental studies—with patients fitting the specific diagnostic criteria of the American Psychiatric Association’s Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV). A precondition for this efficacy research was that treatments were administered according to treatment manuals. Division 12 issued successive lists in the late 1990s. The Division 12 Web site maintains a list of "well established" treatments that meet the original evidentiary criteria. Although the Web site concedes that some "beneficial psychotherapies" may not yet have been studied, potential patients are urged to undergo those on the list because they have met "basic scientific standards of effectiveness."⁸

The Division 12 list has survived nearly a decade of controversy and has influenced mental health policy in a number of states. For example, in response to a 1999 consent decree, Hawaii established a Division 12–like panel to review the effectiveness of treatments for a range of childhood and adolescent mental health conditions. The Empirical Basis to Services (EBS) Task Force searched and evaluated controlled studies in childhood mental health; it included administrators and parents as well as mental health professionals, but much like Division 12, the task force made between-group experimental design the gold standard.⁹ An EBS committee continues to review the literature to decide the content of practice guidelines; these are then appended to requests for proposal (RFPs) issued by the Child and Adolescent Mental Health Division of the Hawaii Department of Health to service providers seeking contracts. The guidelines specify what services will and will not be provided to the division’s clients.¹⁰

Controversies about the list. Critics of Division 12 raise several points. They object that the list glosses over the difference between treatments that do not appear because they have been found ineffective and those that do not appear because they have not had requisite study. Another common criticism is that RCTs measure efficacy but not effectiveness—that they put reliability above validity. EBP proponents have answered this critique in a number of ways, including the revision of earlier evidence hierarchies to favor PCTs, as above. APA Division 29 (Psychotherapy) responded to Division 12 by creating its own task force to evaluate the effectiveness of mental health treatment. Division 29 took the position that much, if not most, of the benefit from psychological services results from the relationship between psychotherapist and patient rather than from specific psychotherapeutic techniques. They proceeded to review psychotherapy research, including RCTs but also naturalistic and process-outcome studies, and to compile a list of psychotherapy relationships that work.¹¹ To be sure, most psychologists recognize that techniques are executed within relationships, which in turn rely on relationship-building techniques. Still, they continue to debate the respective contributions of technique and relationship to the success of psychological treatment and the best way of framing the object of clinical inquiry.¹²

This controversy is complex, because underlying the debate is the question of whether the research methodology to which an activity is suited will determine whether or not it is deemed effective. In other words, assuming that psychotherapeutic relationships are less compatible with, say, PCTs than treatment techniques are, what should be the consequences for the psychological evidence base? Should the public compilation of approved treatments be organized around techniques, or around the kinds of relationships found to be as or more effective in less methodologically controlled studies? Many mental health interventions, as "socially complex services," necessarily violate the assumptions—precise protocols, equivalent trial conditions, and more—of experimental research.¹³ Experimental methodology is especially ill suited to psychodynamic and humanistic psychotherapies: Such methods do not focus solely on a disorder to be alleviated (rather, on a relationship with an individual patient); they do not enlist a predetermined treatment (rather, principles of therapeutic process); and they do not seek uniformity among therapists (rather, each therapist’s adherence to a theoretical orientation and a set of techniques that are compatible with his or her persona and the patient’s needs).¹⁴ Perhaps not surprisingly, RCT findings favor the behavioral and cognitive psychotherapies, where technique is paramount and more easily codified in treatment manuals.¹⁵ On the other hand, high-quality naturalistic and process-outcome studies attest to the effectiveness of psychodynamic and other therapies. The weaknesses of nonexperimental research have been documented (and disputed), but what to do if the object of knowledge—in this case, the value of psychotherapeutic relationships—is not accessible to experimental study?¹⁶ One clinical leader implores, "Can’t we have both relationship and technique?"¹⁷ A more policy-relevant question might be: Can EBP in mental health commit itself to an inclusive enough evidence hierarchy not to privilege technique unfairly over relationship? Can it do so without further stigmatizing psychology vis-à-vis medicine (including psychopharmacology), thus undermining mental health care’s claim to effectiveness worthy of funding?

Controversy 2: Applying Research Evidence

Top Controversy 1: Defining... Controversy 2: Applying Research... Controversy 3: What Is... Caveat: The EBP Policy... Editor's Notes NOTES

 
The second controversy is, Can, and if so, should, practice consist of the faithful application of research evidence? Applying research to practice is the raison d’etre of the EBP movement. Its goal is to change practitioners’ behavior by bridging the distance from research to practice and securing a central place for research in the consulting room. Proponents of EBP are intent on discovering how best to build this bridge. Generally speaking, they apply a diffusion-of-technology model, whereby experimental research is disseminated to clinicians, sometimes in the form of guidelines or protocols. EBP includes a role (for example, in the IOM definition) for clinical judgment. The weight of the EBP literature, however, holds that large numbers of uninformed (or unethical) practitioners are responsible for inadequate clinical care.¹⁸ More research, not more judgment, is prescribed.

Dissemination and practice. The dissemination of research findings to practitioners has not, for the most part, brought practice into line with research. There is, therefore, a large body of work devoted to increasing practitioners’ uptake of study findings. Decision-support tools such as manuals, protocols, and guidelines are designed to specify practice with greater or lesser authority. The Cochrane Collaboration has built an electronic bridge with an accessible database of RCTs, meta-analyses, and systematic reviews. In mental health, the National Institute of Mental Health (NIMH) sponsors the assembly and distribution of "implementation toolkits," which contain information and training resources for creating evidence-based practitioners in NIMH-funded programs.¹⁹ Simultaneously, a team of psychologists is calling for more basic theory on transportability of research to practice.²⁰

Science versus practice. The difficulty of changing practitioners’ behavior contributes to a larger controversy in mental health about whether practice is in fact applied science. Some parties to the dispute argue that no matter how rigorous, probabilistic research cannot say with certainty how to treat the next patient; knowledge of the aggregate is simply not knowledge of an individual. In this view, clinical care necessarily entails judgment, and the exemplary "reflective practitioner" decides what to do based not only on research but also on disciplined consideration of his clinical experience and in-depth knowledge of the patient.²¹ Thus, research pertains, but the attempt to substitute research for discretion is futile. Adherents to this position cite naturalistic studies of real-time expert decision making; these find nothing of the formal decision analysis that is used to turn research findings into clinical directives.²² One prominent psychologist protests that "science and practice are not the same, and no monistic ideology can make them the same."²³

Psychologists who dispute the applied science model offer alternative models of practice-relevant inquiry. According to Donald Peterson, mature psychological practice requires "disciplined inquiry" wherein the practitioner brings a "guiding conception" as well as evidence and experience to bear on patient assessment, plan formulation, and psychological intervention. The clinician’s ongoing evaluation may lead to reformulating the treatment plan, and the final evaluation will feed back into the clinician’s experience base and guiding conception.²⁴ Daniel Fishman, in turn, offers a Pragmatic Case Study Method that uses a highly specified report structure (corresponding to the disciplined inquiry model) to create large, searchable databases of case studies that allow for a practitioner’s inductive generalization.²⁵ Proponents of EBP argue that this kind of practice knowledge is too weak methodologically to identify effective treatment the way experimental studies do. The actual evaluation of EBP (as opposed to clinical trials of some treatments) in mental health is as yet limited: How practitioners know remains a matter of dispute.

Fidelity versus discretion. A recent report from the Milbank Memorial Fund describes several mental health interventions that have proved successful in clinical trials.²⁶ The authors emphasize fidelity in the application of this research—for example, the prescription not just of an effective antipsychotic medication but also of the dosage specified in the research. Among the report’s eight "essential points," then, are that providers should be held accountable for delivering evidence-based practices and that measures of program fidelity are available to further that accountability. The authors go on to make two other points, however, that speak to the other side of the practice controversy. They conclude that "a wide variety of effective treatments" should be available because overall effectiveness may not translate into effectiveness in "significant subgroups." Furthermore, "treatment choice and wide selection" are necessary to "maximize treatment response and adherence to treatment." Admittedly, the authors urge variety and choice among effective treatments, but they also describe individual variation within an overall effect. They note that even an effective treatment does not work for every patient and that individual patients’ treatment preferences help determine what is effective for them. How does this square with maximal practitioner fidelity to research findings, especially when effective treatment but not all effective treatments are mandated? How should EBP operationalize fidelity to allow enough but not too much discretion?

Managed mental health care. The effort to hold practitioners to evidence-based directives is widespread. Roughly two-thirds of Americans with private health insurance are enrolled in managed behavioral health care organizations (MBHOs), and these have used efficacy research to limit types and duration of care.²⁷ In the public sector, the District of Columbia Department of Mental Health (DMH) has proposed a policy regarding evidence-based psychotherapy in that system.²⁸ According to the draft policy, all psychotherapy services delivered to community-based adult consumers will conform to a short list of EBPs. The list includes twelve disorders, five of which have only one treatment option. Psychodynamic psychotherapy does not appear on the list, but techniques such as eye movement desensitization and reprocessing do. The chief clinical officer of the DMH, in consultation with experts, will review the list annually. Providers may submit requests to expand it, but they must also be specifically credentialed for any treatment they render. Otherwise, "they should not attempt to provide that evidence-based psychotherapy." Credentialing criteria are vague but represent a major departure from professional practice. Psychologist psychotherapists, for example, are licensed professionals whose scope of practice is delineated by legislatures. With few exceptions, they are not credentialed for one technique at a time.

The controversy over psychological practice as applied science has implications for the health system as well as individual practitioners. In one well-developed scenario, mental health practice adheres increasingly to research-based manuals or guidelines, and most treatments are performed by clinicians with less training. Highly trained practitioners will design systems, conduct research, manage quality assurance programs, and, when necessary, care for patients whose "manualized" treatment did not succeed.²⁹ There are obvious cost advantages to a system of this kind; some MBHOs now enjoy these advantages.

It is unclear, however, whether cost containment is a felicitous side effect of EBP or whether EBP has been used to legitimate less costly treatment. Most mental health RCTs by design investigate manualizable treatment techniques. These, then, become the psychological services deemed effective in the treatment of mental illness. Proponents of this approach argue that if manualized services, delivered by less highly trained practitioners, are effective, it is inexcusably wasteful not to deliver them. Critics counter that more individualized therapy with a more highly trained clinician is more effective, even if less demonstrably so. From this perspective, the efficacy of manualized treatments is short-term symptom relief based on formal diagnosis under study conditions, which will prove limited for many mentally ill people.

As the first controversy questioned the evidence half of EBP, this second one questions the practice half. As above, the issue is one of inclusiveness. The desirability of more high-quality evidence for mental health practice is a given, but does evidence include the quasi-experimentation suited to much of practice? The pertinence of high-quality research to mental health practice is also a given, but does practice include pluralistic and inductive knowledge that benefits from, but is not, experimental research? At stake in both cases is the rigor and perceived rigor of the mental health enterprise, along with the necessary complexity of effective mental health care.

Controversy 3: What Is ‘Effective’?

Top Controversy 1: Defining... Controversy 2: Applying Research... Controversy 3: What Is... Caveat: The EBP Policy... Editor's Notes NOTES

 
The third controversy is, To the extent that EBP is the means to effective health care, what is meant by "effective," and who decides? For the movement, EBP is a moral imperative: Experimental research produces therapeutic efficacy and therefore better health, a largely uncontested moral good.³⁰ In other words, EBP occupies the moral high ground because its practitioners do "what works." At the societal level, doing what works not only improves health care quality but presumably allows for an efficient allocation of scarce health care resources.³¹

It is not altogether clear, however, what it means for an intervention to work, and this is the occasion for a third controversy. What does effectiveness mean? EBP’s definition of effectiveness (like its definitions of evidence and practice) privileges objectivity; it documents the quantifiable outcome of a standardized intervention. Even RCTs, however, are designed by researchers who make judgments about what effectiveness is, and these decisions are further interpreted by the clinicians who implement the trials.³² Definitions of effectiveness may respond to the mission of a funding agency or the data that are available for analysis, among other things. One review of physiotherapy research found thirty-one trials using twelve different outcome measures; only two were common enough to allow for meta-analysis.³³ Were these the best measures of effectiveness?

In the field of mental health, the meaning of effectiveness is especially contentious. As noted earlier, Division 12 defines effectiveness first as symptom relief. Varieties of functional dis/ability also figure as treatment outcomes. Some schools of psychotherapy, however, view symptoms as manifestations of underlying mental health conditions. For them, it is possible to eliminate symptoms but not suffering, and this does not amount to effective treatment. One possible effectiveness measure is the patient’s understanding of what has happened to him or her. Given that many "effective" treatments produce only partial relief, understanding may help patients relate better to their remaining symptoms.³⁴ A simpler example of competing definitions is whether marital therapy is more effective when the partners remain married or when they are most satisfied, and does it depend on whether or not they have young children?

In Oregon, "evidence based" has itself come to mean "cost-effective," although the meaning of cost-effectiveness is not very clear. In August 2003 the legislature passed Senate Bill 267, which requires that for the biennium beginning 1 July 2005, a number of state agencies, including those that deal with addiction and mental health, will spend 25 percent of their program budgets on evidence-based programs. The figure rises to 50 percent in 2007 and 75 percent in 2009. Agencies that do not meet this requirement will face budget consequences in the following biennium. According to the legislation, an evidence-based program is one that "(a) incorporates significant and relevant practices based on scientifically based research; and (b) is cost effective." The Oregon Office of Mental Health and Addiction Services offers an operational definition of evidence-based practice: RCTs are at the top of the evidence hierarchy, and except at the lowest level of acceptable evidence, implementation must be measured by a fidelity tool.³⁵ For Oregonians, then, EBP research must include relative cost in its definition of effective treatment.

Although effectiveness is a compelling goal, it is not the only one. Patients, for example, may forgo "what works" to avoid inhumane side effects or to preserve the personal meanings they give to health or illness. In mental health, especially, effectiveness can be used to justify coercive treatment of problematic patients. Even the psychiatric recovery movement parts ways with EBP. Recovering mental patients can benefit from information about effectiveness, but at least some see the effectiveness criterion as a possible infringement on their freedom.³⁶ Most mentally ill people want what other people have: houses, jobs, and friends. Effective treatment may or may not be the best means of getting these, and even the most effective means of getting them may not include the value of finding one’s own way.

Finally, who gets to say what effectiveness is? EBP may create it, but what effectiveness looks like is only specified in the research process. Some proponents of EBP in mental health consider its adaptability to market and political goal setting as a strength.³⁷ So if, in this view, managed care sets as an effectiveness goal fewer mental health episodes in the course of a year, EBP can identify the treatment to achieve this. Mental health care suppliers, including drug companies, and public agencies with large mental health caseloads also set effectiveness criteria. Their agendas are more or less transparent, invite more or less criticism, and are set through a more or less participatory process. In any event, EBP as effectiveness research responds to someone’s definition of effectiveness, and although it is often treated as self-evident, that definition has profound consequences. Patients’ values and preferences are putatively part of EBP, but they come into play mostly after the fact; once the effectiveness data have been collected and analyzed, patients may determine whether the "effective" treatment suits them individually. Perhaps this third controversy deserves a mini-hearing whenever EBP researchers state their outcome measures. They should say why they have chosen these measures, and whose interests they represent.

Caveat: The EBP Policy Train Is Leaving The Station

Top Controversy 1: Defining... Controversy 2: Applying Research... Controversy 3: What Is... Caveat: The EBP Policy... Editor's Notes NOTES

 
EBP in mental health may be more controversial than EBM. Psychologists may be more diverse or contentious than physicians. Managed mental health care may be more intrusive and rile practitioners more vigorously. The poignant failures of mental health policy may call more loudly for change. On the other hand, EBM is also broadly and deeply debated, and each of the controversies described here pertains to EBM as well.³⁸

First, despite the dominance of the "medical model," medical practice is pluralistic, especially across specialties, and in general medicine there is new interest in the relational aspects of diagnosis and treatment. The same epistemological question arises: Can clinical research be both rigorous and inclusive? Second, EBM researchers do not easily effect behavioral change in medical practitioners, and this is sometimes blamed on the insufficiency of the applied science model. Disagreements about the nature and value of clinical judgment are joined. Third, the definition of medical effectiveness is in dispute. As in mental health, some physicians focus not on symptom relief or functional measures but on the nature of suffering.³⁹ As in mental health, it is not always clear which effectiveness definition applies in a given study or review of studies, or why. Furthermore, both EBM and EBP in mental health participate in vital debate while the policy train is leaving the station. Proponents of EBP lament the difficulty of making policy responsive to evidence, and this is generally true. To the extent that EBP has not resolved its controversies, however, how much and what kind of policy making is called for? One recent policy analysis warned of the potential undesirability of introducing hospital quality standards into malpractice litigation. The authors examine the consequences of promulgating standards based on questionable evidence and say: "We should require especially convincing evidence when the law steps in to demand universal compliance with the standard."⁴⁰

Implementation of the policies described in this paper deserves close and serious study. Suffice it to say that although they do not demand universal compliance, these policies are authoritative about access to critical services by vulnerable people. The evidence of effectiveness on which they rely is more or less convincing, from an EBP perspective or from others. The District of Columbia’s evidence-based psychotherapy policy permits only dialectical behavioral therapy (DBT) for people with borderline personality disorder (BPD). These patients represent 10–20 percent of all mentally ill people and have high rates of service use, including hospitalization. There is, however, at least one high-quality RCT showing psychoanalytically oriented psychotherapy to be no less effective than DBT at termination and more so afterward; a compilation of clinical guidelines for BPD concludes that different interventions are most effective for different patients.⁴¹ Under circumstances like these—which are surely not uncommon in clinical science—what authority should accrue to evidence-based policy? In EBM as well as in EBP in mental health, policy may risk unnecessary ineffectiveness and deprivation along with the political encumbrances of confusion and mistrust.

Editor's Notes

Top Controversy 1: Defining... Controversy 2: Applying Research... Controversy 3: What Is... Caveat: The EBP Policy... Editor's Notes NOTES

 
Sandra Tanenbaum (tanenbaum.1{at}osu.edu) is an associate professor in the School of Public Health at the Ohio State University in Columbus.

The author acknowledges the help and encouragement of the editors and anonymous referees.

NOTES

Top Controversy 1: Defining... Controversy 2: Applying Research... Controversy 3: What Is... Caveat: The EBP Policy... Editor's Notes NOTES

 

1. K. Patterson, "What Doctors Don’t Know (Almost Everything)," New York Times Magazine, 5 May 2002, 71–74.
2. A.M. Epstein, "The Outcomes Movement—Will It Get Us Where We Want to Go?" New England Journal of Medicine 323, no. 4 (1990): 266–270.[ISI][Medline]
3. S.J. Tanenbaum, "Knowing and Acting in Medical Practice: The Epistemological Politics of Outcomes Research," Journal of Health Politics, Policy and Law 19, no. 1 (1994): 27–44.
4. Patterson, "What Doctors Don’t Know."
5. C. Tavris, "Mind Games: Psychological Warfare between Therapists and Scientists," Chronicle of Higher Education 29, no. 5 (2003): B7–B10.
6. R.M. McFall, "Manifesto for a Science of Clinical Psychology," Clinical Psychologist 44, no. 6 (1991): 75–88; and B. Carey, "For Psychotherapy’s Claims, Skeptics Demand Proof," New York Times, 10 August 2004.
7. S.R. Tunis, D.B. Stryer, and C.M. Clancy, "Practical Clinical Trials: Increasing the Value of Clinical Research for Decision Making in Clinical and Health Policy," Journal of the American Medical Association 290, no. 12 (2003): 1624–1632.[Abstract/Free Full Text]
8. American Psychological Association, Society of Clinical Psychology (Division 12), A Guide to Beneficial Psychotherapy, www.apa.org/divisions/div12/rev_est/index.html (15 November 2004).
9. B.F. Chorpita et al., "Toward Large-Scale Implementation of Empirically Supported Treatments for Children: A Review and Observations by the Hawaii Empirical Bases to Services Task Force," Clinical Psychology: Science and Practice 9, no. 2 (2002): 165–190.[CrossRef][ISI]
10. Eric Daleiden, research and evaluation specialist, Child and Adolescent Mental Health Division, Hawaii Department of Health, telephone interview, 12 July 2004.
11. J.C. Norcross, ed., Psychotherapy Relationships That Work: Therapist Contributions and Responsiveness to Patient Needs (New York: Oxford University Press, 2002).
12. See, for example, L.E. Beutler, "The Empirically Supported Treatments Movement: A Scientist-Practitioner’s Response," Clinical Psychology: Science and Practice 11, no. 3 (2004): 225–229.[CrossRef][ISI]
13. N. Woolf, "Using Randomized Controlled Trials to Evaluate Socially Complex Services: Problems, Challenges, and Recommendations," Journal of Mental Health Policy and Economics 3, no. 2 (2000): 97–109.[CrossRef][Medline]
14. A.C. Bohart, M. O’Hara, and L.M. Leitner, "Empirically Violated Treatments: Disenfranchisement of Humanistic and Other Psychotherapies," Psychotherapy Research 8, no. 2 (1998): 141–157.[CrossRef][ISI]
15. D.L. Chambless et al., "An Update on Empirically Validated Therapies," Clinical Psychologist 49, no. 2 (1996): 5–18.
16. J. Concato, N. Shah, and R.I. Horwitz, "Randomized Controlled Trials, Observational Studies, and the Hierarchy of Research Designs," New England Journal of Medicine 342, no. 25 (2000): 1887–1892.[Abstract/Free Full Text]
17. Beutler, "The Empirically Supported Treatments Movement."
18. E.A. McGlynn et al., "The Quality of Health Care Delivered to Adults in the United States," New England Journal of Medicine 348, no. 348 (2003): 2635–2645[Abstract/Free Full Text]; and K.W. Goodman, Ethics and Evidence-based Medicine: Fallibility and Responsibility in Clinical Science (New York: Cambridge University Press, 2003).
19. W. Torrey et al., "Implementing Evidence-based Practices for Persons with Severe Mental Illness," Psychiatric Services 52, no. 1 (2001): 45–50.[Abstract/Free Full Text]
20. W.K. Silverman, W.M. Kurtines, and K. Hoagwood, "Research Progress on Effectiveness, Transportability, and Dissemination of Empirically Supported Treatments: Integrating Theory and Research," Clinical Psychology: Science and Practice 11, no. 3 (2004): 295–299.[CrossRef][ISI]
21. D.A. Schon, The Reflective Practitioner: Toward a New Design for Teaching and Learning in the Professions (San Francisco: Jossey-Bass, 1983).
22. G. Klein, Sources of Power: How People Decide (Cambridge: MIT Press, 1999).
23. D.R. Peterson, "Science, Scientism, and Professional Responsibility," Clinical Psychology: Science and Practice 11, no. 2 (2004): 196–210.[CrossRef][ISI]
24. D.R. Peterson, "Connection and Disconnection of Research and Practice in the Education of Professional Psychologists," American Psychologist 46, no. 4 (1991): 422–429.[CrossRef]
25. D.B. Fishman, The Case for Pragmatic Psychology (New York: New York University Press, 1999).
26. A.F. Lehman et al., Evidence-based Mental Health Treatments and Services: Examples to Inform Public Policy, June 2004, www.milbank.org/reports/20041ehman/20041ehman.html (18 October 2004).
27. M. Oss, E.L. Jardine, and M.J. Pesare, Open Minds Yearbook of Behavioral Health and Employee Assistance Program Market Share in the United States, 2002–2003 (Gettysburg, Pa.: Open Minds, 2002); and N.A. Cummings, "The First Decade of Managed Behavioral Health Care: What Went Right and What Went Wrong," in Psycho-Economics: Managed Care in Mental Health in the New Millenium, ed. R.D. Weitz (New York: Haworth, 2000), 19–38.
28. D.C. Department of Mental Health, "Policy No. 311.2: Evidence-based Psychotherapy," 2004.
29. S.C. Hayes, D.H. Barlow, and R.O. Nelson-Gray, The Scientist-Practitioner: Research and Accountability in the Age of Managed Care, 2d ed. (Boston: Allyn and Bacon, 1999).
30. M. Gupta, "A Critical Appraisal of Evidence-based Medicine: Some Ethical Considerations," Journal of Evaluation in Clinical Practice 9, no. 2 (2003): 111–121.[CrossRef][ISI][Medline]
31. Norcross, Psychotherapy Relationships That Work; and P.E. Nathan and J.M. Norman, A Guide to Treatments That Work, 2d ed. (New York: Oxford University Press, 2002).
32. J.J. Gonzales, H.L. Ringeissen, and D.A.Chambers, "The Tangled and Thorny Path of Science to Practice: Tensions in Interpreting and Applying Evidence," Clinical Psychology: Science and Practice 9, no. 2 (2002): 204–209[CrossRef][ISI]; and M. Berg, Rationalizing Medical Work: Decision-Support Techniques and Medical Practices (Cambridge: MIT Press, 1997).
33. W.A. Rogers, "Evidence-based Medicine in Practice: Limiting or Facilitating Patient Choice?" Health Expectations 5, no. 2 (2002): 95–103.[CrossRef][Medline]
34. S.B. Messer, "Empirically Supported Treatments: What’s a Non-Behaviorist to Do?" in Critical Issues in Psychotherapy: Translating New Ideas into Practice, ed. B.D. Slife, R.N. Williams, and S.H. Barlow (Thousand Oaks, Calif.: Sage Publications, 2001), 3–19; and Bohart et al.,, "Empirically Violated Treatments."
35. State of Oregon, "An Act: SB 267, Chapter 669 Oregon Laws" (2003), www.leg.state.or.us/orlaws/sess0600.dir/0669ses.htm (15 November 2004); and Oregon Office of Mental Health and Addiction Services, "Proposed Operational Definition for Evidence-based Practices, Final Draft," 1 June 2004, www.dhs.state.or.us/mentalhealth/ebp/definition0722.pdf (10 November 2004).
36. F. Frese et al., "Integrating Evidence-based Practices and the Recovery Model," Psychiatric Services 52, no. 11 (2001): 1462–1468.[Abstract/Free Full Text]
37. Hayes et al., The Scientist-Practitioner.
38. See, for example, S.J. Tanenbaum, "What Physicians Know," New England Journal of Medicine 329, no. 17 (1993): 1268–1271[Free Full Text]; and Tanenbaum, "Knowing and Acting in Medical Practice."
39. E.J. Cassell, The Nature of Suffering and the Goals of Medicine (New York: Oxford University Press, 1991).
40. M.M. Mello, D.M. Studdert, and T.A. Brennan, "The Leapfrog Standards: Ready to Jump from Marketplace to Courtroom?" Health Affairs 22, no. 2 (2003): 46–59.[Abstract/Free Full Text]
41. A. Bateman and P. Fonagy, "Treatment of Borderline Personality Disorder with Psychoanalytically Oriented Partial Hospitalization: An Eighteen-Month Follow-up," American Journal of Psychiatry 158, no. 1 (2001): 36–42[Abstract/Free Full Text]; and M.H. Stone, "Clinical Guidelines for Psychotherapy for Patients with Borderline Personality Disorder," Psychiatric Clinics of North America 23, no. 1 (2000): 193–210.[CrossRef][ISI][Medline]

                      What's this?

This article has been cited by other articles:

				E. J. Mullen, S. E. Bledsoe, and J. L. Bellamy Implementing Evidence-Based Social Work Practice Research on Social Work Practice, July 1, 2008; 18(4): 325 - 338. [Abstract] [PDF]

				T. B. Sullivan Evidence-Based Practices in Mental Health: Debate and Dialogue on the Fundamental Questions Psychiatr Serv, June 1, 2007; 58(6): 882 - 883. [Full Text] [PDF]

				C. Hilarski Book Review: Cognitive Therapy for Challenging Problems: What to Do When the Basics Don't Work. Research on Social Work Practice, September 1, 2006; 16(5): 540 - 542. [PDF]

				C Waddell and R Godderis Rethinking evidence-based practice for children's mental health Evid. Based Ment. Health, August 1, 2005; 8(3): 60 - 62. [Full Text] [PDF]