Health Affairs, 24, no. 4 (2005): 938-948 doi: 10.1377/hlthaff.24.4.938 © 2005 by Project HOPE	New Online   How Would Obama, McCain Cover The Uninsured?   Debating Cost Of Uninsured   Try Medicare-For-All   HA Blog Top 10

This Article Abstract Reprint (PDF) Submit a response to this article Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services E-mail this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Add to My Personal Archive Download to Citation Manager Reprints & Permissions Citing Articles Citing Articles via HighWire Citing Articles via ISI Web of Science (5) Citing Articles via Google Scholar Google Scholar Articles by Gottlieb, S. Search for Related Content PubMed PubMed Citation Articles by Gottlieb, S. Related Collections Business Of Health Consumer Issues Legal/Regulatory Issues Quality Of Care Research And Technology Politics

FDA Under Scrutiny

Opening Pandora’s Pillbox: Using Modern Information Tools To Improve Drug Safety

Abstract

 
How the Food and Drug Administration (FDA) responds to criticism of its drug safety process will determine whether drug safety actually improves. Propping up the Office of Drug Safety with more bureaucratic prominence or adding new requirements to the preapproval process will add to the cost of drug development and not make drugs safer. New information tools can dramatically improve postmarketing surveillance and collection of data on safety. This information could then be used to reach more definitive regulatory conclusions sooner. New incentives will be needed to entice payers and product developers to work on building a broader, more robust system for collecting data on drug safety.

The principal task of evaluating the safety and effectiveness of new medical products falls to the U.S. Food and Drug Administration (FDA). The agency has come under withering criticism recently from Congress and the press for failing to detect rare but serious side effects that emerged with some popular medications, mainly the COX-2 painkiller Vioxx (linked to an increased risk of cardiovascular disease) and the class of antidepressants known as selective serotonin reuptake inhibitors (SSRIs), some of which were linked to an increased propensity toward suicidal behavior in children.² These side effects emerged only after the drugs had been on the market for years and given to millions of patients.

Charges against the FDA. Critics charge that evidence was available all along to suggest a problem with these drugs and to prompt stronger regulatory action much sooner. The FDA was slow to see the problem and then slow to act on it, they argue.³ However, the FDA’s job is not to guarantee 100 percent safety; rather, it is to approve medicines with an appropriate risk-benefit ratio and remove unreasonably unsafe drugs when necessary. The baseline is not the perfectly safe drug, but rather, the drug with benefits that outweigh reasonable risks. Yet the fact remains that the FDA could be doing more to monitor drugs after they have been approved and to gather information about both their evolving risks and their emerging benefits. How the FDA handles these challenges and responds to recent criticisms may well determine what kind of system we achieve.⁴

The problem: quality of information. The fundamental problem inside the FDA is not tied to its review of safety problems—which is rooted in a deliberative science that will never move at a speed that satisfies ardent critics or that accommodates the media cycle—but the quality of information on which the FDA can base its evaluations. Today, the data that medical reviewers receive in conjunction with the process for approving new products are from highly structured clinical trials, carried out on homogenous populations of patients that are carefully screened and preselected and then given new drugs under special protocols. There is little chance that such trials will ever provide a complete review of how a new treatment will perform when it is used in much broader populations of patients in real-world clinical settings, where patients do not always take their medicines on time or at all; where patients might have other medical problems or be of advanced age or in frail health; and where they have comorbidities or unusual diets, or they fill prescriptions for medications or dietary supplements that interact with one another, subtly or otherwise.⁵

Taking up some recent proposals to lengthen clinical trials or require them to include more patients is only going to add to the cost of drug development and eventually the list price of new drugs. The approval process for a new drug—the part of the drug development process that involves testing new medicines in people—can already take as long as ten years and cost as much as $466 million.⁶ Some cardiovascular drugs are tested in clinical trials that enroll 10,000 patients or more at an average cost of up to $20,000 per patient.⁷

Longer trials will not add a modicum of safety because the data that these trials generate will never reach a magnitude sufficient to unearth the kinds of side effects that were evident in the case of Vioxx or the SSRIs. It is easy in hindsight to point to subtle signals or postulations that were apparent with these drugs even prior to approval to argue that all we would have needed was a little more preapproval data. However, subtle signals exist for almost all new medicines, and for many of rare or uncertain problems it is impossible to recruit rigorous clinical trials sufficiently large to ferret out fact from background noise and reach clear conclusions. The analgesic bromfenac, for example, resulted in four deaths and eight liver transplants among 2.5 million treated patients before it was withdrawn from the market. Such an incidence could not have been detected in the product’s clinical trial of 2,500 patients.⁸ Even if a drug company could afford to sponsor trials large enough to detect events this statistically remote, it would be unable to recruit enough eligible patients.

New tools, not blunt remedies. The good news is that new tools can help the FDA address some of the shortcomings that have led to the current criticism, without all of the trade-offs in time and cost that come with blunt remedies such as requirements for larger and longer clinical trials. In particular, information technology (IT), properly deployed, will enable the FDA to pursue fundamentally better ways to monitor the safety and effectiveness of new medical products after they are made available in the marketplace. But political expediency and the desire for a quick, if not cosmetic, fix to the current criticism (especially calls to rearrange the FDA’s structure to give more bureaucratic prominence to the Office of Drug Safety) run the risk of adding to the agency’s operating inefficiency while doing little to help it better meet its new challenges.⁹ The history of carving out offices inside the FDA has had its shortcomings. Offices that earn their own fiefdom often become marginalized because they are removed from the day-to-day flow of information and activity inside the agency’s drug center.

Current Deficits Of The FDA’s System

Top Current Deficits Of The... Making A Slow System... Developing Safety Information In... NOTES

 
Underinvestment in IT. The data gaps that plague the FDA’s drug review process, particularly its postmarketing surveillance, stem directly from its passive information-gathering process. The agency has consistently underinvested in the IT resources that could help it more actively gather high-quality data about how drugs are used in the marketplace and what side effects might be occurring. Instead, the FDA has used IT as a deferrable spending item that can be cut during lean budget years, when savings need to be found for its continuing operations. Although it is impossible to calculate exactly how much the agency’s review programs spend on IT-related infrastructure (because it is embedded in many different programs), consider that total spending on IT-related activities at the FDA was cut $29.1 million in 2004 from what the agency had requested so that the FDA could find savings to stay inside its congressional budget allocation. That exceeds the entire $23.8 million budget of the FDA’s Office of Drug safety for 2004.¹⁰

Passive reporting. What is important for improving drug safety is not how much we spend on the enterprise but, rather, the kinds of systems we spend it on. The current system relies on others to undertake the time and cost of monitoring by sending news of potential problems to the agency. This passive reporting system leaves the FDA dependent upon busy doctors to fill out lengthy drug-safety reports. It is estimated that doctors report as few as 1 percent of all actual adverse drug events.¹¹ For example, in the early to mid-1990s the FDA received an average of eighty-two reports each year about adverse reactions caused by the heart drug digoxin.¹² This relatively small number of reports seemed to show that digoxin was not a big problem. But a systematic survey of Medicare records over the same period revealed 202,211 hospitalizations for adverse reactions to digoxin.¹³ The FDA even maintains some staffers whose job it is to review the letters submitted by doctors to medical journals; the belief inside the agency is that doctors will sometimes report adverse events to medical journals but not to the FDA, or will report them to the FDA only after they have submitted their reports for publication.

This passive reporting process is both slow and expensive and, of course, woefully incomplete. Meanwhile, while the doctors end up reporting too few actual adverse events, nervous companies that are far removed from the point of care report too many, which creates noise that makes it that much harder to separate real safety problems from routine findings. In 2002 more than 171,000 adverse-event reports came from manufacturers’ periodic reports—35,095 were marked "serious" and 136,451, "non-serious" events. Another 128,000 came from expedited, fifteen-day reports that manufacturers reserve for the potentially most serious adverse drug events. This compares with 20,455 reports filed in 2002 by individual patients and doctors.¹⁴

Making A Slow System A Little Faster

Top Current Deficits Of The... Making A Slow System... Developing Safety Information In... NOTES

 
So far, the fixes that have been made to this system have mostly focused on adding only a veneer of sophisticated IT to make the outdated process work faster. For example, more of the forms that doctors and manufacturers complete are now fully entered electronically. But doctors still have to key the information in by hand. Two of these IT systems—the Med Watch Safety Information system, which receives reports from doctors, and the Adverse Event Reporting System (AERS), which receives reports from drug makers—make it easier for reports to be filed electronically with the FDA and, therefore, for these results to be more easily mined for relevant signs of potential safety problems.

But as productive as these and other more automated reporting programs have been (including the Vaccine Adverse Event Reporting System [VAERS], the Centers for Education and Research on Therapeutics [CERTs], and the FDA’s Medical Product Surveillance Network [MedSun] pilot program for devices, which requires rapid adverse-event reporting on medical devices by a group of hospitals and nursing homes), they are only small steps toward fully exploiting IT for the task of drug safety.

The backbone of the FDA’s effort, the MedWatch system, was put in place in 1993 to consolidate what had become a confusing number of forms and reporting procedures. But doctors complain that the system is still too cumbersome to fit easily into their busy practice routines.¹⁵ Meanwhile, AERS is aggressively migrating reports from paper to digital format. In fact, electronic submissions into AERS represented only 15 percent of the total expedited reports the FDA received in 2002, but by 2003 the FDA received most of its reports—an estimated 369,839—this way.¹⁶ The system also saves money, with the cost of filing a report cut from $31 per paper report to $3–$19 per report for those submitted electronically.¹⁷ But once again, there are shortcomings. Only 21 percent of urgent safety reports to the FDA were submitted electronically in 2003.¹⁸ The system is also primitive by modern IT standards. For example, the size of attached files is limited, and PDF files can be sent only on physical media (not via e-mail) or to a special electronic FDA drop box called the Gateway.

All of this leaves little doubt that even the most basic IT improvements have been slow in coming, hobbled by a lack of budget and vision. As a result, information is made available to the FDA slowly and takes even longer to analyze by the FDA’s trained personnel. Subtle side effects—especially medical problems that occur naturally in a large population or as a consequence of the condition that a drug aims to treat (the side effects at issue with Vioxx and the SSRIs met these criteria)—could be easily dismissed as normal or "background" events as a result of inadequate sample sizes and the inability to easily aggregate and analyze population-based data on actual drug use.

Right now, under the current system, if a drug safety problem is made apparent by a single case report, it may escape notice. Even if similar signals are produced by other case reports, the likelihood of a causal relation is often simply debated unless more evidence emerges. The real linchpin often requires additional study—sometimes well-designed epidemiological studies to provide additional verification. The FDA undertakes these kinds of studies in collaboration with health care systems for some of the most pressing safety questions, but the entire FDA budget for these efforts in 2004 was only $12.4 million.¹⁹ The result is that it could take many years to develop the level of evidence required for drawing causal links between a drug and its unwanted outcome.

Developing Safety Information In Real Time

Top Current Deficits Of The... Making A Slow System... Developing Safety Information In... NOTES

 
International comparisons. In comparison with the United States, the United Kingdom actively solicits reports through its Prescription-Event Monitoring system, which surveys doctors about any adverse experiences among the first 10,000 patients who use a new drug. Japan conducts active surveillance for the first six months after a new product is launched, issuing repeated announcements about pharmacovigilance during the postmarketing phase and sending in trained drug company representatives to query doctors using the new medicine.²⁰

The FDA needs similar systems that allow it to collect more information about a drug’s use in real-world, and in some cases real-time, clinical practice and to use this information more effectively. This requires two simultaneous efforts. First, the agency needs tools for collecting safety information more quickly at the point of care, to detect potential problems earlier. Second, it needs resources for making better, more frequent use of the practical clinical data culled from these real-world settings. The FDA should be directing more epidemiological evaluations using population-based data that can provide important hypothesis tests and help define evolving safety problems quickly. Although randomized, prospective trials will remain the gold standard for defining safety issues, they are slow, expensive, and difficult to undertake. Wider use of practical trials could help focus resources on only the most important and elusive safety questions while providing important insights into emerging safety questions.

Both efforts require the FDA to have better tools for collecting health information electronically and then using IT to access and manipulate this information. As electronic medical records (EMRs) and other IT systems gain wider adoption, these kinds of opportunities will be more easily accessible. It behooves the FDA to implement drug safety reforms that envision and accommodate these opportunities, instead of implementing more expedient but fleeting fixes to the current, inefficient monitoring system that are predicated on an old way of doing things.

Prospective detection tools. The first step to creating a modern safety monitoring system is developing prospective detection tools. For the past two years, the FDA has devoted a limited amount of resources, much of it authorized by the Prescription Drug User Fee Act (PDUFA) of 2002, to start developing and pilot-testing more active reporting systems. Under these programs, the agency can receive more timely, and in some cases, comprehensive, information by linking more directly into EMRs and other information tools used in health care settings.

These pilot programs are too small and still too primitive in their function to have had an appreciable impact on safety monitoring or even provide a meaningful proof-of-concept. They also do not rely on the type of real-time surveillance that IT is enabling, but they are a start, allowing the FDA to tap electronic health information networks to achieve more active monitoring of health information for selected signals of adverse drug events. Once more doctors are using EMRs, this kind of reporting can become a routine component of the software in their offices. These data can be integrated directly with the increasingly sophisticated tools the FDA is using to assess patterns in the data for potential safety signals.

One example of where these tools could provide an early warning is for a new drug that has a rare but serious toxicity to the liver. Similar safety problems have been at the heart of a number of recent drug withdrawals. If a new drug is launched that has a certain rare toxicity to the liver, a real-time surveillance network might eventually be able to detect subtle elevations in the liver enzyme tests of patients who were started on the drug and also happened to have blood work done around the same time. If enough of these signals were detected, it might alert the FDA that there is a potential liver toxicity and allow the agency to intervene before real harm is done to any patients.

In the old system, such a side effect might have gone unnoticed until a few patients developed severe liver failure; even then, it might have been hard to link the problem to the medicine without taking months to go back and review the medical records of many thousand of patients who were started on the same medicine. Although more widespread use of these systems requires greater adoption of EMRs, there is already a critical mass of EMRs and a lot that can be gained by conducting real-time surveillance on the existing IT infrastructure inside many large health care networks and academic medical centers. The FDA has already formed collaborations to start exploring these linkages with some large health networks, including Veterans Affairs (VA) hospitals and Columbia Presbyterian Hospital in New York. Expanding these efforts will again require additional funding.

Retrospective determination tools. The second step for improving the monitoring of drug safety is to develop better retrospective determination tools to complement the better detection systems described above. Determination tools are analytical capabilities for evaluating potential safety signals from data developed by the use of new drugs and for establishing a causal link between a drug and a suspected side effect. By making more routine use of very large medical data sets, to retrospectively evaluate how many thousands or in some cases millions of patients fared when they took a new drug, the FDA would be in a better position to evaluate whether a drug is causing a rare side effect. Efforts to make better use of electronic health care information to more easily conduct these kinds of practical studies are already well under way inside the FDA. Work on these kinds of determination capabilities is further along, for example, than is development of the rapid detection systems. A practical study that the FDA conducted in collaboration with Kaiser Permanente provided some of the first precise evidence that Vioxx might be associated with certain adverse cardiovascular events and indicated the need for further, prospective study of this potential safety problem. (In addition to collaborations with Kaiser Permanente, the FDA has worked with United Health Group, Harvard Pilgrim Health Plan/Fallon Clinic/Health Partners, Vanderbilt University, and AdvancePCS Dimension Rx, to retrospectively mine health information data to study suspected side effects with drugs.)²¹

A role for drug makers. All of the efforts described above have become a routine part of some of the FDA’s architecture for drug safety, yet these activities remain underused, fragmented, and poorly funded. They could grow up faster if they had wider support, beyond government. Drug manufacturers are in a particularly good position to help expand on efforts, by creating proactive online registries at the time of drug approval to collect safety data from sufficient numbers of patients until a true risk-benefit analysis can take place. The technology to do these things easily already exists, and this approach has been used in several other countries.

Moreover, the Centers for Medicare and Medicaid Services (CMS) recently discussed publicly a policy of encouraging and underwriting practical data collection around coverage decisions for new medical products.²² Private and public payers have an incentive to sponsor practical data collection to help better delineate the benefits of new medical products and stimulate more effective and efficient prescribing habits by doctors. This data collection, which would include development of clinical data registries, also could be used to help monitor for potential problems with drug safety.

The Medicare model. The policy promulgated by Medicare in particular could become a model for how this evidence could be collected in the private sector. Postmarketing data collection is one area where the two agencies—the FDA and the CMS—as well as product developers, patients, and doctors could benefit from better coordination and collaboration. In particular, this kind of data collection should not become an additional and costly requirement on top of the postmarketing studies that manufacturers are required to do after the time of approval. Instead, the same kind of practical data collection should be able to satisfy the concerns of both regulators and payers.

In particular, the policy developed by the CMS, which was recently articulated in a guidance document issued by the agency, provides a framework for just such a postmarketing data collection system for the small group of drugs for which the CMS makes national coverage decisions (mostly oral cancer drugs and injectable biologics delivered in doctors’ offices).²³ The CMS policy envisions low-cost ways of developing more evidence on risks and benefits in actual practice for technologies that are generally shown to be safe and effective for some populations but for which doctors and patients still have insufficient information for answering practical questions such as effects in certain subgroups of patients, effects in settings that differ from those in the trial, and risks and benefits in off-label uses. The CMS used this approach with its coverage of implantable cardiac defibrillators; as a result, it expanded coverage more broadly and did it more quickly.

Among other things, the additional evidence that the agency will be collecting provides CMS medical staff with more confidence that doctors and patients will make effective decisions. This allows the agency to make reimbursement for the treatment broader than would have been the case otherwise. In other words, selective decisions that would have been made by CMS medical staff to narrow coverage can be supplanted by broader reimbursement and then more careful decision making at the bedside, assuming that appropriate information to guide optimal clinical practice is being independently collected, analyzed, and disseminated in the medical community.

The second element of the policy is to provide new support for clinical studies in new indications where a treatment’s safety and effectiveness have not been demonstrated. The recent example undertaken by the CMS was explicit funding for some randomized studies for colorectal cancer drugs for indications that were not only off-label but not even mentioned in any drug formulary (again because the existing evidence is so limited). Prior to this policy, use of these drugs in off-label indications would not have been covered by the CMS at all.

Reimbursement can be a powerful tool to drive data collection on the drugs where Medicare has legal authority to make Part B coverage determinations for outpatient drug payment. This is especially true if payers are willing to provide more flexible coverage on the assumption that narrow coverage decisions can now be supplanted with more routine data collection. The CMS policy recognizes the fact that if good information is being collected and made available to patients and doctors, then on the aggregate, consumers and physicians will be making optimal decisions about the use of new technology for individual patients.

Uses of practical data sets. Although Medicare is also going to be collecting much information on prescription drug use as part of its new drug benefit, it is less clear how the agency can aggregate and use this information. Clearly, if the drug benefit provides easier mechanisms for collecting the information, it should be made available in some form for exploring both new risks and benefits of these medicines. Although the CMS has the legal authority to collect this information, it clearly does not have the authority to make Part D coverage decisions. But it can work to develop better evidence to support private decisions about using the drugs.

To these ends, the ability to also tease out new benefits using the same kind of population-based information is an important opportunity that should not be ignored. It should be a necessary component of any one of these efforts. Here the FDA has been the primary obstacle, and additional guidance and rule making could give product developers appropriate incentives to participate in this kind of practical data collection by allowing companies to talk publicly about the results of these practical studies.

Specifically, the FDA should allow drug companies to use conclusions reached from practical data sets to update labels and make limited claims in marketing material. Since the New Drug Approval (NDA) process already establishes that a drug is safe for marketing, there is little reason why Supplemental New Drug Applications (SNDAs) that seek additional indications cannot rely on conclusions that were derived wholly or in part from practical data. The FDA’s risk-benefit balancing may be different for new indications, but since the drugs are already available for off-label use in follow-on indications, the added information derived from practical data, to better define both the efficacy and the safety of the drug, would outweigh the alternative—the total absence of data. Right now, the FDA’s hear-no-evil, see-no-evil approach to the off-label use of drugs ignores the realities taking place in the medical setting and often denies patients the opportunity to receive exiting information about these off-label uses, simply because the trials do not conform to the single standard that the FDA sets for generating information the agency is willing to use to approve efficacy claims.

The statistical standard used in a practical trial, where patients are not randomized and participants are not blinded, would be lower than that used in the NDA setting. But these kinds of trials can nonetheless be rigorous if sample sizes are large and the statistical tools for evaluating practical data are well defined and precise. Statistical tools for conducting rigorous analyses using these kinds of practical data sets and methods for aggregating very large collections of data have become far more advanced since randomized trials were first conceived as a part of the drug approval process.

Once better systems for active drug surveillance are in place, there is also the potential for doing better practical as well as randomized clinical studies at a much more affordable cost. For example, drug developers and the FDA will be able to perform more efficient postapproval studies using clinical protocols administered over the Internet via electronic records, with patients selected and results recorded directly from EMRs , which will allow low-cost clinical trials to be performed more quickly and easily in real-world medical settings.²⁴

What the FDA can do. The FDA has already taken some steps to create more active and proactive surveillance tools. With improved resources for conducting this kind of surveillance, as well as resources for undertaking large simple safety studies in collaboration with health care networks on newly approved products, the FDA can improve its safety-monitoring program without burdening its drug approval process. Most importantly, by partnering with health care providers, institutions, product developers, as well as other government agencies, the FDA will more quickly and thoroughly identify and understand the risks associated the products it regulates. This information will improve the FDA’s ability to communicate timely concerns and prevention strategies and help improve the safety of medical care.

With all of the recent advances in the science behind the discovery of new drugs, we should be investing commensurate resources in bringing twenty-first-century science to the task of ensuring their safety.

Editor's Notes

 
Scott Gottlieb (Scott.Gottlieb{at}mssm.edu) is a fellow at the American Enterprise Institute in Washington, D.C. He was a senior policy adviser to the Food and Drug Administration (FDA) commissioner and the administrator of the Centers for Medicare and Medicaid Services (CMS).

NOTES

Top Current Deficits Of The... Making A Slow System... Developing Safety Information In... NOTES

 

1. M. Pirmohamed et al., "Adverse Drug Reactions as Cause of Admission to Hospital: Prospective Analysis of 18,820 Patients," British Medical Journal 329, no. 7456 (2004): 15–19.[Abstract/Free Full Text]
2. G.A. Fitzgerald, "Coxibs and Cardiovascular Disease," New England Journal of Medicine 351, no. 17 (2004): 1709–1711[Free Full Text]; U.S. Food and Drug Administration, "Worsening Depression and Suicidality in Patients Being Treated with Antidepressant Medications," FDA Public Health Advisory, 22 March 2004, www.fda.gov/cder/drug/antidepressants/AntidepressanstPHA.htm (26 April 2005); and H. Jick, J.A Kaye, and S.S. Jick, "Antidepressants and the Risk of Suicidal Behaviors," Journal of the American Medical Association 292, no. 3 (2004): 338–343.[Abstract/Free Full Text]
3. J. Appleby et al., "FDA Asks Pfizer to Suspend Bextra Sales," USA Today, 7 April 2005.
4. R.B. Reich, "A Suitable Remedy; When the FDA Is Weak," Washington Post, 9 January 2005.
5. J.D. Kleinke and S. Gottlieb, "Is the FDA Approving Drugs Too Fast?" British Medical Journal 317, no. 7163 (1998): 899.[Free Full Text]
6. Tufts Center for the Study of Drug Development, Impact Report, vol. 5, no. 3 (2003).
7. E.J. Emanuel et al., "The Costs of Conducting Clinical Research," Journal of Clinical Oncology 21, no. 22 (2003): 4145–4150.[Abstract/Free Full Text]
8. E. Tanouye, "American Home Withdraws New Drug," Wall Street Journal, 23 June 1998.
9. G.M. Lamb, "Drug Tests: Too Speedy—or Safe Enough?" Christian Science Monitor, 6 January 2005.
10. M. Kaufman and B.A. Masters, "FDA Is Flexing Less Muscle: Some Question Its Relationship with Drugmakers," Washington Post, 18 November 2004.
11. A. Trontell, "Expecting the Unexpected—Drug Safety, Pharmacovigilance, and the Prepared Mind," New England Journal of Medicine 351, no. 14 (2004): 1385–1387.[Free Full Text]
12. T.J. Moore, B.M. Psaty, and C.D. Furberg, "Time to Act on Drug Safety," Journal of the American Medical Association 279, no. 19 (1998): 1571–1573.[Free Full Text]
13. Ibid.
14. FDA, Center for Drug Evaluation and Research, 2002 Report to the Nation: Improving Public Health through Human Drugs, www.fda.gov/cder/reports/rtn/2002/rtn2002.pdf (11 May 2005).
15. S.J. Landers, "Doctors Key to Triggering FDA’s Warning Signals," American Medical News, 17 November 2003.
16. FDA, CDER, 2002 Report to the Nation.
17. Ibid.
18. Ibid.
19. FDA, Office of Financial Management, Food and Drug Administration FY 2004 Justification of Estimates for Appropriations Committees, www.fda.gov/oc/oms/ofm/budget/2004/toc.htm (11 May 2005).
20. Trontell, "Expecting the Unexpected."
21. FDA, Office of Drug Safety Annual Report FY 2003, 28 December 2004, www.fda.gov/cder/Offices/ODS/AnnRep2003/default.htm (26 April 2005).
22. S.R. Tunis, "A Clinical Research Strategy to Support Shared Decision Making," Health Affairs 24, no. 1 (2005): 180–184.[Abstract/Free Full Text]
23. Centers for Medicare and Medicaid Services, "Draft Guidance for the Public, Industry, and CMS Staff: Factors CMS Considers in Making a Determination of Coverage with Evidence Development," 7 April 2005, www.cms.hhs.gov/coverage/download/guidanceced.pdf (23 April 2005).
24. M.B. McClellan, "Protecting and Advancing America’s Health through Twenty-First-Century Patient Safety" (Speech at the Urban Institute, Washington, D.C., 12 November 2003).

                      What's this?

This article has been cited by other articles:

				R. Platt and A. Ommaya A Beneficial Side Effect of the Medicare Drug Benefit N. Engl. J. Med., December 29, 2005; 353(26): 2742 - 2743. [Full Text] [PDF]