This Article Abstract Submit a response to this article Services E-mail this article to a friend Alert me to new issues of the journal

E L E C T R O N I C  P R E S C R I B I N G
E X P E R T  C O N S E N S U S
W E B  E X C L U S I V E   

25 May 2004  
  

Recommendations For Comparing 
Electronic Prescribing Systems: 
Results Of An Expert Consensus Process

Guidance to help early adopters and policymakers
select the systems most likely to benefit patients.

By Douglas S. Bell, Richard S. Marken, Robin C. Meili,
C. Jason Wang, Mayde Rosen, Robert H. Brook,
and the RAND Electronic Prescribing Expert Advisory Panel

ABSTRACT:

Commercially available electronic prescribing systems may differ in their effects on patients’ health outcomes and on patients’ ability to manage costs. An expert panel convened to recommend specific features that would enable electronic prescribing systems to advance these goals. The panel authored sixty recommendations and rated each using a modified Delphi process. Ratings identified fifty-two recommendations as clearly positive for patient safety and health outcomes and forty-three recommendations as achievable in the average clinician’s office within three years. Overall, these recommendations offer a synthesis of evidence and expert opinion that can help guide the development of electronic prescribing policy.

Electronic prescribing systems are computerized systems that clinicians use to prescribe medications. Both clinicians and policymakers expect these systems to improve health care quality.¹ Although the evidence to support these expectations is limited, the implementation of electronic prescribing in a few academic hospitals has been associated with decreased medication errors, improved formulary adherence, and shorter lengths-of-stay.² Few providers use electronic prescribing, but new incentives for its adoption are emerging.³ The Medicare Prescription Drug, Improvement, and Modernization Act (MMA) of 2003 will fund grants for implementation of electronic prescribing, and it specifically permits third parties to fund such systems on behalf of providers. In addition, the Leapfrog Group has made electronic prescribing a quality criterion for hospitals.⁴

Provider organizations wishing to implement electronic prescribing can choose from a variety of systems, but implementation can be difficult, and providers have no basis for selecting those that can improve health care quality.⁵ Furthermore, funding from third-party organizations could shape these systems in ways that distort medication choices toward the funders’ interests. Recommendations are therefore needed to ensure that electronic prescribing systems meet patients’ needs for better health outcomes and a greater ability to manage their own costs.

Recommendations for ensuring the effectiveness of electronic prescribing would ideally be based on scientific evidence, but the available evidence is insufficient to guide comparisons among alternative systems.⁶ When evidence is incomplete, expert opinion can offer valid guidance through the use of the modified Delphi panel process.⁷ In this process, a panel of experts initially rates a set of recommendations, they then meet to discuss their ratings, and finally each panelist has a chance to revise his or her ratings in light of the discussion.⁸

In this paper we present the results of a modified Delphi panel process that produced recommendations for judging an electronic prescribing system’s ability to meet the goals of improving patient safety and patients’ ability to manage costs without damaging patients’ privacy or clinicians’ acceptance of electronic prescribing. The recommendations are intended for clinicians and provider organizations to use in selecting among electronic prescribing alternatives, for manufacturers to use in designing future electronic prescribing systems, and for policymakers to use in promoting the adoption of electronic prescribing systems that advance health care quality. The recommendations focus primarily on systems for outpatient office environments. However, inpatient and outpatient prescribing have many basic processes in common; thus, wherever possible, we constructed the recommendations to make them applicable to both environments.

Study Methods

We recruited a panel of recognized experts from professional domains related to electronic prescribing: medicine, nursing, pharmacy, managed care, pharmacy benefit management, consumer advocacy, medical informatics, health care oversight, health care quality and safety, and health economics. The panel chair was recruited first; he then worked with project staff to select the remaining panelists. (Panel members are listed at the end of the paper.) The project team provided panelists with a manuscript that included a conceptual framework for electronic prescribing evaluation and a systematic review of the relevant evidence.⁹ At an initial meeting in December 2001, the panelists used a nominal group process to create forty-one initial draft recommendations for electronic prescribing. Using an affinity-grouping procedure, project team members merged similar recommendations.¹⁰ The team then rephrased the draft recommendations to read as standards that could be objectively assessed in comparing electronic prescribing systems and wrote a rationale statement for each based on the evidence, conceptual framework, and panel discussion. From June through December 2002, the recommendations and rationale statements were substantially revised and extended by the panel through two rounds of preliminary ratings, teleconference discussions, and written comments. This process resulted in a set of sixty final recommendations.

Panelists rated the expected effects of each final recommendation along four dimensions: patient safety and health outcomes, helping patients manage costs, maintaining patient privacy, and promoting clinician acceptance. The fifteen-point rating scale ranged from –7 to +7, where –7 represented the largest negative impact on the dimension and +7 represented the largest positive impact. Panelists were asked to use ratings of +3 or higher to indicate “clearly positive” effects and ratings of –3 or lower to indicate “clearly negative” effects.¹¹ Clearly positive and negative effects were those deemed “major” enough to warrant consideration by anyone choosing among electronic prescribing alternatives.

Panelists were also asked to judge the time frame within which each recommendation could be implemented in the average clinician’s office, using two criteria: whether the recommendation is technically feasible, and whether the implied workflow changes could be achieved by the average clinician. To assist panelists in assessing technical feasibility, we provided them with preliminary results of a field study that assessed the implementation of each recommendation by six commercially available outpatient electronic prescribing systems.

In March 2003 the panel held a teleconference to discuss its initial ratings of the final recommendations. Panelists then submitted final revisions of their ratings. For each recommendation we determined the median rating on each dimension. For each median, we also examined the spread of the panelists’ ratings for “disagreement,” which was defined as having more than one panelist giving a “clearly positive” rating and more than one panelist giving a “clearly negative” rating. To identify relationships among the recommendations, four authors (Douglas Bell, Richard Marken, Robin Meili, and Jason Wang) independently reviewed each recommendation and listed dependencies or underlying mechanisms. Team members met to discuss and achieve consensus on these interpretations.

Study Results

Final recommendations. All sixty final recommendations received a median rating in the clearly positive range (+3 or greater) for at least one of the health care dimensions. For patient safety and health outcomes, fifty-two had clearly positive median ratings. For helping patients manage costs, eighteen had clearly positive medians. None of the median ratings were in the clearly negative range (–3 or less) for any of the final recommendations. Forty-three recommendations were rated as being achievable in the average physician’s office within three years. The complete final recommendations and median ratings are available in an online data supplement.¹²

The final recommendations were organized into ten categories, seven of which correspond to the steps that prescribers follow in managing a patient’s medications, and three of which relate to broader aspects of electronic prescribing (for example, security and confidentiality).¹³ Here we summarize the highest-rated recommendations for patient safety and health outcomes that were also considered achievable within three years.

Patient identification. Electronic prescribing usually begins with selecting the patient’s name from a menu. To mitigate the risk that slips in these menu selections would send prescriptions to the wrong patient, systems should display the selected patient’s identifying information throughout the process of creating a prescription (recommendation 1). To facilitate integration of prescribing with other patient information, patient-identifying data should be imported from electronic medical record or practice management systems (recommendation 2), but because these linkages may intermittently fail, electronic prescribing systems should also provide for manual entry of these data (recommendation 3). Exhibit 1 shows the median ratings for these recommendations.

Access to patients’ historical data. Before selecting a medication, prescribers often need to review elements of the patient’s history. The patient’s current medication list was rated as a critically important element for electronic prescribing systems to provide (recommendation 7). The current medication list should integrate data from all clinicians caring for the patient, but to mitigate privacy threats, this list should be available only to prescribers with care responsibility for the patient. To improve the accuracy and completeness of the list, systems should extract the patient’s prescription data from external pharmacy and hospital systems (recommendation 5) and support the manual entry of prescription information that is not available from external systems (recommendation 9). Exhibit 2 shows the highest-rated, immediately achievable recommendations in this category.

Medication selection. After deciding that a medication is indicated, prescribers must specify a medication and dose. Some prescribing systems assist prescribers at this step by listing the medications that are indicated for particular diagnoses.¹⁴ As shown in Exhibit 3, panel members considered this feature important for patient safety and health outcomes (recommendation 13), but their ratings also indicate that systems should not force the assignment of a diagnosis (recommendation 14). Some electronic prescribing systems attempt to influence prescribers by altering the order in which medications are presented or by displaying special symbols (such as an asterisk) next to favored or disfavored options. The panel recognized that this potentially beneficial feature could also be used to create commercial advantages for third parties. To curb these potential conflicts of interest, the panel strongly recommended that the display of medication options should not be influenced by promotional considerations (recommendation 16).¹⁵ Furthermore, the meaning of any symbols or special typefaces used to differentiate medication choices should be made clear (recommendation 17), and the rationale for these and any other medication recommendations should be immediately available (recommendation 18).

Alerts and other messages to prescribers. After a medication is selected, many electronic prescribing systems attempt to alert prescribers to potential problems. Exhibit 4 shows that the panel strongly recommended safety alerts for contraindications or significant precautions (recommendation 27). However, systems should implement methods for prioritizing alerts and allow prescribers to suppress less clinically important alerts (recommendations 31 and 32), and prescribers should be able to clearly distinguish safety-related messages from other types of messages, such as formulary alerts (recommendation 30).

Patient education. Before finalizing a new prescription, clinicians should educate patients about the new medication and negotiate patients’ adherence to treatment. The panel recommended that electronic prescribing systems provide information tailored for patients on how and why the medication should be used (recommendation 39) and provide a printout of a complete current medication list in nontechnical language (recommendation 40).

Data transmission and storage. Once completed, prescription orders must be transmitted to a pharmacy. As shown in Exhibit 5, the panel recommended that prescribing systems enable these transmissions following established standards from Health Level 7 (HL7) or the National Council for Prescription Drug Programs (recommendations 41 and 42). Data transmissions should use National Provider Identifiers when these become available (recommendation 44).¹⁶ Prescribers should also be notified when transmissions fail (recommendation 46).

Monitoring and renewals. Monitoring is an important but underappreciated step in medication management.¹⁷ The panel expected important benefits from alerts for patient adherence problems (recommendation 48) and from assistance with ordering and tracking laboratory monitoring tests (recommendations 49–51), but none of these recommendations was rated as achievable in three years.

Transparency and accountability. Many developers and implementers of electronic prescribing are receiving support from third-party organizations that have incentives to influence the prescribing process. To help deter this sponsorship from introducing biases in prescribing, the panel made several recommendations intended to open the results of third-party support to scrutiny. Some of these recommendations modify individual steps in the prescribing process (recommendations 18, 29, 34, and 45).¹⁸ In addition, disclosing the details underlying clinical decision-support rules would further expose potential biases (recommendation 54), and full disclosure of third-party support should help focus scrutiny where biases are most likely (recommendation 53).

Prescriber-level feedback. Most systems collect data that could be used to give feedback about prescribing behavior. Systems could promote practice-based learning by giving clinicians access to their own prescribing patterns (recommendation 55). Profiling the use of alert overrides may help to identify unintended problems with alerts (recommendation 56), but some panelists expressed the belief that clinicians would find this capability unacceptable, given the potential for its use in a punitive manner. The rating of this recommendation on the dimension of “promoting clinician acceptance” was one of only two ratings that met the formal criteria for disagreement among panelists.¹⁹

Security and confidentiality. Patients need assurances that prescribing data will not be used to harm them—for example, through disclosure to a prospective employer. However, the panel concluded that adherence to the privacy and security rules arising from the Health Insurance Portability and Accountability Act (HIPAA) of 1998 would provide the necessary protections (recommendation 57).²⁰ In addition, audit trails (recommendation 58) and role-based access privileges (recommendation 59) are strongly recommended for privacy (Exhibit 6).

Core electronic prescribing capabilities. In examining the relationships among recommendations, we noted that some recommendations exist to “refine” other recommendations, meaning that they specify particular details of another recommendation’s implementation. These “refining” recommendations would be applicable only if the core capability were implemented. Other recommendations serve to “support” a core capability by contributing toward its implementation. Here we summarize four core capabilities to which many of the recommendations relate.

Diagnosis-based medication menus. Several recommendations (15–19) serve to “refine” the selection of medications by diagnosis (recommendation 13) because they call for specific modifications of diagnosis-based menus. Two other recommendations serve to “support” diagnosis-based menus—external interfaces could make historical diagnosis data available (recommendation 5), and permitting prescribing without entering a diagnosis would help prevent the accumulation of inaccurate diagnoses (recommendation 14).

Safety alerts. Nine recommendations (29–37) refine the core safety-alerting capability (recommendation 27); the highest-rated ones call for systems to clearly distinguish safety messages from other messages and to prioritize safety alerts according to their clinical importance. Safety alerts would also be supported by having a current medication list (recommendation 7) to provide data for detecting drug interactions, and by the ability to correct or flag erroneous patient data (recommendation 38) to reduce the triggering of nuisance alerts.

Current medication list. Four recommendations (5, 8, 9, and 12) serve to support the achievement of an accurate and comprehensive current medication list (recommendation 7). For example, when prescribers issue an order to discontinue a medication (recommendation 12), the pharmacy system would record that fact and take the drug out of the patient’s current medication list.

Systems integration. Three separate recommendations (2, 5, and 47) call specifically for the integration of electronic prescribing systems with practice management systems, electronic medical records, and other external systems. Four recommendations (6, 42–44) refine systems-integration features, and two recommendations (2 and 4) support integration. Systems integration, in turn, is a prerequisite for accurate safety alerts, current medication lists, patient monitoring, and other recommended capabilities.

Discussion

Electronic prescribing is expected to improve many aspects of health care, but these high expectations will be achieved only if provider organizations select systems that have the appropriate features. We report a set of sixty recommendations for electronic prescribing that resulted from a modified Delphi process integrating scientific evidence, expert opinion, and observations from a field study of electronic prescribing systems. Each recommendation describes a feature expected to have a positive effect on at least one important dimension of health care performance. The recommendations should aid purchasers of electronic prescribing in comparing alternative systems, and they may also help electronic prescribing vendors to prioritize their development efforts.

The results of this study should contribute to current policy efforts. MMA contains provisions intended to accelerate the adoption of electronic prescribing, both through grants and by authorizing third-party funding for electronic prescribing systems. Qualifying systems will need to meet federal standards, which are to be developed with the goals of promoting safety, quality of care, and cost savings. Our results provide information that federal policymakers can use in developing standards that are likely to achieve these goals.

Some of the recommendations with the greatest expected benefits were among those considered not immediately achievable. These included the highest-rated recommendations (21 and 22) for helping patients manage their costs and recommendations (28 and 48) that would reduce the underuse of highly effective medications, a major quality problem for elderly Medicare beneficiaries.²¹ The main barrier to achieving these recommendations is the lack of integration among information systems used by physicians, pharmacies, laboratories, hospitals, payers, and others responsible for patient care in a given community. In the current environment, these separate entities have little incentive to share data. Policy initiatives are needed to encourage the development of business, political, and technical infrastructure that would enable communitywide integration of health information systems. The Agency for Healthcare Research and Quality (AHRQ) is beginning to fund the development of such infrastructure, but these efforts will likely need to be sustained and expanded.²² In the meantime, providers considering the implementation of electronic prescribing should not lose sight of the benefits that can be gained from more achievable electronic prescribing capabilities.

Although federal grants under MMA are not scheduled to begin until 2007, third-party initiatives to promote electronic prescribing are already under way.²³ Our panel’s recommendations for transparency and accountability can address the immediate need for preventing biases as third-party funding in this area grows. In general, the panel believed that substantial biases could be deterred through full disclosure of any third-party sponsorship coupled with full disclosure of decision-support rules. However, as the complexity and power of electronic prescribing grow, subtle distortions could become more difficult for providers and consumer groups to detect. Further research should address this possibility, and if the potential for bias appears significant, an independent group with expertise in evidence-based medicine may be needed to monitor the development of decision support.

More research is needed into most of the effects that the panel postulated, but the most fruitful questions for research may be those on which panelists disagreed. The panel’s most significant disagreement centered on whether fallibility of clinicians or of electronic prescribing systems should pose a greater concern. Although the patient safety literature provides ample evidence of clinician fallibility, emerging evidence suggests that prescribing system design flaws can also introduce unanticipated hazards.²⁴ Research is needed into the ways that electronic prescribing systems may create new hazards. Until more evidence is available, providers who install an electronic prescribing system should be vigilant for new types of adverse events.

More research is also needed into physician adoption of electronic prescribing. The most rigorous surveys available show that adoption remains low overall.²⁵ Although the panel included several practicing clinicians and clinician-managers, it did not represent the broad range of clinicians in community practice. Thus, its ratings for promoting clinician acceptance should be followed up with additional research on adoption among representative groups of clinicians. As new evidence emerges, ongoing efforts will be needed to keep these recommendations updated. New electronic prescribing tools or approaches may have emerged in the year since the recommendations were finalized, but we are not aware of any new evidence that would warrant immediate changes in the recommendations.

The recommendations we have presented here offer a starting point to guide early adopters of electronic prescribing toward the systems that are most likely to benefit patients. However, for the majority of patients to experience these benefits, substantial additional investments will be needed to foster adoption and to gather adequately detailed evidence about the effectiveness of electronic prescribing. Electronic prescribing will achieve its full potential if these investments are made with continuing attention to the features that would most improve patients’ safety and health.

The following people were members of the RAND Electronic Prescribing Expert Advisory Panel. The professional domains that each panelist represented are shown in parentheses. Donald M. Berwick, Institute for Healthcare Improvement (panel chair; health care quality and safety); Phyllis Borzi, George Washington University (consumer advocacy); Lonnie R. Bristow (medicine); Christine K. Cassel, American Board of Internal Medicine (medicine); Schumarry H. Chao, MedImpact (pharmacy benefit management); Paul B. Ginsburg, Center for Studying Health System Change (health economics); Peter I. Juhn, CareTouch Inc. (managed care, health care quality and safety); Helene Levens Lipton, University of California, San Francisco (pharmacy); Clement J. McDonald, Regenstrief Institute For Health Care (medical informatics); Mary O’Neil Mundinger, Columbia University (nursing); and Margaret E. O’Kane, National Committee for Quality Assurance (health care oversight). This study was funded through a contract from Pfizer Inc. that obligated RAND to publish its findings regardless of the study’s outcome. Pfizer’s aim for this study was to make health care quality and patients’ health outcomes the focus of competition among electronic prescribing systems. Pfizer did not select any of the panelists. Pfizer representatives were permitted to observe panel meetings without comment. Pfizer had the right to comment on manuscripts for publication, but RAND was not obliged to respond to Pfizer’s comments. The study was approved by institutional review boards at RAND and the University of California, Los Angeles. The authors are grateful for editorial assistance from Sydne Newberry and Kristen Mukae.

NOTES

1. G.D. Schiff and T.D. Rucker, “Computerized Prescribing: Building the Electronic Infrastructure for Better Medication Usage,” Journal of the American Medical Association 279, no. 13 (1998): 1024–1029.
2. R. Kaushal and D.W. Bates, “Computerized Physician Order Entry (CPOE) with Clinical Decision Support Systems (CDSSs),” in Making Health Care Safer: A Critical Analysis of Patient Safety Practices, ed. K.G. Shojania et al., Pub. no. AHRQ-01-E058 (Rockville, Md.: Agency for Healthcare Research and Quality, July 2001); D.W. Bates et al., “Effect of Computerized Physician Order Entry and a Team Intervention on Prevention of Serious Medication Errors,” Journal of the American Medical Association 280, no. 15 (1998): 1311–1316; R.S. Evans et al., “A Computer-Assisted Management Program for Antibiotics and Other Antiinfective Agents,” New England Journal of Medicine 338, no. 4 (1998): 232–238; D.W. Bates et al., “The Impact of Computerized Physician Order Entry on Medication Error Prevention,” Journal of the American Medical Informatics Association 6, no. 4 (1999): 313–321; J.M. Teich et al., “Effects of Computerized Physician Order Entry on Prescribing Practices,” Archives of Internal Medicine 160, no. 18 (2000): 2741–2747; and W.M. Tierney et al., “Physician Inpatient Order Writing on Microcomputer Workstations: Effects on Resource Utilization,” Journal of the American Medical Association 269, no. 3 (1993): 379–383.
3. J.S. Ash et al., “Computerized Physician Order Entry in U.S. Hospitals: Results of a 2002 Survey,” Journal of the American Medical Informatics Association 11, no. 2 (2004): 95–99; and D.J. Brailer and E.L. Terasawa, Use and Adoption of Computer-Based Patient Records (Oakland: California HealthCare Foundation, October 2003).
4. Leapfrog Group, “Patient Safety: Setting Standards: An Rx for Rx,” www.leapfroggroup.org/safety1.htm#CPOE (3 May 2004).
5. D.F. Doolan and D.W. Bates, “Computerized Physician Order Entry Systems in Hospitals: Mandates and Incentives,” Health Affairs 21, no. 4 (2002): 180–188, highlights the difficulty of implementation.
6. Kaushal and Bates, “Computerized Physician Order Entry”; and D.S. Bell et al., “A Conceptual Framework for Evaluating Outpatient Electronic Prescribing Systems Based on Their Functional Capabilities,” Journal of the American Medical Informatics Association 11, no. 1 (2004): 60–70.
7. P.G. Shekelle et al., “The Reproducibility of a Method to Identify the Overuse and Underuse of Medical Procedures,” New England Journal of Medicine 338, no. 26 (1998): 1888–1895; R.L. Kravitz et al., “Validity of Criteria Used for Detecting Underuse of Coronary Revascularization,” Journal of the American Medical Association 274, no. 8 (1995): 632–638; and J.V. Selby et al., “Variation among Hospitals in Coronary-Angiography Practices and Outcomes after Myocardial Infarction in a Large Health Maintenance Organization,” New England Journal of Medicine 335, no. 25 (1996): 1888–1896.
8. R.H. Brook et al., “A Method for the Detailed Assessment of the Appropriateness of Medical Technologies,” International Journal of Technology Assessment in Health Care 2, no. 1 (1986): 53–63.
9. Bell et al., “A Conceptual Framework.”
10. W. Nicholas et al., “Group Process Methods,” in Putting Practice Guidelines to Work in the Department of Defense Medical System, Pub. no. MR-1267-A (Santa Monica, Calif.: RAND, 2001).
11. Previous Delphi panels have used a nine-point scale, with the upper third of the scale (7–9) being used for clearly appropriate indications, and the lower third of the scale (1–3) being used for clearly inappropriate indications, as described in Brook et al., “A Method for the Detailed Assessment.” The scale used in our study spans fifteen points, but the upper and lower thirds of the scale have analogous interpretations.
12. The online supplement is available at content.healthaffairs.org/cgi/content/full/hlthaff.w4.305v1/DC2.
13. Bell et al., “A Conceptual Framework.”
14. Commercial drug-indication databases exist to support this feature, such as First DataBank, “Indications Module,”
www.firstdatabank.com/integrated_content/drug_information/clincial_content_us/index.asp?module_id=13 (2 February 2004); and Medi-Span, “Drug Indications Database,”
www.medi-span.com/products/product_drugindications.asp (2 February 2004). In the Netherlands a national database of drug indications is maintained for this purpose. See Nederlands Huisartsen Genootschap (Netherlands College of General Practitioners), EVS (Electronic Prescribing System), 2004, nhg.artsennet.nl/content/resources/AMGATE_6059_104_TICH_L1310214546/ (17 May 2004).
15. “Promotional considerations” were defined as payments or in-kind support given to providers, prescribing system vendors, or any other third party, in exchange for their promoting a medication or group of medications. This definition is intended to encompass circumstances that could put the provider or vendor’s financial interests in competition with the patient’s best interests. Note that medication formularies and other information related to the patient’s health care coverage would not fall within this definition.
16. U.S. Department of Health and Human Services, Office of the Secretary, “45 CFR Part 162. Standard Unique Health Identifier for Health Care Providers,” Federal Register 69, no. 15 (2004): 3434–3469.
17. T.K. Gandhi et al., “Adverse Drug Events in Ambulatory Care,” New England Journal of Medicine 348, no. 16 (2003): 1556–1564 .
18. To make vendors accountable without giving prescribers unwanted information, recommendations 18 and 29 require only that disclosures be available in the system upon the user’s request. The sponsorship referred to in recommendations 34 and 45 could be disclosed by simply including the sponsor’s name in the message. Thus, the panel considered all of these recommendation to be achievable within three years.
19. See Note 12.
20. U.S. Department of Health and Human Services, “Administrative Simplification in the Health Care Industry,” aspe.hhs.gov/admnsimp/index.shtml (5 February 2004).
21. T. Higashi et al., “The Quality of Pharmacologic Care for Vulnerable Older Patients,” Annals of Internal Medicine 140, no. 9 (2004): 714–720. See Note 12.
22. AHRQ, “Transforming Healthcare Quality through Information Technology (THQIT)—Implementation Grants,” grants.nih.gov/grants/guide/rfa-files/RFA-HS-04-011.html (10 April 2004).
23. R.L. Rundle, “WellPoint to Pay $30 Million for Physicians’ Computers,” Wall Street Journal, 15 January 2004.
24. J.S. Ash, M. Berg, and E. Coiera, “Some Unintended Consequences of Information Technology in Health Care: The Nature of Patient Care Information System Related Errors,” Journal of the American Medical Informatics Association 11, no. 2 (2004): 104–112; and Bates et al., “The Impact of Computerized Physician Order Entry.”
25. Brailer and Terasawa, Use and Adoption of Computer-Based Patient Records; and Ash et al., “Computerized Physician Order Entry.”

Douglas Bell (dbell{at}rand.org) is an associate natural scientist at RAND in Santa Monica, California; an assistant professor in the Department of Medicine, David Geffen School of Medicine, University of California, Los Angeles; and a Robert Wood Johnson Foundation Generalist Physician Faculty Scholar. Richard Marken is a senior behavioral scientist at RAND, where Robin Meili is a senior management systems analyst. Jason Wang is a doctoral fellow in the Pardee RAND Graduate School. Mayde Rosen is an associate natural scientist at RAND. Robert Brook, vice president, directs the RAND Health Program. Names and affiliations of the Expert Panel members are listed in the paper's acknowledgments.

Read related papers by: David Brailer, Jonathan Javitt, and W. Ed Hammond.

DOI: 10.1377/hlthaff.W4.305
©2004 Project HOPE–The People-to-People Health Foundation, Inc.