Health Affairs, 22, no. 3 (2003): 134-148 doi: 10.1377/hlthaff.22.3.134 © 2003 by Project HOPE	New Online   How Would Obama, McCain Cover The Uninsured?   Cost To Cover Uninsured   Let Health IT Market Evolve   How Rate Health IT Progress?   HIV/AIDS In India, China

This Article Abstract Reprint (PDF) Submit a response to this article Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services E-mail this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to My Personal Archive Download to Citation Manager Reprints & Permissions Citing Articles Citing Articles via HighWire Citing Articles via ISI Web of Science (24) Citing Articles via Google Scholar Google Scholar Articles by Marshall, M. N. Articles by Smith, P. C. Search for Related Content PubMed PubMed Citation Articles by Marshall, M. N. Articles by Smith, P. C. Related Collections Consumer Issues International Issues Quality Of Care

International Comparisons

Public Reporting On Quality In The United States And The United Kingdom

PROLOGUE: Report cards represent one of the most publicly visible aspects of hospitals’ quality improvement efforts, and they are not without controversy. Media coverage often overreacts to low "grades" given to hospitals in a region. Providers find ways to stack the decks, selecting the healthiest patients to improve their scores. Purchasers look only at costs, and consumers either can’t figure them out or ignore them completely. At least, this is the popular perception. The need for basic research to develop accurate and useful quality-related performance measures continues, but quality report cards are here to stay.

Problems, challenges, obstacles, and innovations mark the trail, but there is incremental progress toward widespread use of quality report cards in the United States and the United Kingdom, as Martin Marshall and his colleagues report. In the United States a strong emphasis on informing consumer choice and controlling costs has guided reporting. Different traditions and incentives are at work across the Atlantic. The two countries have much to learn from each other as they move toward greater accountability and quality in health care.

A part-time general practitioner (GP) in an inner-city practice and a professor of general practice at the National Primary Care Research and Development Centre, University of Manchester, England, Marshall is also vice-president of the European Working Group on Quality in Family Practice. He gained U.S. experience while a Harkness Fellow in Health Care Policy at RAND in California. Paul Shekelle is a consultant in health sciences at RAND; a professor of medicine at the University of California, Los Angeles, School of Medicine; and a staff physician at the VA Medical Center in West Los Angeles. He spent a year in the United Kingdom as an Atlantic Fellow in Public Policy. Huw Davies is a professor of health care policy and management at the University of St. Andrews, Scotland, where he also directs the Center for Public Policy and Management and the Research Unit for Research Utilization in the School of Social Science. Peter Smith is a professor of economics at the University of York, England.

Abstract

 
The public reporting of comparative information about health care quality is becoming an accepted way of improving accountability and quality. Quality report cards have been prominent in the United States for more than a decade and are a central feature of British health system reform. In this paper we examine the common challenges and differences in implementation of the policy in the two countries. We use this information to explore some key questions relating to the content, target audience, and use of published information. We end by making specific recommendations for maximizing the effectiveness of public reporting.

Historical background. An interest in public disclosure of performance data is not new in either country. In the 1860s Florence Nightingale highlighted differences in mortality rates of patients in London hospitals, and in 1917 the U.S. surgeon Ernest Codman complained that his colleagues failed to publish their results because they feared that the public might not be impressed.³ However, the vision of Nightingale and Codman has started to become a reality only in recent years, made feasible by developments in information technology and encouraged by rising popular expectations regarding patient choice and the accountability of health systems.

Objectives of reporting. Advocates of the public release of performance data are often unclear about the objectives of reporting initiatives and how they expect the various stakeholders to respond.⁴ This is likely to depend upon the context within which reporting takes place, but broadly there are two reasons for putting performance data in the public domain. The first is to increase the accountability of health care organizations, professionals, and managers. This greater accountability offers patients, payers, and purchasers a more informed basis on which to hold providers to account, either directly through purchasing and treatment decisions or indirectly through political processes.⁵ The second reason is to maintain standards or stimulate improvements in the quality of care provided, or both. A range of mechanisms are being used in the United States and the United Kingdom to achieve this aim: economic competition, performance management with or without incentives, or appeals to the professional interest of those working in health care in doing a good job. Within each of these mechanisms, the stakeholders—patients, the insured, purchasers, managers, and health professionals—are expected to play different roles.

Quality Reporting In The United States

Top Quality Reporting In The... Quality Reporting In The... A Comparison Of U.S.... Evidence On The Impact... Maximizing The Benefits Of... NOTES

 
Key U.S. initiatives. The United States has led the modern public disclosure movement. The first major U.S. report cards were published more than fifteen years ago by the federal government agency that administers Medicare.⁶ This initiative was withdrawn in 1993 following criticism of the validity of the rankings, but it led to a plethora of other performance reports produced by state governments, employers, consumer advocacy groups, the media, private enterprises, and coalitions.

Information is now readily available in the United States about the comparative performance of health insurance plans, hospitals, and individual physicians, and there has been an ongoing (and sometimes acrimonious) debate about the content of the data, the process of disclosure, and the associated merits and risks. Public reporting in the United States is now much like health care delivery in that country: It is diverse, is primarily market-based, and lacks an overarching organizational structure or strategic plan. Public reporting systems vary in what they measure, how they measure it, and how (and to whom) it is reported. Some exemplar organizations involved in public reporting of quality information are outlined below.

National Committee for Quality Assurance (NCQA). The NCQA is a nonprofit organization that evaluates health care quality, primarily of health maintenance organizations (HMOs). The NCQA’s Health Plan Employer Data and Information Set (HEDIS) is one of the oldest and best-known public reporting systems. It measures a growing number of mainly technical processes of care, collected from both administrative data and medical record review. Health plans participate voluntarily, and comparative quality information is posted on the NCQA’s Web site, www.ncqa.org.

Pacific Business Group on Health (PBGH). This is a consortium of large employers in California that promote public reporting (Healthscope, www.healthscope.org) in order to improve the quality of health care for their employees and other California residents. Health plans, hospitals, and medical groups are all included. Data come from patient surveys and providers’ self-reports.

National Quality Forum. The forum is a public-private, not-for-profit organization that does not produce its own public reporting system but rather promotes core sets of quality measures and standardized measurement specifications, collection, verification, and audit tools (www.qualityforum.org).

Leapfrog Group. Leapfrog is a coalition of major business purchasers (Fortune 500 companies) that does not produce public report cards but rather sets certain standards for processes of care (such as the use of computerized physician order entry and volume standards for certain procedures). It then encourages health care purchasers to require providers to meet these standards as a condition of participation in health insurance plans (www.leapfroggroup.org). Data come from self-reports by urban hospitals.

Healthgrades. This for-profit company uses its own analyses and those of others (including the Leapfrog Group) to present comparative information on hospitals for a range of health conditions, nursing homes, and home health agencies (www.healthgrades.com). Some of this information is distributed free, and some must be purchased.

State-based initiatives. States also have played an important role in driving public reporting forward. For example, New York State has produced and published what many regard as the most sophisticated report cards, focusing primarily on cardiac procedures.⁷

Centers for Medicare and Medicaid Services (CMS). The federal CMS (formerly HCFA) launched the hospital mortality report described earlier. It is now committed to developing individual physician-specific quality report cards and has selected diabetes care as the first condition. No decisions have yet been made regarding measures, but both process and outcome measures are under consideration. The CMS has also embarked on an ambitious public reporting project on the quality of long-term care facilities that included full-page advertisements in newspapers in major U.S. cities and all fifty states.⁸

Consumer Assessment of Health Plans (CAHPS). This is a government-sponsored effort to develop a common survey to be used by patients to assess the care they receive. It has been the subject of numerous studies assessing its psychometric properties and utility in real-life situations. Studies have shown that in experimental situations, CAHPS information can influence patients’ choice of providers; however, observational studies in real-world situations have reported mixed results.⁹

Key drivers. Public reporting of quality information in the United States is being propelled mainly by two factors. First, the "business case for quality" argues that high-quality care will lower business costs by reducing employers’ contribution to the health care costs of their employees (since U.S. employers pay for the majority of workers’ health care costs). Second, there is rapidly increasing interest in the use of "tiered pricing"—the coupling of the portion of health care costs that a person pays to the price of health care. For example, a person may face a minimal personal cost to be hospitalized at a hospital that is "preferred" by a health insurer but a personal cost of several hundred dollars a day to be hospitalized at other hospitals.

Price has been the dominant factor determining which hospitals are "preferred," but because of concern from employers and patients, publicly reported quality information is starting to play a part. The goal is to promote the use of providers that deliver the best quality for price ("value-based purchasing"). Tiered pricing had been introduced in many health plans in California. There is much ongoing work, both in the private sector and in academe, to determine the types of quality information people desire and the format in which they can understand and use the information.

Quality Reporting In The United Kingdom

Top Quality Reporting In The... Quality Reporting In The... A Comparison Of U.S.... Evidence On The Impact... Maximizing The Benefits Of... NOTES

 
Key U.K. initiatives. In contrast to the United States, there are relatively few examples of purposeful release of information about quality of care in the United Kingdom. Some basic information about hospital performance in England and Wales has been in the public domain since the early 1980s. These data had embraced hospital mortality rates by 1992 but, although publicly released, were intended principally for managerial rather than public use and had little discernible impact.¹⁰ There were also isolated examples of publishing comparative information of interest to a limited audience, such as renal transplant and in vitro fertilization success rates. The first reporting initiatives deliberately aimed at the public, known as the Patient’s Charter, focused on waiting times rather than clinical quality.

First report cards on hospitals. A range of hospital outcome data have been published in Scotland since the early 1990, but, again, the release of the information was purposefully low key to discourage hostile responses from clinicians and the media.¹¹ High-profile national reporting initiatives were not introduced in the United Kingdom until 1998, a decade later than in the United States. From this date, the United Kingdom has adopted a much more coordinated and strategic approach to public reporting than has been seen in the United States. A national Performance Assessment Framework (PAF) was introduced in 1998 that sought, among other things, to report clinical outcomes at the hospital level.¹² In 2001 the NHS Plan placed this initiative at center stage and became the first government publication to refer specifically to "report cards."¹³

Government hospital rating system. In September 2001 the Department of Health published a new system for rating the performance of all National Health Service (NHS) nonspecialist hospitals in England. Each was to be classified annually into one of four categories, from three to zero stars, depending upon their performance against a range of indicators and the outcome of their clinical governance review by the Commission for Health Improvement (CHI). Three stars were awarded to hospitals with the highest performance rating and a favorable CHI review and zero stars to hospitals with the poorest performance and an unfavorable review. The intention is to reward three-star organizations with increased financial and strategic autonomy. Zero-star hospitals will be investigated and senior management changed where necessary.

Private initiative. Most reporting schemes in the United Kingdom have been led by the national health ministry, but in late 2000 an independent initiative entered the arena. Dr Foster, established by two Sunday Times journalists, makes hospital performance data available on the Web and sells its information to the media (www.drfoster.co.uk). The group published its first Good Hospital Guide as a supplement to the Sunday Times in January 2001, subsequently produced a Good Birth Guide, and in March 2002 produced a second Good Hospital Guide.

The Dr Foster guides report on the performance of all public acute care hospitals and the majority of hospitals belonging to the main independent providers. The early guides contained information about hospital-specific mortality rates, number of staff, waiting times, number of complaints, and services. These data were not published in a ranked or "league table" format, but the most recent guide explicitly ranks the hospitals in terms of their relative performance. Dr Foster has invited a number of high-profile figures from the medical establishment to endorse its publications and has worked closely with the health ministry to secure compliance and coordinate reporting activities. The main contribution of the Dr Foster group has been to bring communication skills to performance reporting that government agencies had failed to achieve. It is currently focusing its resources on local rather than national reports and in the near future intends to publish data on individual specialists and primary care practices.

NHS surveys. The NHS has also initiated a series of surveys designed to monitor NHS performance from the patient’s perspective, offering systematic comparisons of experiences over time and between different parts of the country. An example is the survey of heart disease patients, conducted by Picker Europe, involving 194 NHS hospitals and more than 84,000 patients. An extensive report card on the performance of each hospital is available on the Internet.¹⁴ This reports on about seventy-five different aspects of the patient experience, with responses for each question compared with the best and worst 20 percent of hospitals nationally.

Future plans. The enquiry into pediatric cardiac surgery deaths at the Bristol Royal Infirmary revealed that a great deal of unpublished information on the high mortality rates at Bristol was available but not acted upon. It therefore recommended the creation of an independent Office for Information on Health Care Performance, a recommendation that has now been carried out by the national government. The new office will be responsible for the collection of data, the analysis of the data to identify good and poor performance, the publication of the data in the form of both national and local report cards, and the conduct of patient and staff surveys. In addition, the office will be responsible for assessing data quality and making recommendations to improve data systems. It will be part of a redesigned CHI, which will become an independent regulator of U.K. health care in both the public and independent sectors. Prompted in part by the Bristol Enquiry, the Society of Cardio-thoracic Surgeons publishes postoperative mortality rates by unit, and there are plans to publish thirty-day mortality rates for individual surgeons by April 2004.¹⁵

A Comparison Of U.S. And U.K. Public Reporting

Top Quality Reporting In The... Quality Reporting In The... A Comparison Of U.S.... Evidence On The Impact... Maximizing The Benefits Of... NOTES

 
In both countries quality report cards are seen as central to improving the accountability of health providers, a key lever to improving quality, and an important principle to pursue.¹⁶ In an open and democratic society it is inconceivable that the policy of increasing disclosure will be reversed. However, both countries face similar challenges as they attempt to engage the key stakeholders. Politicians and the media have embraced the idea with enthusiasm, but we summarize below a growing body of evidence to suggest that many consumers, purchasers, health professionals, and, to a lesser extent, provider organizations are either ambivalent, apathetic, or actively antagonistic toward report cards. There is still much that we do not know about public reporting, and there are major opportunities for collaboration between the two countries for finding the answers.

Reporting standards. Some important differences between the countries should be recognized when seeking to translate lessons from one setting to the other. In principle, the British health care system offers a comparative advantage for securing data standardization and universality of coverage (although the United Kingdom has until recently been reluctant to take advantage of this benefit of a unitary system). There is no reason in principle why the major payers in the U.S. system should not insist on minimum reporting standards as a condition of doing business.¹⁷ However, there is evidence that some U.S. providers are becoming reluctant to participate. For example, only 50 percent of California hospitals are now participating in a statewide initiative to report patients’ evaluations of their care.¹⁸

Diversity notwithstanding, the amount and quality of routinely available data are considerably greater in the United States. Problems with the coverage and quality of data have been a major barrier to progress in the United Kingdom, and a recent study of the quality of U.K. cardiac surgery data, supported by the Nuffield Trust, demonstrated some large inconsistencies among various data sources, even for an outcome as apparently unequivocal as death.¹⁹ There are plans to rectify this with a major investment in information technology over the next three years.

Report card design. U.S. policymakers and researchers are searching for ways of increasing popular engagement with report cards. The most recent initiative, using interactive Web sites that allow users to select features that they regard as relevant and then display this information in their chosen format, have great potential but have not yet been formally evaluated.²⁰ Traditionally, the British public have not been expected (and, indeed, at present have little capacity) to exercise choice on the basis of information. The policy focus has instead been on reducing variations between providers rather than encouraging patient choice. However, recent policy announcements have signaled major changes in the philosophy of the NHS, with patient choice, diversity of providers, and funding mechanisms for diagnosis-related groups (DRGs) playing a central role. Report cards are expected to be a central element in this new arrangement.

Provider incentives. The two health systems have focused on different incentives for providers to act on quality reports. The United Kingdom has emphasized greater autonomy and other nonfinancial rewards for provider organizations that report good performance. In contrast, the principal external motivators for U.S. providers take the form of economic incentives. High reported quality should in principle increase market share, and there is some evidence that report cards are used actively by high-performing organizations to that end.²¹ Experience with the U.S. "Rewarding Results" initiative and new contractual arrangements for U.K. clinicians should offer additional evidence in this domain.²²

Evidence On The Impact Of Public Reporting

Top Quality Reporting In The... Quality Reporting In The... A Comparison Of U.S.... Evidence On The Impact... Maximizing The Benefits Of... NOTES

 
There is a large body of research evidence from the United States on how the various stakeholders respond to public reports and their impact on the processes and outcomes of health care. There is considerably less evidence of this kind in the United Kingdom, although what does exist tends to support the key conclusions drawn from the U.S. data.²³

Impact on quality of care. Given that one of the two broad reasons for public disclosure of quality information is to maintain standards or stimulate improvements in the quality of care, it is surprising that there are few published studies on this subject. We could find no published data from randomized controlled trials that assess the effect of public reporting specifically on quality. The strongest existing evidence is based on observational studies of short-term mortality and morbidity following cardiac surgery. These indicate that U.S. states that have public reporting systems have experienced declines in cardiac surgery mortality that are more rapid than the declines in states without public reporting.²⁴

The Cleveland Health Quality Choice project has also been the subject of several evaluations, but the data are limited by their lack of a control group.²⁵ There are no data regarding the effect of public reporting on long-term outcomes of cardiac surgery or outcomes from care for other health conditions. There are some observational studies of the effect of public reporting on processes of care judged to be related to health outcomes, such as influenza vaccination or screening mammography. Prominent among these are the greater improvements over time in the processes of obstetrics care for those hospitals reported as low-quality outliers compared with other hospitals, and the observation that U.S. health plans that publicly reported their data showed greater improvements over three years on some HEDIS measures than did health plans that measured but did not publicly report the assessment of their care.²⁶

Consumers’ use of report cards. One possible mechanism for public reporting to stimulate efforts to improve the quality of care is consumer pressure. Several studies have demonstrated that U.S. consumers want more information about providers’ performance and are willing to identify the content and format of the information of greatest use to them.²⁷ However, most of the evidence (from both the United States and Scotland) suggests that when this information is published, the public does not search it out, does not understand it, distrusts it, and fails to make use of it.²⁸ There are some notable exceptions; for example, higher reported performance has been associated with greater employee enrollment and a lower desire to switch health care providers.²⁹

In the United Kingdom, recent focus group data indicate that some members of the public consider public reporting to be a punitive tool used by politicians to punish hard-working professionals, and they expressed concern about the practical implications of introducing report cards on general practice in the United Kingdom.³⁰ Some studies have shown that those who do respond to report cards are more likely to be young and well-educated.³¹ In addition, where user interest has been demonstrated, it appears to decline over time, which suggests that the public responds primarily to new information.³²

The predominant lack of user response has been explained in terms of difficulty in understanding the information, lack of trust in the data, problems with timely access, and lack of genuine choice.³³ However, recent evaluations of even the newest state-of-the-art report cards that address many of these potential barriers have failed to demonstrate significant or sustained public interest.³⁴ Because several of the above-cited studies dealt with consumer assessments of cardiac surgery report cards, the association between public reporting and improvements in health outcomes described above suggests that consumers’ use of report cards is not a necessary precondition if public reporting is to have an effect. Much work in the United States continues to try to understand how to present quality information in ways that are meaningful to consumers, but most experts do not believe that consumer pressure will be an important mechanism to stimulate quality improvements for the foreseeable future.³⁵

Purchasers’ use of report cards. Some commentators were not surprised by the lack of consumer interest in comparative data but did expect purchasers (mostly U.S. employers and U.K. Primary Care Trusts) to be more responsive on behalf of their constituency groups. Published evidence from the United States suggests that early interest in report cards quickly waned and that most purchasers were more interested in costs, or in gross indicators of quality such as accreditation, than in detailed comparative information.³⁶ However, in the past few years there has been an accelerated interest by some U.S. employers in the public reporting of quality information, stimulated by the burden placed on business by health care costs and the perception that higher-quality care may lead to healthier and more productive employees. In contrast, there is little evidence to indicate that U.K. purchasers of health care are yet using the comparative hospital performance data to guide their contracting decisions to any great extent.

Physicians’ use of report cards. While both American and British physicians are more aware of report cards than consumers are, they too make little use of report card information.³⁷ Evidence in the United States suggests that physicians distrust and attempt to discredit the data, and there are some examples of their responding defensively by demanding that their managers’ performance be judged using report cards.³⁸ Exceptions are emerging in the United Kingdom, where some professionally based collaboratives and networks (such as cardiothoracic surgeons) are beginning to support benchmarking arrangements based on publicly released data.

Hospitals and other provider organizations’ use of report cards. A growing body of evidence indicates that both U.S. and U.K. provider organizations are the most sensitive of the various stakeholder groups to report cards and can respond in ways that improve the quality of the care they provide.³⁹ U.S. hospitals that are shown by the data to be performing poorly are inclined, at least initially, to discredit the reports and question their value. However, a recent study suggests that for these organizations, the considered and private response is different: They use the published information to help them focus on quality issues, improve their internal data systems, and improve the quality of their care.⁴⁰

The media’s response. The media have played a leading role in promoting the use of report cards in both countries. However, considerable anxiety has been expressed about the media coverage of comparative information: in particular, the propensity of the media to be alarmist, to engender a culture of blame, and to present complex data as overly simplistic league tables. Evidence from both countries suggests that these claims are probably exaggerated, particularly when those responsible for the release of the data work closely with the reporting journalists.⁴¹

Unintended consequences. The focus of some quality reporting systems has been criticized for measuring "what can be easily measured" rather than "what is important." The evidence and perception that organizations and provider groups will devote resources to improve on any quality measures that are publicly reported create the possibility that other important areas of health care may suffer for lack of those same resources. There are no randomized trial data or observational data that associate public reporting in one health domain with worsening performance in other health domains. However, the potential for this and other unintended consequences of public disclosure have been described in health and nonhealth sectors and should be taken seriously.⁴²

Maximizing The Benefits Of Public Reporting

Top Quality Reporting In The... Quality Reporting In The... A Comparison Of U.S.... Evidence On The Impact... Maximizing The Benefits Of... NOTES

 
The debates over the merits or deficits of reporting health care quality have been extraordinarily heated. On the one hand, advocates of public reporting see the current reservations as a necessary evolutionary step and think that report cards will soon become an integral and accepted tool in a modern health care system. Opponents see report cards as largely unproven and a distraction, with potentially unhelpful side effects.

Our view is that—whatever the merits of the two arguments—the imperatives of accountability and quality improvement make the wider development and implementation of report cards inevitable. Public reports are here to stay, and the debate should now be moving on from whether to use them to how best to deploy them in particular circumstances. In this respect, public reporting should be treated like any other technology or policy option. Its benefits against stated objectives should be evaluated in the light of its costs, including both direct costs and inadvertent side effects.

Mandatory reporting. This paper has sought to summarize the principal costs and benefits. In the light of this evidence, how should policy on public reporting evolve? The first point is that—if it is to be effective—public reporting may need to be mandatory. Recent evidence from the United States suggests that where this is not the case, health care organizations may simply withdraw from a reporting scheme when they perceive that participation is not in their interest, thereby severely compromising its usefulness.⁴³ Mandatory reporting of course places a duty on the regulator to ensure that all reporting requirements are manifestly useful to stakeholders and cost-effective to the health system, and during developmental phases it may be more effective to rely on voluntary participation.

Tailoring the data. The next design issue concerns what data to report. Most public reporting schemes until now have opportunistically relied on readily available information. In the future there will be increasing pressure to tailor reports more closely to the needs of users, necessitating the implementation of new data collection mechanisms. The most obvious pressures are toward increased use of outcome data. However, for many specialities, especially those dealing with chronic conditions or public health, it is likely that reporting schemes will have to rely on measures of process, preferably those that are known from research evidence or professional opinion to be strongly linked to good eventual outcomes.

Broadening the scope of data. The scope of reporting schemes is of course circumscribed by the data collection capacity of participating organizations, and the development of electronic health records is likely to offer a good opportunity to broaden the scope of information reported. The unitary U.K. system offers particularly fertile ground for seizing new information opportunities. For example, there will be a dramatic increase in information relating to primary care arising from new general practitioner (GP) remuneration systems, information that could readily be used for reporting purposes.⁴⁴

Adequate risk adjustment. Almost all performance reporting requires some sort of risk adjustment to secure meaningful comparability, particularly when data on outcomes are reported and when the information is used to make definitive judgments. The credibility of a reporting scheme may be fatally compromised if risk adjustment is inadequate. In addition, inadequate risk adjustment gives physicians the incentive to discriminate against higher-risk patients (cream skimming). The science of risk adjustment is developing rapidly, but many issues remain unresolved. Further development of methodology is a clear research priority, and as new domains of performance reporting are introduced, associated risk adjustment methods need to be put in place.

Organizational focus levels. In designing a reporting scheme, a fundamental question is at what organizational level to report results. The New York cardiac surgery scheme (and its proposed U.K. counterpart) focus on the individual physician, and there are clear benefits to the user in being able to scrutinize a specific provider of care. However, cardiac surgery may be a special case. For many other procedures and specialities, it is likely that outcome measures and risk adjustment methodologies are less well developed and that volumes of activity are lower. In these circumstances, focusing publicly on individual physicians may provoke negative responses from physicians. It is therefore likely that, for many circumstances, some broader organizational level will be most appropriate, such as hospitals, larger group practices, health plans, and purchasers. Of course, scrutiny of individual physicians and teams should continue to be an important internal task of health care organizations.

Increasing the public’s interest. A consistent finding in both countries is the lack of public interest in quality reports. This may change in the future, as the public becomes better informed and more assertive (and, in the United Kingdom, enjoys greater choice). Increased interest may be particularly marked among patients with chronic conditions, supported by advocacy groups. However, the main scope for increased engagement in both countries seems to be among provider and purchaser organizations. The former need more help and support than they now receive if they are to respond positively to public reports: The benefits are more likely to be maximized and the adverse consequences minimized in an environment that values learning and improvement. Purchaser organizations also need encouragement, since, with some notable exceptions, they seem to be remarkably passive in scrutinizing the quality of care secured for their patients.

Using incentives. This raises the important issue of what incentives to attach to public reports. This is an underdeveloped area, but both countries are evaluating schemes to reward quality, and we await the results with interest. It is also important not to neglect the intrinsic incentives associated with being seen as doing a good job. But we feel that explicit incentives need to be built into the system if the public reports are to achieve maximum effectiveness. In the same vein, many of the alleged adverse side effects of public reporting arise from accidental incentives that may need to be countered. For example, if cream skimming is found to take place, then it may be necessary to adjust both payment and risk adjustment schemes. It will also be important to nurture a professional climate in which gaming and misreporting are considered unacceptable. Given the frequent professional hostility to public reporting, it is essential to monitor all schemes for unintended consequences to allay fears or take remedial action as appropriate.

The trend toward increased disclosure of quality information is in our view irresistible, and, therefore, determining what style of reporting works best and in what circumstances is a major policy task. To that end, the emerging experience from the two countries offers a rich source of contrasting evidence for policymakers to consider.

Editor's Notes

 
An earlier version of this paper was presented at the international symposium, Improving Quality of Health Care in the United States and the United Kingdom: Strategies for Change and Action, organized by the Commonwealth Fund and the Nuffield Trust and held at Pennyhill Park Conference Centre, Bagshot, England, 12–14 July 2002.

NOTES

Top Quality Reporting In The... Quality Reporting In The... A Comparison Of U.S.... Evidence On The Impact... Maximizing The Benefits Of... NOTES

 

1. P. Smith, ed., Measuring Up: Improving Health System Performance in OECD Countries (Paris: Organization for Economic Cooperation and Development, 2002).
2. M.N. Marshall et al., "Public Disclosure of Performance Data: Learning from the U.S. Experience," Quality in Health Care 9, no. 1 (2000): 53–57.[Free Full Text]
3. F. Nightingale, Notes on Hospitals, 3d ed. (London: Longman, Green, Longman, Roberts, and Green, 1863); and E.A. Codman, A Study in Hospital Efficiency (1917; reprint, Oakbrook Terrace, Ill.: Joint Commission on Accreditation of Healthcare Organizations, 1996).
4. M.N. Marshall et al., The Public Disclosure of Performance Data in Health Care: Learning from the U.S. Experience (London: Nuffield Trust, 2000); and M.N. Marshall and H.T.O. Davies, "Public Release of Information on Quality of Care: How Are Health Services and the Public Expected to Respond?" Journal of Health Services Research and Policy 6, no. 3 (2001): 158–162.
5. A. O’Neil, "A Question of Trust," BBC Radio 4 Reith Lectures (2002); and H.T.O. Davies, "Falling Public Trust in Health Services: Implications for Accountability," Journal of Health Services Research and Policy 4, no. 4 (1999): 193–194.
6. A. Epstein, "Sounding Board: Performance Reports on Quality—Prototypes, Problems, and Prospects," New England Journal of Medicine 333, no. 1 (1995): 57–61.[Free Full Text]
7. New York State Department of Health, "Info for Researchers, Heart Disease," October 2000, www.health.state.ny.us/nysdoh/research/heart/heart.htm (19 January 2003); and M.R. Chassin, "Achieving and Sustaining Improved Quality: Lessons from New York State and Cardiac Surgery," Health Affairs (Jul/Aug 2002): 40–51.
8. Centers for Medicare and Medicaid Services, "Nursing Home Quality Initiative," 14 February 2003, www.cms.hhs.gov/providers/nursinghomes/nhi (21 February 2003).
9. M. Spranca et al., "Do Consumer Reports of Health Plan Quality Affect Health Plan Selection?" Health Services Research 35, no. 5 (2000): 933–947[Medline]; and E. Guadagnoli, "Providing Consumers with Information about the Quality of Health Plans: The Consumer Assessment of Health Plans (CAPHS) Demonstration in Washington State," Joint Commission Journal on Quality Improvement 26, no. 7 (2000): 410–420.
10. A. Street, "The Resurrection of Hospital Mortality Statistics in England," Journal of Health Services Research and Policy 7, no. 2 (2002): 104–110.
11. R. Mannion and M. Goddard, The Impact of Performance Measurement in the NHS—Report 1: Empirical Analysis of the Impact of Public Dissemination of the Scottish Clinical Resource and Audit Group Data (York: Centre for Health Economics, University of York, 2000).
12. Department of Health, AFirstClassService,QualityintheNHS (London: Department of Health, 1998); and Department of Health, The NHS Performance Assessment Framework (London: Department of Health, 1999).
13. Department of Health, The NHS Plan: A Plan for Investment, a Plan for Reform (London: Department of Health, 2001).
14. Department of Health, "The National Survey of NHS Patients, Coronary Heart Disease Survey, Trust Level Data," 19 February 2003, www.doh.gov.uk/nhspatients/chdsurvey2a.htm (21 February 2003).
15. Department of Health, Learning from Bristol: The Department of Health’s Response to the Report of the Public Inquiry into Children’s Heart Surgery at the Bristol Royal Infirmary 1984–1995 (London: Department of Health, 2002). For details on the society’s activities, see www.scts.org.
16. For a detailed description of the U.S. and U.K. reporting systems, see V. Raleigh, Performance Assessment in the NHS (London: Commission for Health Improvement, 2002); R. Mannion and M. Goddard, The Impact of Performance Measurement in the NHS—Report 3: Performance Measurement Systems; A Cross-Sectoral Study (York: Centre for Health Economics, University of York, 2000); M.N. Marshall et al., Dying to Know: Public Release of Comparative Data in Health Care (London: Nuffield Trust, 2000); and E.C. Schneider and T. Lieberman, "Publicly Disclosed Information about the Quality of Health Care: Response of the U.S. Public," Quality in Health Care 10, no. 2 (2001): 96–103.[Abstract/Free Full Text]
17. D. Lansky, "Improving Quality through Public Disclosure of Performance Information," Health Affairs (Jul/Aug 2002): 52–62.
18. Schneider and Lieberman, "Publicly Disclosed Information."
19. L. Fine et al., "How to Evaluate and Improve the Quality and Credibility of an Outcomes Database: Validation and Feedback Study on the U.K. Cardiac Surgery Experience," British Medical Journal 326, no. 7379 (2003): 25–28.[Abstract/Free Full Text]
20. M.E. Vaiana and E.A. McGlynn, "What Cognitive Science Tells Us about the Design of Reports for Consumers," Medical Care Research and Review 59, no. 1 (2002): 3–35.[Abstract/Free Full Text]
21. Schneider and Lieberman, "Publicly Disclosed Information."
22. National Health Care Purchasing Institute, "Rewarding Results: About the Program," www.nhcpi.net/rewardingresults/nac.cfm (7 March 2003).
23. For a review of the literature, see M.N. Marshall et al., "The Public Release of Performance Data: What Do We Expect to Gain? A Review of the Evidence," Journal of the American Medical Association 283, no. 14 (2000): 1866–1874.[Abstract/Free Full Text]
24. E.L. Hannan et al., "Improving the Outcomes of Coronary Artery Bypass Surgery in New York State," Journal of the American Medical Association 271, no. 10 (1994): 761–766[Abstract]; and E.D. Peterson et al., "The Effects of New York’s Bypass Surgery Provider Profiling on Access to Care and Patient Outcomes in the Elderly," Journal of the American College of Cardiology 32, no. 4 (1998): 993–999.[Abstract/Free Full Text]
25. G.E. Rosenthal et al., "Using Hospital Performance Data in Quality Improvement: The Cleveland Health Quality Choice Experience," Joint Commission Journal on Quality Improvement 24, no. 7 (1998): 347–360; and G.E. Rosenthal, L. Quinn, and D.L. Harper, "Declines in Hospital Mortality Associated with a Regional Initiative to Measure Hospital Performance," American Journal of Medical Quality 12, no. 2 (1997): 103–112.
26. D.R. Longo et al., "Consumer Reports in Health Care: Do They Make a Difference in Patient Care?" Journal of the American Medical Association 278, no. 19 (1997): 1579–1584[Abstract]; and J. Bost, "Managed Care Organizations Publicly Reporting Three Years of HEDIS Data," Managed Care Interface 14 (2001): 50–54.
27. S. Edgman-Levitan and P.D. Cleary, "What Information Do Consumers Want and Need?" Health Affairs (Winter 1996): 42–56; and S. Robinson and M. Brodie, "Understanding the Quality Challenge for Health Consumers: The Kaiser/AHCPR Survey," Joint Commission Journal on Quality Improvement 23, no. 5 (1997): 239–244.
28. Schneider and Lieberman, "Publicly Disclosed Information"; Mannion and Goddard, The Impact of Performance Measurement in the NHS—Report 1; and M.N. Marshall, J. Hiscock, and B. Sibbald, "Attitudes to the Public Release of Comparative Information on the Quality of General Practice Care: A Qualitative Study," British Medical Journal 325, no. 7375 (2002): 1278.[Abstract/Free Full Text]
29. D. Scanlon et al., "The Impact of Health Report Cards on Managed Care Enrollment," Journal of Health Economics 21, no. 1 (2002): 19–41[Medline]; and N.D. Beaulieu, "Quality Information and Consumer Health Plan Choices," Journal of Health Economics 21, no. 1 (2002): 43–63.[Medline]
30. Marshall et al., "Attitudes to the Public Release of Comparative Information."
31. E.C. Schneider and A.M. Epstein, "Use of Public Performance Reports," Journal of the American Medical Association 279, no. 20 (1998): 1638–1642.[Abstract/Free Full Text]
32. D.B. Mukamel and A.I. Mushlin, "Quality Care Information Makes a Difference: An Analysis of Market Share and Price Changes after Publication of the New York State Cardiac Surgery Mortality Reports," Medical Care 36, no. 7 (1998): 945–954.[Medline]
33. Hannan et al., "Improving the Outcomes"; Schneider and Epstein, "Use of Public Performance Reports"; S.T. Mennemeyer, M.A. Morrisey, and L.Z. Howard, "Death and Reputation: How Consumers Acted upon HCFA Mortality Information," Inquiry (Summer 1997): 117–128; J.R. Gabel et al., When Employers Choose Health Plans: Do NCQA Accreditation and HEDIS Data Count? (New York: Commonwealth Fund, 1998); and P.S. Romano, J.A. Rainwater, and D. Antonius, "Grading the Graders: How Hospitals in California and New York Perceive and Interpret Their Report Cards," Medical Care 37, no. 3 (1999): 295–305.[Medline]
34. H.H. Schauffler and J.K. Mordavsky, "Consumer Reports in Health Care: Do They Make a Difference?" Annual Review of Public Health 22 (2001): 69–89[Medline]; and K. Harris, J. Schultz, and R. Feldman, "Measuring Consumer Perceptions of Quality Differences among Competing Health Benefit Plans," Journal of Health Economics 21, no. 1 (2002): 1–17.[Medline]
35. Schneider and Lieberman, "Publicly Disclosed Information."
36. Gabel et al., When Employers Choose Health Plans; and J.H. Hibbard et al., "Choosing a Health Plan: Do Large Employers Use the Data?" Health Affairs (Nov/Dec 1997): 172–180.
37. Mannion et al., The Impact of Performance Measurement in the NHS—Report 1; E.L. Hannan et al., "Public Release of Cardiac Surgery Outcomes Data in New York: What Do New York State Cardiologists Think of It?" American Heart Journal 134, no. 1 (1997): 55–61[Medline]; and E.C. Schneider and A.M. Epstein, "Influence of Cardiac-Surgery Performance Reports on Referral Practices and Access to Care: A Survey of Cardiovascular Specialists," New England Journal of Medicine 335, no. 4 (1996): 251–256.[Abstract/Free Full Text]
38. J. Kaplan et al., "Managed Care Report Cards: Evaluating Those Who Evaluate Physicians," Managed Care Interface 13 (2000): 88–94.
39. Mannion et al., The Impact of Performance Measurement in the NHS—Report 1; Rosenthal et al., "Using Hospital Performance Data in Quality Improvement"; Longo et al., "Consumer Reports in Health Care"; J.M. Bentley and D.B. Nash, "How Pennsylvania Hospitals Have Responded to Publicly Released Reports on Coronary Artery Bypass Graft Surgery," Joint Commission Journal on Quality Improvement 24, no. 1 (1998): 40–49; S.W. Dziuban et al., "How a New York Cardiac Surgery Program Uses Outcomes Data," Annals of Thoracic Surgery 58, no. 6 (1994): 1871–1876[Abstract]; D.M. Berwick and D.L. Wald, "Hospital Leaders’ Opinions of the HCFA Mortality Data," Journal of the American Medical Association 263, no. 2 (1990): 247–249[Abstract]; and J.A. Rainwater, P.S. Romano, and D.M. Antonius, "The California Hospital Outcomes Project: How Useful Is California’s Report Card for Quality Improvement?" Joint Commission Journal on Quality Improvement 24, no. 1 (1998): 31–39.
40. H.T.O. Davies, "Public Release of Performance Data and Quality Improvement: Internal Responses to External Data by U.S. Health Care Providers," Quality in Health Care 10, no. 2 (2001): 104–110.[Abstract/Free Full Text]
41. M.R. Chassin, E.L. Hannan, and B.A. DeBuono, "Benefits and Hazards of Reporting Medical Outcomes Publicly," New England Journal of Medicine 334, no. 6 (1996): 394–398. Similar results were found in an unpublished audit of the British media’s coverage of hospital performance reports conducted by Simon Rinaldi and Martin Marshall in 2002.[Free Full Text]
42. P. Smith, "On the Unintended Consequences of Publishing Performance Data in the Public Sector," International Journal of Public Administration 18, nos. 2 and 3 (1995): 277–310; and D. Dranove et al., Is More Information Better? The Effects of Report Cards on Health Care Providers (Cambridge, Mass.: National Bureau of Economic Research, 2002).
43. D. McCormick et al., "Relationship between Low Quality-of-Care Scores and HMOs’ Subsequent Public Disclosure of Quality-of-Care Scores," Journal of the American Medical Association 288, no. 12 (2002): 1484–1490.[Abstract/Free Full Text]
44. M.N. Marshall and M.O. Roland, "The New Contract: Renaissance or Requiem for General Practice?" British Journal of General Practice 52, no. 480 (2002): 531–532.

                      What's this?

This article has been cited by other articles:

				P. Sousa, A. S. Uva, F. Pinto, and on behalf of the investigators of PCI registry of Risk-adjustment model in health outcomes evaluation: a contribution to strengthen assessment towards quality improvement in interventional cardiology Int. J. Qual. Health Care, July 11, 2008; (2008) mzn029v1. [Abstract] [Full Text] [PDF]

				A. Dixon, J. Greene, and J. Hibbard Do Consumer-Directed Health Plans Drive Change In Enrollees' Health Care Behavior? Health Aff., July 1, 2008; 27(4): 1120 - 1131. [Abstract] [Full Text] [PDF]

				C. H. Fung, Y.-W. Lim, S. Mattke, C. Damberg, and P. G. Shekelle Systematic Review: The Evidence That Publishing Patient Care Performance Data Improves Quality of Care Ann Intern Med, January 15, 2008; 148(2): 111 - 123. [Abstract] [Full Text] [PDF]

				A. T. Chien, M. H. Chin, A. M. Davis, and L. P. Casalino Pay for Performance, Public Reporting, and Racial Disparities in Health Care: How Are Programs Being Designed? Med Care Res Rev, October 1, 2007; 64(5_suppl): 283S - 304S. [Abstract] [PDF]

				E. Moy, E. Dayton, and C. M. Clancy Compiling The Evidence: The National Healthcare Disparities Reports Health Aff., March 1, 2005; 24(2): 376 - 387. [Abstract] [Full Text] [PDF]

				M Marshall and P Smith Rewarding results: using financial incentives to improve quality Qual. Saf. Health Care, December 1, 2003; 12(6): 397 - 398. [Full Text] [PDF]