|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Achieving And Sustaining Improved Quality: Lessons From New York State And Cardiac Surgery
Since 1989 the New York State Department of Health has published annual data on risk-adjusted mortality following coronary artery bypass graft surgery by hospital and surgeon. It was the first such program in the nation and is now the most long-lived. Many hospitals were prompted by the data to improve their cardiac surgery programs, and statewide mortality fell substantially as a result. This paper examines what physicians and hospitals did in response to the data, how the market reacted, and whether this approach to quality measurement and improvement could be used more widely.
Quality has taken center stage away from cost and access in the U.S. public debate about health care in the past several years. Concern about errors; the managed care backlash; and the proliferation of information purporting to judge the performance of health plans, hospitals, and doctors have all contributed to a vigorous discussion about what quality is, what problems exist, and how they might be remedied.1 Many proposals and programs have been suggested or initiated, including public reporting of error rates, employers supplying information about quality to their employees, and health plans providing enrollees with data on the performance of hospitals and medical groups.2 Nevertheless, our track record in improving quality is poor.3 Very few large-scale programs have established quality of care among their highest priorities, achieved major improvements, or sustained those improvements over time. The New York State Cardiac Surgery Reporting System (CSRS) is one of the few that have. It was the first statewide program to produce public data on outcomes for cardiac surgery. It is also the nations longest-running program of this kind. This paper summarizes the CSRS experience, focusing on how physicians and hospitals responded to the program, what they did to improve, and what impacts the program has had. It concludes with a discussion of the programs strengths and limitations and a consideration of whether the CSRS is a model for quality improvement that could be emulated more broadly.
Regulating capacity. The context in which New Yorks CSRS was implemented has important implications for understanding its success and the potential limits of its generalizability. New York is one of few states that has maintained an approach to regulating the capacity of the health care delivery system that was developed in the 1970s: the certificate-of-need program. As it applies to cardiac surgery, New York has a regulatory procedure by which hospitals may apply to establish new programs to offer this service or to expand existing programs. The regulations define the qualifications required for approval in terms of volume, access to high-quality supporting services, and lack of detrimental impact on other institutions offering similar services. The explicit aim of these regulatory activities is to establish and maintain high-volume, high-quality programs distributed to meet the needs of the states population. The effect of these regulations and their implementation has been to limit the number of hospitals that perform cardiac surgery.4 As a consequence, a large and growing proportion of New York hospitals performing cardiac surgery do so at very high volumes, particularly compared with other large states that have not attempted to regulate this aspect of health care. Kevin Grumbach and colleagues reported that in 19871989, 32 percent of hospitals in New York State had annual volumes of coronary artery bypass graft (CABG) surgery of 500 cases or more, compared with only 8 percent of California hospitals.5 Only 10 percent of New York hospitals had annual volumes of fewer than 100 cases, compared with 31 percent of California hospitals. The death rate at the very low volume California hospitals was double that of their high-volume counterparts. In New York at that time, 59 percent of patients underwent their CABG surgery at the highest-volume hospitals, compared with 26 percent of patients in California. A decade later that gap had widened considerably. By 1998, 73 percent of patients in New York obtained their CABG surgery at these high-volume hospitals, compared with 27 percent in California. Patients who underwent surgery at very low volume hospitals accounted for 0.3 percent of all CABG surgery patients (n = 55) in New York and 5.5 percent (n = 1,510) in California.6 Measuring outcomes. Although it is not embodied in any regulation itself, the program to measure outcomes developed in the context of this regulatory environment. For more than fifty years a distinguished committee of physicians and lay people has advised the New York State Department of Health about issues related to heart disease. This committee, which also plays an advisory role in the certificate-of-need process, initiated the effort to measure risk-adjusted outcomes following CABG surgery in 1989, to understand the significance of differences in unadjusted death rates among hospitals. The CSRS has operated continuously ever since. The program, described in detail elsewhere, produces annual data for public dissemination on risk-adjusted death rates following CABG surgery, by hospital and by surgeon.7
In the early years of the program, media attention was intense.8 Hospitals were identified in public reports as outliers if their risk-adjusted mortality rate was significantly greater than the average for the state. These institutions attracted the attention of both the press and the health department. Previous reports in the published literature have not described how most of these hospitals went about the tasks of improvement. I obtained this information at Winthrop Hospital, Erie County Medical Center, Strong Memorial Hospital, and Bellevue Hospital Center in the spring of 2001 during interviews with key physicians, hospital administrators, and state officials who were directly involved in these efforts. Winthrop University Hospital. Winthrop University Hospital had one of the highest risk-adjusted mortality rates in the first public report; following an outside review, the health department put its cardiac surgery program on probation. The hospital recruited its first full-time cardiac surgery chief from YaleNew Haven Hospital, who completely revamped the service.9 He concentrated it on a single floor of the hospital and added new clinical nurse specialists and physician assistants who were dedicated to cardiac surgery. To understand exactly which systems needed the most attention, he reviewed all of the deaths from the previous years of poor performance. For the first year of his tenure, which began in March 1990, he personally reviewed each case with the patients surgeon before the operation. He also lobbied successfully to install a dedicated cardiac anesthesia service. Risk-adjusted mortality fell from 9.2 percent in 1989 to 4.6 percent in 1990 and then to 2.3 percent in 1991. In the most recent report Winthrop had the lowest risk-adjusted mortality of any hospital in the state (0.82 percent for 1998) and was again a statistical outlier, this time on the side of excellent performance. Erie County Medical Center. Erie County Medical Center, a publicly owned hospital in Buffalo, was also in this situation. Indeed, its high mortality and low volume had attracted the attention of the state health department before the first risk-adjusted mortality data were available. The first six months (JanuaryJune 1989) of risk-adjusted mortality data revealed that the hospital had the worst performance in the state (17.6 percent), significantly higher than the state average (4.87 percent). A site visit by the advisory committee in December 1989 recommended a series of changes, and the hospital voluntarily suspended operations in January 1990 to reorganize. The changes included establishment of a quality assurance program specific to cardiac surgery; credentialing and ongoing evaluation of surgeons performance; training of dedicated cardiac anesthesiologists; agreement to create designated cardiac surgery intensive care beds; and agreement to recruit a permanent, full-time service chief. Sufficient progress had been made by April 1990 for the health department to agree to permit Erie County to resume operations under probation until all elements of the recommended changes were implemented. From 1989 to 1991 the hospitals risk-adjusted mortality was 7.31 percent, significantly greater than the state average of 3.23 percent, and it had a low annual volume of just over 100 cases. Close state oversight continued until a new, full-time chief was hired. He arrived at the beginning of 1993 and hired operating room nurses, cardiopulmonary bypass technicians, and cardiac surgery intensive care staff, all of whom were dedicated to cardiothoracic surgery. The previous surgeons stopped doing cardiac surgery. The new chief introduced the operating microscope to cardiac surgery and instituted weekly teaching conferences for the entire cardiac surgical team as well as weekly conferences with cardiologists. In 19931995 risk-adjusted mortality fell to 2.51 percent, just below the state average (2.57 percent), and in 19961998 it fell further to 1.77 percent, well below the state average (2.27 percent). Volume increased concomitantly, to 219 cases per year (19961998).10 Early statewide findings. One of the earliest findings from the statewide analysis was that low-volume surgeons (those who performed fifty or fewer procedures per year) had higher risk-adjusted mortality than high-volume surgeons had. A number of hospitals took action to restrict the surgical privileges of low-volume surgeons, especially those who had high mortality rates. Some left the state, and some continued to perform noncardiac surgery in New York. From 1989 to 1992 a total of twenty-seven such surgeons ceased performing this surgery in New York. As a group, in the last year in which they performed CABG surgery in the state, these twenty-seven surgeons experienced a risk-adjusted mortality rate of 11.9 percent, nearly four times the state average of 3.1 percent.11
St. Peters Hospital.
Other hospitals encountered different problems. In 1991 and 1992 St. Peters Hospital, a large private hospital in Albany, was a statistical outlier with poor outcomes. Data revealed that the problem was limited to emergency cases, for which its mortality rate was 26 percent (state average, 7 percent). Elective and urgent cases, which made up 91 percent of all cases, had a mortality rate about the same as the state average. The hospital conducted a multidisciplinary review of how it managed the emergency cases, many of which involved transferred patients with complicated acute myocardial infarction. The review concluded that physicians were taking insufficient time to stabilize these patients before going to surgery. The review led to major changes in the management of these patients, and mortality dropped from eleven of forty-two emergency cases in 1992 to zero of fifty-four cases in 1993. Stanley Dziuban and colleagues described this experience in the only paper published to date from a hospital recounting its use of CSRS data to improve.12 As Exhibit 1
Strong Memorial Hospital. Strong Memorial Hospital in Rochester faced yet a different set of problems. During 19891991 its two highest-volume surgeons had very different mortality experiences (Exhibit 2
Bellevue Hospital. More recently, another public hospital confronted its performance history. Bellevue Hospital is the only one of eleven publicly owned hospitals in New York City authorized by the state health department to offer cardiac surgery services. Bellevues annual volume has been among the lowest in the state (sixty to ninety cases per year). Its mortality rate emerged as significantly higher than the state average in the mid-1990s. In 19931995 its risk-adjusted mortality rate was 5.63 percent (state average, 2.57 percent), and in 19961998 its risk-adjusted mortality rate was 5.14 percent (state average, 2.27 percent).13 In early 2000, following an outside review requested by the state health department, Bellevue agreed to voluntarily suspend this service until it could respond to the suggestions made for improvement. The service was extensively redesigned with the objective of creating a smoothly functioning, multidisciplinary team. Nurse practitioners and physician assistants dedicated to the care of cardiovascular surgery patients were hired, as was a new team of perfusionists. Nurses were retrained. Previously, several cardiac surgeons (typically five or six) from neighboring New York University Hospital had operated at Bellevue. The redesign limited this number to two, and Bellevue hired its first full-time cardiac surgeon. The service reopened in the spring of 2001, following a site visit by outside experts at the request of the health department. Whether Bellevues future performance will improve as Erie Countys did under somewhat similar circumstances remains to be seen.
Impact. Over its fourteen years the New York CABG surgery mortality reporting and quality improvement program has had a positive impact. Between 1989 and 1992 risk-adjusted mortality fell 41 percent statewide in New York.14 Eric Peterson and colleagues analyzed Medicare data from 1987 to 1992 and calculated risk-adjusted mortality rates following CABG surgery for every state (Exhibit 3
Impact on market share. The improvements in New York happened because individual hospitals and cardiac surgery programs used the data to make specific changes in the way they provided care to CABG patients. Market forces played no role. Managed care companies did not use the data in any way to reward better-performing hospitals or to drive patients toward them.18 Nor did patients avoid high-mortality hospitals or seek out those with low mortality. Exhibit 4
Critiques. The CSRS has been criticized for encouraging hospitals and physicians to exaggerate the presence of serious risk factors, to refuse to operate on high-risk cases, or to refer high-risk cases out of state.19 Also, one study claimed that in the absence of any public reporting or organized quality improvement efforts, CABG mortality rates in Massachusetts fell by a comparable amount.20 Risk-factor coding. The issue of risk-factor coding has been discussed in detail elsewhere.21 Briefly, the criticism was based in large part on the observation that several risk factors increased in prevalence over the first few years of the program. The large majority of risk factors (nine of fourteen), however, showed no overall increase in prevalence. A few did, driven largely by changes made by the advisory committee to more clearly specify their definitions. Further, the health department conducts an annual audit in which half of the hospitals in the state have samples of the risk-factor data they submitted compared with data abstracted from medical records. Hospitals with large, unexplained discrepancies recode their data. Of course, the acid test for whether the risk-factor data are accurate or not is how well the logistic regression model that is used for risk adjustment works to predict mortality. On this score, the model has proved extremely robust over the years, with C-statistics averaging just over 0.8 and tests of calibration demonstrating accuracy of prediction over all levels of predicted mortality.22 Out-of-state referral and high-risk cases. The analysis of Medicare data conducted by Peterson and colleagues also addressed the questions of outmigration, high-risk patients, and improvement in other states. They found that the proportion of New York State residents who underwent CABG surgery outside New York fell from 14.3 percent in 1989 to 11.3 percent in 1992. Elderly patients with acute myocardial infarction are among the highest-risk patients for CABG surgery. The proportion of such patients ages 6570 who received CABG surgery in New York increased from 6.0 percent in 1989 to 8.4 percent in 1992, with similar results for those ages 7580.23
Massachusetts versus New York.
The study from Massachusetts used very different methods than those used in New York to measure improvements in mortality. The principal differences were in identifying cases (diagnosis-related groups versus all isolated CABG operations), risk-adjustment methods (administrative versus clinical data), and measuring the cumulative decline in mortality over five years (19901994). That cumulative decline was compared to the previously published decline over four years (19891992) in New York. As Exhibit 3 Limitations. Although its impact has been positive, New Yorks CSRS is encumbered by important limitations that have blocked even greater improvement. First, as the individual hospital experiences described above indicate, each hospital was required to undertake considerable additional work to determine exactly what was required to improve. When outcomes data are used as measures of quality, valid as they may be, they do not by themselves point to processes that must be fixed. Further, as these narratives also show, each hospital faced unique circumstances as it strove to improve. In each case, complex systems were broken in different ways, so different remedial actions were called for and undertaken. Second, the impetus to use the data to improve has been limited almost entirely to hospitals that have been named as outliers with poor performance. Programs that have registered mediocre or below-average performance have not been so motivated. More than a decade into the program, some residual antagonism exists among practicing cardiac surgeons and cardiologists.24 It is perhaps for this reason that hospitals not faced with the opprobrium attached to being named as poorly performing outliers have largely failed to use the rich performance data to find ways to lift themselves from mediocrity to excellence. Finally, the program has not proved to be generalizable either within New York or outside it. New York has produced data on measures of quality only for hospitals performing percutaneous transluminal coronary angioplasty (PTCA) and for centers providing care for patients with HIV.25 Only Pennsylvania and New Jersey have emulated New Yorks cardiac surgery mortality reporting system, although five hospitals in Maine, New Hampshire, and Vermont conducted private quality improvement activities to reduce mortality, with positive results.26 In 2001 California produced a report on risk-adjusted mortality following CABG surgery by hospital for the two-thirds of California hospitals that participated in this voluntary effort.27 A new California state law now requires all hospitals to participate in this data collection and reporting process.28
The New York program has proved its durability, having lasted through the terms of four commissioners of health in governments administered by two governors of different political parties. I believe that its endurance is attributable to three factors: its integration into the routine processes of a governmental agency, the continuous commitment of those closest to the program to publishing in scientific journals a variety of analyses of its impact, and the vigorous involvement of the states leading cardiac surgeons and cardiologists in the advisory committee process. Also, it was born in a state that heavily regulates its health care delivery system. The broad regulatory power of the health commissioner in 1989, more than any other factor, explains why no hospital refused when he asked them to provide the clinical data on risk factors, without any compensation for the cost of the activity. The active engagement of the health department continues to be a primary force for improvement.29 Press coverage, so copious at the outset of the program, is waning. A Lexis-Nexis search in April 2002 revealed no newspaper reports on the most recent public release of CSRS data in January 2001. The obstacles to quality improvement are many and potent. Several of these are systemic and have been discussed elsewhere, including lack of public demand for marked improvement, perverse payment incentives, inadequate education and training of physicians and other health professionals, lack of effective use of information technology, and lack of leadership.30 In the face of these powerful forces, quality improvement efforts will have to muster strong countervailing pressures. In New York the particular combination of circumstances described here has been up to the task. It is unlikely that another state could exactly duplicate these circumstances. The lessons from the New York experience will have to be adapted to succeed in the different cultures and markets of other states or jurisdictions. I believe that the critical elements of a successful program are (1) required reporting from all hospitals; (2) regular audits to verify data quality; (3) analysis and public reporting by a neutral, respected third party; (4) close oversight by an advisory group of recognized clinical leaders; (5) a commitment to studying and publishing reports on the impact of the system; and (6) continuous pressure on poor and mediocre performers to improve. Is public data disclosure really necessary? I believe that the answer is yes, most of the time. The northern New England program was successful without this feature. But that effort involved close collaboration among only five hospitals, widely separated in three states, that do not directly compete with each other. In most other large regions, with multiple competitive relationships among hospitals, such collaboration would not be likely to occur, even if its costs could be defrayed. Quite apart from the valid argument that the public has a right to have this information, making the data public proved to be a compelling force for improvement in New York. It galvanized physicians by providing them for the first time with clinically sound, comparative data on performance. And it drove hospital administrators and governing bodies to provide the additional resources required to improve their cardiac surgical services. Several different kinds of organizations can put these pieces together in different ways in different places to create effective quality improvement programs. The time to proceed is now.
Mark Chassin is professor and chairman of the Department of Health Policy at the Mount Sinai School of Medicine in New York City. He is also senior vice-president for clinical quality at the Mount Sinai/NYU Medical Center and Health System. An earlier version of this paper was presented at the Commonwealth Fund/Nuffield Trust conference, "Improving Quality of Health Care in the United States and United Kingdom: Strategies for Change and Action, 2001," in Bagshot, England, 2224 June 2001. I am grateful to the many persons who provided invaluable insights into how the improvements described here were achieved. Any errors are solely my responsibility.
This article has been cited by other articles:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||