|
Measuring Performance For Treating Heart Attacks And Heart Failure: The Case For Outcomes Measurement
To complement the current process measures for treating patients with heart attacks and with heart failure, which target gaps in quality but do not capture patient outcomes, the Centers for Medicare and Medicaid Services (CMS) has proposed the public reporting of hospital-level thirty-day mortality for these conditions in 2007. We present the case for including measurements of outcomes in the assessment of hospital performance, focusing on the care of patients with heart attacks and with heart failure. Recent developments in the methodology and standards for outcomes measurement have laid the groundwork for incorporating outcomes into performance monitoring efforts for these conditions.
MANY ORGANIZATIONS HAVE TARGETED the care of patients with heart failure and with heart attacks in efforts to improve health care quality.1 The importance of these cardiovascular conditions and the existence of strong evidence about clinical care led the Centers for Medicare and Medicaid Services (CMS) and the Joint Commission on Accreditation of Health-care Organizations (JCAHO) to develop and publicly report cardiovascular performance measures.2 These "core" measures focus on the frequency with which hospitals provide key therapies and interventions to patients who would benefit from them. The emphasis on these process measures has been a major advance in medicine and resulted in favorable practice changes.3 The core measures are endorsed by the National Quality Forum, American College of Cardiology, and American Heart Association and are the focus of many quality improvement activities.4 Recent studies indicate that performance on these processes has improved for patients with heart attacks and heart failure over the past decade.5 The success of process measurement in improving compliance with recommended guidelines for care represents a major achievement in focusing hospitals and clinicians on improving the translation of science into practice within cardiovascular care. Nevertheless, success in improving performance as measured by the seven heart attack process measures and four heart failure measures is accompanied by interest in expanding the spectrum of measures of hospital performance. Although process measures are an excellent approach to targeting gaps in quality for specific actions that are strongly supported in the medical literature, they do not capture all of the actions in hospitals that could influence outcomes. Information about outcomes can complement information provided by process measures. In this paper we make the case for including measurements of outcomes in the assessment of hospitals’ performance in caring for patients with heart attacks and with heart failure. The intent is not to replace process measures but to convey the complementary value of outcomes measurement to process measures, with particular emphasis on standards for measuring and reporting outcomes.
There are several reasons to complement process measures with outcomes measures to assess the hospital care for heart attacks and heart failure. Patients care about results. The most important argument for adding outcomes measurement is that patients likely care more about the results of their care than how those outcomes are achieved. Patients are interested in surviving, avoiding hospital admissions, minimizing symptoms, regaining optimal functioning, and rapidly recovering their quality of life. No set of process measures can be comprehensive enough to serve as a surrogate for outcomes. Process measures might contribute to but are not surrogates for outcomes. Although useful in targeting specific interventions and addressing gaps in care, cardiovascular process measures are narrowly focused and omit many processes that are important in clinical care. These other factors include staffing patterns, interdisciplinary communication, emergency response teams, supportive infrastructure, the transition of care from the inpatient to the outpatient setting, and effective health information technology (IT), all of which could have important effects on a patient’s clinical course. Not surprisingly, the link between hospital performance on the current set of process measures and short-term outcomes is modest. Scott Williams and colleagues found that the substantial improvement in hospitals’ performance in the heart attack core measures during recent years was not accompanied by a concomitant decrease in hospital mortality rates.6 Elizabeth Bradley and colleagues demonstrated that hospitals’ performance on heart attack core measures explains only about 6 percent of the variation among hospitals in risk-adjusted thirty-day mortality rates. Moreover, they reported that only about 30 percent of the hospitals in the top quintile of risk-adjusted thirty-day mortality rates for heart attacks are also in the top quintile of performance on publicly reported core process measures for this condition.7 No set of process measures could be comprehensive enough to fully serve in the place of direct outcomes measurement. This assertion does not undermine the value of individual process measures, which allow for key actions to be assessed and improved, but it supports the value of adding outcomes measures to the evaluation of hospital performance. Process measures commonly assess care in a subset of patients with the clinical condition. Although outcomes measures tend to include all patients with a given condition, process measures typically include only a subset of patients who are considered "ideal" for a specific intervention (that is, they have an indication and no documented contraindication). For example, when the use of beta-blockers for patients with heart attacks is being assessed, those with a heart conduction disorder (a contraindication to beta-blockers) are excluded from the calculation. As a result, more than one-fourth of all Medicare beneficiaries hospitalized with heart attacks might not be eligible for any of the core process measures.8 The time-to-reperfusion therapy measure includes only 10 percent of Medicare beneficiaries with heart attacks; measures of angiotensin-converting enzyme (ACE) inhibitors and beta-blockers at discharge are applicable for only 15 percent and 18 percent, respectively, of all beneficiaries with heart attacks. Only 22 percent of beneficiaries with heart failure were eligible for ACE inhibitors at discharge.9 Therefore, the care for many beneficiaries with heart attacks or with heart failure is invisible to the public, despite the reporting of hospital performance measures. Process measures can lack information on the effectiveness of the process being measured. Process measurement can be burdensome and, in the interest of efficient collection of information, may miss information related to the effectiveness of the process being measured. For example, information about whether a drug is prescribed may miss information about whether the proper dose was used or whether ill-advised drug combinations were avoided. Information about whether smoking cessation counseling was performed may miss the duration and quality of the interaction, which could be related to its ultimate success. Thus, hospitals with similar performance on measures could have differences in their outcomes. The updating of process measures is often difficult. As medical science progresses, new insights are continually redefining best practice. Thus, the ability of process measures to keep pace with the rapidly expanding body of medical evidence is difficult. The challenge ahead for process measurement lies in establishing credible mechanisms, insulated from inappropriate industry influence, which allow for the continual and responsive refinement of process measures over time. Furthermore, these efforts will need to determine when to retire a measure that is no longer relevant. For the core measures, the path for modifying or retiring a current measure or introducing a new measure is under active development but is not yet codified. Exclusive emphasis on measured processes may divert attention from important unmeasured processes. The sole focus on a concise set of process measures could have unintended consequences for the overall care of patients. For instance, hospitals might divert resources from other important aspects of care to ensure that they do well on the reported measures. Consequently, a performance report that focuses on a narrow set of process measures might be worse than no report at all. As recognized in the literature on multitasking, rewarding performance in only a subset of important tasks could undermine the overall goals of an organization, with rewards for one task causing an organization to reduce efforts in other unrewarded processes that are important to the final product.10 Similarly, in medicine, the diversion of resources and attention from key processes that are not publicly reported could have the net effect of worsening care and outcomes for patients, even as the institution’s performance according to public quality measures improves.
The interest in tracking outcomes is not new. The use of objective data to profile hospital performance dates from the work of Florence Nightingale in the mid-to-late nineteenth century.11 In the United States in the early twentieth century, the Boston surgeon Ernest Amory Codman promoted the importance of outcomes measurement, but it did not become a standard in his time.12 In 1986 the Health Care Financing Administration (HCFA, now known as the CMS) published hospital-specific mortality rates for numerous diagnoses, referred to by some as the HCFA "death list."13 The effort received substantial media coverage, with twice as many negative headlines as positive ones.14 Ultimately, amid criticisms of the methodology, the program was abandoned. In the late 1980s, clinical databases for cardiac surgery were initiated by the state of New York, the Northern New Eng-land Cardiovascular Disease Study Group, the Society of Thoracic Surgeons, and the Department of Veterans Affairs.15 Results of risk-adjusted outcomes by surgeon and institution from these databases were employed for a variety of purposes ranging from internal quality improvement activities to public report cards. The focus on outcomes measurement for cardiac surgery evolved because of an absence of medical literature supporting specific processes of care. Notably, early reports demonstrated wide variability in outcomes not attributable to case-mix, and this led to numerous interventions that ultimately reduced both interprovider variability and absolute mortality rates.16 Some states, such as California and Pennsylvania, have intermittently used databases constructed from billing data and reported mortality rates for patients hospitalized with heart attacks.17 These efforts had only modest effects on hospital volumes.18 However, public reporting of outcomes was lauded by some experts and perceived to be useful by many administrators; it also might have led hospitals to make changes in marketing, governance, and clinical care.19
Several recent advances have laid the groundwork for the use of outcomes as a part of an effort to publicly report hospital-level performance measures for patients hospitalized with heart attacks and with heart failure. Development of standards for outcomes measures. The American Heart Association, in collaboration with the American College of Cardiology, issued a Scientific Statement that details the following seven attributes of statistical models that are suitable for public reporting of performance results: They should (1) employ a clear and explicit definition of the patient sample; (2) include a clinically sensible choice of covariates; (3) use high-quality and timely data; (4) not include factors that could represent complications rather than comorbidities; (5) designate an appropriate outcome evaluated at a standardized period of time; (6) apply an appropriate statistical approach; and (7) disclose the methods in detail, including information about the performance of the measurement system. These attributes set standards and allow for the comparison of different approaches. An important concern about outcomes performance measurement is that profiling systems cannot fully account for differences in the patients among institutions. This issue is particularly important with the use of administrative data, which contain limited information about patients’ clinical condition. National profiling efforts are constrained in their data source because CMS claims are the only national database that can be used to characterize hospitals. The American Heart Association Scientific Statement addressed this concern by recommending that models using administrative data should be validated against models based on medical record data to ensure that the data source does not affect the classification of hospitals. HealthGrades, U.S. News and World Report, and Solucient use models to determine performance using outcomes data, but none yet fully complies with these standards. In particular, these proprietary systems do not fully disclose their methods or their approach to validating their models. Development of outcomes measurement models that comply with these standards. Outcomes measurement approaches can be developed that comply with the American Heart Association standards. Under contract with the CMS, some of the authors recently conducted a project to determine whether a model using administrative claims data could be considered suitable for hospital public reporting of thirty-day mortality rates among patients with heart attacks or with heart failure.20 Models were developed that complied with the standards. With respect to validation, for the calculation of hospital-level risk-standardized mortality rates, the estimate from the statistical model based on administrative claims data was an excellent proxy for the estimates from a model based on detailed medical record information. For example, for heart attack, using a sample of more than 4,500 hospitals, the estimates of hospital-specific risk-adjusted mortality rates from the claims data–based model were strongly correlated with the estimates from the model based on medical record data (correlation coefficient, 0.90 [standard error = 0.003]; slope of the weighted regression line between the two estimates, 0.95 [standard error = 0.007]). Furthermore, the median difference between the estimated claims-based hospital risk-standardized mortality rates and the medical record–based rates was 0.001 (25th, 75th percentile, –0.003, 0.003, respectively).21 This indicates strong agreement of the hospital estimates between the two data sources. Similar results were found for heart failure. These models were subsequently endorsed by the National Quality Forum. For hospital profiling, although the models’ discrimination at the patient level was higher with the medical record model, the focus should be on whether the results of the claims model with respect to hospital profiling are similar to those of the chart model. The models focus exclusively on the Medicare population. The lack of availability of national data limits our ability to characterize the outcomes for non-Medicare patients, although the vast majority of patients with heart attacks and with heart failure are eligible for Medicare. Another important issue is the standardized period of follow-up. The CMS model addresses outcomes within thirty days of hospital admission. This approach circumvents the problem of how to count transfer patients, who tend to be healthier, on average, than other patients. Hospitals with high transfer rates can appear to have worse outcomes if patients transferred out are excluded. This approach ensures that variation in lengths-of-stay do not influence hospitals’ comparative performance. In addition, the focus on a period that is longer than the length-of-stay ensures that events early after discharge are captured and places a premium on appropriate discharge planning. A keystone in the effort to develop these new models was the use of "hierarchical" or multilevel models.22 Such models permit separation of within- and between-hospital variation in observed outcomes and provide more-precise estimates of hospital-specific outcomes. In fact, hospital estimates obtained from hierarchical models are intended to circumvent the "regression to the mean" problem—that is, the tendency for hospitals that have been identified as outlying to become less extreme in future profiles.
The models of hospital-specific risk-adjusted mortality for heart attacks and for heart failure demonstrate important hospital-level variation in the outcomes of patients who are hospitalized throughout the country (Exhibits 1  and 2). The breadth of this variation across institutions, which persists even after differences in case-mix are adjusted for, highlights the value of examining outcomes as another measure of hospital performance. For instance, for heart attacks, if all hospitals achieved the risk-adjusted mortality rate of the hospitals in the top quartile, almost 10,000 lives would be saved each year.
This variation should provide a rationale for hospitals to invest in systems to improve care and survival rates. Such systems require substantial root-cause analysis to better understand the relationships among various hospital attributes (that is, structures, processes, and environmental features) that are essential to exemplary outcomes. Root-cause analysis, failure-mode analysis, and process redesign efforts could lead to a more general understanding about how organizational context influences clinical quality, which might improve care for all patients, even those whose outcomes are not routinely reported.
Sample sizes. There are several key concerns about outcomes measurement. First, many hospitals have small numbers of patients with a particular clinical condition. This sample-size issue is also a challenge to process measurement. The hierarchical modeling approach minimizes the impact of the small sample sizes by borrowing power across the entire sample and ensuring that hospitals with small numbers are not identified as outliers because of adverse events in just a few patients. Attribution. Attribution is another important issue. Who should be responsible for the outcomes of care? The issue is also relevant to process, but it might be even more pertinent to outcomes because of the many people and institutions that contribute to a patient’s care. In particular, hospital transfers can affect the perception of an institution’s performance.23 The thirty-day mortality measure developed for the CMS assigns responsibility to the hospital to which the patient was originally admitted. This approach puts the responsibility on that original hospital to transfer patients appropriately, with regard to both timing and location. If the receiving hospital cannot provide high-quality care, then the sending hospital should consider other options. Another issue in attribution is whether the outcome measured is related to the condition being assessed, especially when outcomes are measured across inpatient and outpatient venues. For example, the death of a patient discharged from a hospital after an admission for a heart attack who dies in a traffic accident on the way home is counted the same as that of a patient who died a preventable death in the hospital. In a national outcomes measurement system, it will be difficult to adjudicate outcomes, and some adverse outcomes might be attributed to an institution that did not have responsibility. However, some outcomes that initially appear to be unrelated could be a direct result of care. A patient who is overmedicated for blood pressure control might have a fainting episode and break a hip, resulting in an extended hospitalization and even death. The death certificate might describe the immediate cause as hip fracture, even though it was related to improper medications after discharge for a heart attack. In assessments of short-term results after an admission for a life-threatening condition, it seems reasonable to assume that the vast majority of adverse outcomes are related to the condition for which the patient was admitted to the hospital. Access to care. Another key concern pertains to whether close scrutiny of outcomes will lead practitioners and hospitals to shun high-risk patients, restricting access to needed services. This issue might be more of a concern for elective procedures, such as bypass surgery, than it is for urgent admissions such as a heart attack or heart failure. A valid risk-adjustment method attenuates this incentive to some degree; however, the introduction of such outcomes measurement systems should be accompanied by surveillance systems to ensure that access is not restricted as a result. Clarity of approach. Finally, the approach needed to improve outcomes is less clear than that for process measures. Clinicians and organizations need to find ways to respond positively to information about their outcomes. This response is different from the targeted solutions required by process measures and might entail the adoption of root-cause analyses and the development of approaches that diagnose and treat system failures, including coordination failures that cross clinical disciplines and provider organizations.
There are several challenges for the future. First, continued efforts are needed to refine methodological approaches to measuring outcomes. The United States invests only 0.1 percent of its biomedical research budget in health services research, yet there is a great need to generate more evidence to improve health care delivery in general and measurement of health care performance in particular.24 The future of outcomes measurement should also ideally extend beyond mortality and readmission. The trajectory of recovery after illness, health status (including symptoms, functioning, and quality of life), and satisfaction are key outcomes of health care that are currently not visible to the public and need to be developed to complement mortality measures. Outstanding survival rates will not convey the performance of an institution adequately if they are associated with high rates of disability. Although credible instruments are available to assess these domains, the reliable collection of such data is expensive and not routinely done. Nevertheless, any comprehensive examination of outcomes ought to include a full spectrum of results that are important to the patient. It is essential to determine the best approach to distilling and communicating both process and outcomes data for actual use by consumers. Important research has been done on how consumers process and use information to make health care decisions.25 The challenge is important, because adding outcomes to existing quality reporting on processes increases the complexity and breadth of the data. Ultimately, the appropriate communication of the findings, acknowledging the strengths and weaknesses of the information, might be the critical link to translating these measures into meaningful actions that benefit patients. INTEGRATING OUTCOMES WITH EXISTING PROCESS MEASURES provides a more comprehensive view of hospital performance than is possible if outcomes are ignored. The assessment of outcomes is particularly important given the hospital-level variability in risk-adjusted mortality rates and the limited degree to which process measures predict some important outcomes at the hospital level. These measures are readily interpretable by the public and address the results of care that are important to them. The establishment of standards and advances in methodology are making these measures appropriate for public reporting. Ideally, routine measurement and reporting of hospital-specific risk-adjusted outcomes might motivate a close and systematic examination of adverse results so as to identify common causes and possible interventions to reduce their occurrence. To be most effective, however, public reporting of quality must be interpretable by patients and their families. Presented clearly and in a user-friendly way, with appropriate caveats and limitations, data on both process and outcome provided to the public offer an important advance in transparency and quality assessment for the public and health care professionals.
Harlan Krumholz (harlan.krumholz{at}yale.edu) is the Harold H. Hines Jr. Professor of Medicine and Epidemiology and Public Health, Yale University School of Medicine, in New Haven, Connecticut. Sharon-Lise Normand is a professor of health care policy (biostatistics) in the Department of Health Care Policy, Harvard Medical School, in Boston, Massachusetts. John Spertus is a professor of medicine at the Mid America Heart Institute of St. Luke’s Hospital in Kansas City, Missouri. David Shahian is a professor of surgery at Tufts University School of Medicine in Boston. Elizabeth Bradley is a professor of epidemiology and public health at the Yale University School of Medicine. Harlan Krumholz has a research contract with the Colorado Foundation for Medical Care and serves on the advisory board of United Healthcare. John Spertus has a research grant from Cardiovascular Therapeutics Inc. and is a consultant for that company and for United Healthcare.
This article has been cited by other articles:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Top
Reasons To Add Outcomes...
Background On Outcomes...
