|
Report Card BiasTo the Editor:Mark Chassin’s July/Aug 02 paper on New York’s cardiac surgery report cards should encourage those who believe that competition can enhance quality. Unfortunately, the report suffers from a potential statistical bias. A well-known problem with report cards is that providers can observe patient characteristics that are not accounted for by the statisticians who compute mortality rates. If providers operate more often on "healthy" patients (that is, healthier than they appear to the statisticians) and less often on "sicker" patients, surgical mortality rates will fall. But overall mortality rates may increase because the wrong patients undergo surgery. Avoiding this bias requires examining an entire population at risk for heart surgery. Together with colleagues, I recently examined the effects of report cards in New York and Pennsylvania (which introduced report cards in 1992) on such a population: all acute myocardial infarction (AMI) patients.1 We compared treatments and outcomes in the two states against other urban states, thereby controlling for the time trend. Using two severity measures that are observed by providers but not used by statisticians—prior hospitalizations and prior hospital expenditures—we found that New York and Pennsylvania surgeons operated less often on sicker AMI patients and more often on healthy patients. For the AMI population as a whole, average health status deteriorated, mortality increased, and average treatment costs increased during the year after the AMI. Report cards seem to have left the AMI population worse off. We confirmed Chassin’s finding that average surgical mortality rates fell, but this is the wrong yardstick. Our analysis only extended to 1994. Chassin describes several provider responses that may have yielded long-run quality improvements. But the only way to know for sure will be to perform the proper statistical analyses.
Northwestern University, Evanston, Illinois
The author responds: The analysis by Dranove and colleagues, referred to directly above, is conceptually, clinically, and rhetorically flawed. Conceptually, the authors confuse two important but separate dimensions of quality: the appropriateness with which patients are selected to receive a treatment, and the skill with which that treatment is rendered. Their attempt to assess whether "the wrong patients undergo surgery" addresses the former. The New York system of reporting risk-adjusted operative mortality following coronary artery bypass graft (CABG) surgery addresses the latter by using clinically and statistically proven measures to adjust for differences in risk of death among different patient populations. In contrast, the analysis by Dranove and colleagues uses severity measures that have no clinical or statistical validity. Ignoring a host of clinically and statistically valid measures of myocardial infarction (MI) severity, they use data on prior hospital expenditures and hospitalizations but provide no evidence that establishes the validity of those data as severity measures. Applying the term "sicker" to a group of patients simply because their prior hospital expenditures were higher than those of another group is wholly inappropriate. Dranove’s conclusion that after the initiation of mortality data reporting, New York surgeons operated more often on "healthy" MI patients based on measures of lower prior hospital expenditures is similarly unwarranted. In fact, comprehensive clinical data document that the opposite occurred. The severity of illness of patients undergoing CABG surgery in New York—as measured by their predicted risk of operative mortality—increased 35 percent over the first four years of the reporting system, and the numbers of the highest-risk patients increased 73 percent between 1990 and 1992.1 Further, using clinically valid and "proper statistical analyses" to address the appropriateness question raised by Dranove, a group of RAND researchers found an extremely low level (2 percent) of inappropriate CABG surgery in New York.2 New Yorkers should be reassured to know that their physicians and hospitals are using clinically sensible and analytically robust data to markedly improve the outcomes of patients who undergo CABG surgery. We should all hope that providers elsewhere do the same.
Mount Sinai School of Medicine, New York, New York NOTES
|
