|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hospital Performance Reports: Impact On Quality, Market Share, And Reputation
This study builds on earlier work by assessing the long-term impact of a public hospital performance report on both consumers and hospitals. In doing so, we shed light on the relative importance of alternative assumptions about what stimulates quality improvements. The findings indicate that making performance data public results in improvements in the clinical area reported upon. An earlier investigation indicated that hospitals included in the public report believed that the report would affect their public image. Indeed, consumer surveys suggest that inclusion did affect hospitals reputations.
Several studies have assessed the efficacy of public performance reports in stimulating hospital quality improvement; they have produced mixed results.1 These studies relied primarily on pre-post observational designs and generally did not assess the reports impact on consumers. A more recent evaluation, using a controlled experimental design, produced strong evidence for the effectiveness of public reporting on quality improvement.2 Analyses that examined the impact of the public performance report, QualityCounts, found that nine months after the release of the report, hospitals included in the public report were significantly more likely to be engaged in quality improvement efforts than those given a confidential, private quality report or no report at all.3 In addition, surveys of consumers two months after the reports release indicated that they had changed their views regarding the relative quality of community hospitals and accurately recalled hospitals ranked as higher or lower performers.4 Three different mechanisms appear to drive hospital quality improvement: regulation, professionalism, and market forces.5 Public reporting is thought to be a key strategy for influencing market forces and, to a lesser extent, professionalism; there are at least three alternative assumptions about how this process works: (1) Public reporting promotes informed consumer choice and subsequent increases in market share.6 (2) Public reports can affect the public image or reputation of a provider or medical care organization, and concern for protecting or enhancing professional or institutional reputation will motivate quality improvement. (3) The feedback inherent in both public and private reports will be sufficient to stimulate efforts to improve quality, simply because of professional norms around maintaining standards and self-governance. These three mechanisms are not mutually exclusive. If a hospitals reputation is affected, it may eventually experience market share declines via consumer choice, purchaser choice, or physician referral. A declining hospital reputation may pose other challenges such as recruiting and retaining qualified physicians and nurses. Similarly, both market share and reputation can affect a hospitals ability to maintain legitimacy and professional standing. Both the market share and reputation assumptions assign a key role to the consumer in motivating quality improvements, by attending to performance reports, identifying high and low performers, and sharing that information with others. Under the feedback assumption, there is no role for consumers.
Context. The Alliance, a large employer purchasing cooperative in Madison, Wisconsin, sponsored the public report on hospital quality and safety used in our study.7 The report, QualityCounts, compared performance on twenty-four hospitals in south central Wisconsin. Two summary indices of adverse events occurring within the broad categories of surgery and nonsurgery were included, along with indices summarizing three clinical areas: hip/knee surgery, cardiac care, and obstetric care.8 Hospitals were rated as better than expected (fewer deaths/ complications), as expected, or worse than expected. The data were derived from the Wisconsin Bureau of Health Information inpatient public use data sets. Measures were adapted from the original HCUP (Healthcare Cost and Utilization Project) Quality Indicators and were summarized and risk-adjusted by Medstat using its disease staging methodology. Several aspects of the report might have helped increase the impact on consumers. First, data were displayed in a way that made it very easy for consumers to determine which were the better- and worse-performing hospitals.9 For example, hospitals were ordered by performance, and the top-tier performers were highlighted with a color band. Second, the report was widely disseminated. It was inserted into the Madison newspaper, sent to employees homes, and featured in newspaper stories throughout the Alliances service region. It was also available on a Web site, and hard copies were distributed by community groups and at libraries. This was the first public report on hospital quality issued in this region, and it generated substantial public interest. Finally, the report indicated significant performance variation in two key clinical areas, obstetric and cardiac care. Many performance reports fail to show much variation and often do not capture the publics attention. Because of the variations in this report, it might have been more interesting to consumers. Three characteristics of the hospital markets in Wisconsin may have affected how hospitals responded to the report. First, Wisconsin hospital markets tend to be tightly aligned with physician groups. Physicians in these physician hospital organizations (PHOs) practice only at the hospital with which their organization is aligned. This alignment includes rural physicians who are linked with a rural hospital and, in turn, aligned with a particular regional hospital. More than 85 percent of physicians in the Alliance service area participate in these arrangements.10 Virtually all of the hospitals in south central Wisconsin have either formal or informal PHO arrangements, compared with about 30 percent of hospitals nationally.11 Second, there are far fewer multihospital systems in Wisconsin than nationally. During the time of the study, only two hospitals in the Alliance service area (8.3 percent) were part of a multihospital system, compared with 63 percent of hospitals in urban areas across the nation.12 Finally, there are no for-profit hospitals in the Alliance service region, compared with 27 percent of hospitals nation-wide.13 Although the Alliance did not release another public report after disseminating QualityCounts, other reporting efforts in the community could have heightened consumers awareness of quality issues. In spring 2004 the Wisconsin Hospital Association released an online report on hospital performance; however, it did not include any of the measures that were included in QualityCounts, was available only on the Web, and presented information in a format that would not be easy for consumers to understand or evaluate. Also in 2004, the Wisconsin Collaborative for Healthcare Quality, a voluntary statewide consortium of health care organizations, reported a set of common measures of health care quality outcomes. The measures reported by the collaborative also do not overlap with the Quality-Counts report and were aimed primarily at a purchaser audience. Background. An earlier analysis of consumers views of the quality of Madison-area hospitals two months after the reports release in 2001 indicated that views on better- and worse-performing hospitals shifted from the pre-report period and that the shift corresponded to the rankings in the public report. Respondents who saw the public report recalled which hospitals were ranked as higher or lower performers. Also, the majority of consumers who saw the report talked to others or planned to talk to others about it. This combination has the potential to affect hospitals reputation.14 An earlier analysis also examined the public reports impact on hospitals quality improvement efforts in the first nine months after its release. The findings indicated that making performance information public stimulated quality improvement activities in the areas where performance was reported to be low. Quality improvement efforts among the hospitals listed in the public report were significantly greater than among hospitals given only a private confidential report and those given no report. Also, hospitals viewed the report as affecting their public image but not their market share. Concern for public image appeared to be a key motivator for hospitals quality improvement efforts.15 In this paper we assess hospitals performance in the two years following the release of the report, along with the reports long-term effects on hospital reputation. Specifically, we address the following research questions: (1) Did performance of hospitals in the public report improve in the areas reported upon during the two years following the reports release? Did improvements occur beyond what was observed in the "no report" and "private report" hospitals? Was improvement related to reported quality improvement activities? (2) To what degree did "private confidential reports" result in performance improvements (a test of the feedback assumption)? (3) Was there any long-term impact on hospitals reputation? Did consumers remember which hospitals were ranked as high- or low-performing hospitals (a test of the reputation assumption)? (4) Did the report affect consumers choice or market share (a test of the market share assumption)?
Long-term impact on hospital performance. Our experimental design includes two intervention groups and one control group. The primary intervention group was the twenty-four hospitals in south central Wisconsin that are in the Alliance service area and that were included in the public QualityCounts report. (They also received a more detailed report on their own performance.) These hospitals were not randomly selected or assigned but were included as an intervention group because they were in the public report. The other ninety-eight general hospitals in Wisconsin were randomly assigned to either the secondary intervention or the control condition. The former received a "private, confidential report" on their own performance (performance information was not made public). The latter received "no report." Seven hospitals were eliminated from the analysis because performance data were unavailable or incomplete, which left 115 hospitals. At baseline, we observed no statistically significant differences among the three groups of hospitals in their characteristics, size, or baseline levels of performance. The public report was widely disseminated and the private reports were delivered in fall 2001. We obtained performance data for the two-year post-report period and risk-adjusted them using the same methodology employed for the QualityCounts report. The only performance areas in which there was notable variation in performance in the original report was obstetric care and, to a lesser extent, cardiac care. Thus, our analysis focuses only on changes in these two clinical area for the three groups of hospitals (public, private, and no report). To evaluate the impact of reporting on hospital quality improvement, we used three different approaches. First, we simply compared the level of hospital performance (worse than expected, as expected, or better than expected) at baseline to two years post-report across the three groups. We paid special attention to the hospitals identified as doing worse than expectedthe group that should have been most motivated to change. Second, even though the risk-adjusted performance scores took into account a wide variety of factors that could affect outcomes, we also conducted a meta-analysis using the Mixed procedure in SAS. This allowed us to introduce even more stringent controls for the different variances in performance scores and sizes of the hospitals included in the sample. Finally, we looked at the degree to which hospitals quality improvement efforts reported in the immediate post-report period were related to changes in performance levels. We used an approach to market-share analysis that examined the proportional changes in discharges from different hospitals both for total care and for specific medical conditions.16 Claims data for the hospitals in the public report condition were obtained from the Alliance, including hospital claims for employees of member employers and their families. The data contained the total number of discharges and dates of service for all hospital care and for obstetrics and cardiac care. A total of 2,234 discharges were examined, encompassing hospital care one year before the release of the report and one year afterward. We assessed changes in the proportion of discharges from the pre to the post period. Because there might have been a lag in the effect on market share, this assessment included looking at different increments of time during the post-report period: 04 months, 46 months, 612 months, and 112 months. In each analysis, we examined the relative proportion of market share. Long-term impact on consumers. Although the portion of the evaluation that focused on the hospitals used an experimental design and included the entire state of Wisconsin, the consumer portion focused just on the Alliance service area and used a pre-post design. We used telephone interviews to assess the public reports impact on consumers. Because the report was sent to employees and inserted into the local newspaper, we investigated its impact on both employees and the general public. One month before the report was released, a baseline telephone survey of employees (n = 93) was conducted along with a baseline community survey using a random-digit-dial sampling strategy. The same respondents were resurveyed one to two months after the release of the report. Because the baseline might have sensitized respondents to the issue of hospital performance, a post-only community random sample was also interviewed (n = 469). A newspaper story about the public hospital report came out during the baseline survey data collection period; to avoid any "contamination" from the story, the baseline community random survey was cut short, which curtailed the size of the community panel (n = 67). Two years after the release of the report, a final new random sample of community members was surveyed by telephone (n = 803).
Impact on hospital performance. Exhibit 1
When we examined only the hospitals that had obstetric performance scores that were worse than expected at baseline, the differences were more dramatic (Exhibit 2
These changes appear to be related to quality improvement efforts. In the immediate post-report period we asked each hospital to indicate how many quality improvement activities to reduce obstetrical complications it was doing.17 Out of seven possible activities, the average was 4.1 among the "public report" hospitals. However, these activities were not evenly spread among the hospitals. Among the nine hospitals that showed an improved rating in the postreport period, the average number of activities was 5.7. Among the ten hospitals with no change, the average number of activities was 2.6. The one hospital whose rating declined reported four activities to reduce obstetrical complications. Results of the meta-analysis are slightly different. The fixed-effects model indicates significant differences in weighted mean change scores across the three groups (F = 4.02; df = 2,92; p < .05). The model indicates that after hospital size and differences in variances were adjusted for, the "public report" hospitals had improved performance scores significantly more than the "no report" hospitals (t = 2.25, df = 92, p = .03). In contrast to the results from the analysis of change in performance ratings, the "private report" hospitals also had improved performance scores significantly more than the "no report" hospitals (t = 2.50, df = 92, p = .01), and the difference in average performance changes between "public report" and "private report" hospitals was not statistically significant (t = 0.13). Further inspection revealed that the discrepancy between the two analyses occurred primarily because of the changes at one large regional hospital in the public report, the only one within that group to decline in ratings from the baseline to post-report period (from "as expected" to "worse than expected"). When this hospital was omitted, the fixed-effects model was still significant (F = 10.09, df = 91, p = .0001), but the differences in mean change scores between all three groups were significant and paralleled the analysis of changes in ratings. The largest weighted mean change score occurred for the "public report" hospitals (estimate = .69, standard error = .33) and the smallest for hospitals in the "no report" group (estimate = .10, SE = .33; for private hospitals, estimate = .37, SE = .33). (The comparison of weighted mean change scores for public and private report hospitals resulted in t = 2.36, df = 91, p = .02; between public and no report, t = 4.42, df = 91, p < .0001; between private and no report, t = 2.50, df = 91, p = .01.) Results for cardiac care mirrored those for obstetrics but failed to reach statistical significance. This finding is not surprising, given that there were far fewer hospitals with poor scores in cardiac care. Impact on market share. There were no significant changes in market share among the hospitals in the public report from the pre to the post period. We observed no shifts away from low-rated hospitals and no shifts toward higher-rated hospitals in overall discharges or in obstetric or cardiac care cases during any of the examined post-report time periods. Impact on consumers. We observed some demographic differences among the groups that were surveyed at the three different points in time. The employee sample was younger and less-well-educated than the random community samples; and the final random community sample had more women than the other samples. We included these demographic factors as control variables in our analyses; these differences did not affect our conclusions. The employee panel had the greatest exposure to the report: 57 percent reported having seen the report, and 61 percent had been exposed to it in some way (saw the report, heard about it from someone else, or read about it in the newspaper). This is to be expected, because some of the employers in the Alliance sent the report directly to employees homes. In addition, the baseline survey apparently sensitized respondents to the issue of hospital quality because a higher percentage of the random community panel members (39 percent) were exposed to the report than the random post-only group (24 percent). Six percent of respondents in the new random sample surveyed two years after the reports release remembered seeing it, and 14 percent were exposed in some way. In the immediate post period, only 4 percent of consumers who were exposed to the report used it to recommend or choose a hospital, and only 10 percent reported having done so in the two years after its release. However, 24 percent had talked to others about the report in the immediate post period, and almost half had talked to others in the next two years. Almost no one spoke with their doctor about it.
To determine if the report affected how consumers viewed the quality of hospitals in their community, all respondents were asked which hospitals had fewer preventable complications and which made fewer medical mistakes. Exhibit 3
Recall of poorly performing hospitals was better than recall of high performers. Respondents were asked about which hospitals were more likely to make medical errors and which had more preventable complications. Exhibit 4
These findings provide substantial evidence that making performance information public stimulates long-term improvements beyond those stimulated by private reports. Among hospitals with low scores in obstetric care at the baseline period, those in the public report were significantly more likely than those in the other two conditions to have improved their scores by the post-report period. The one exception to this pattern was one large regional hospital, a result that led to no significant differences between the "public report" and "private report" hospitals in the meta-analysis. Differences between the "no report" hospitals and the "private report" hospitals lend some support for the feedback hypothesis. Hospitals that improved their performance were much more likely to engage in a wide range of increased improvement efforts immediately after the report. In other words, the improvements appear to be linked to quality improvement efforts that began immediately after the reports release. Effect on reputation. Our earlier investigation indicated that "public report" hospitals believed that the report would affect their public image but not their market share. The data support this belief. Consumers exposed to the public report were much more likely than other consumers to have accurate perceptions of the relative quality of local hospitals, and these perceptions persisted for at least two years after release of the report. Taken together, these findings suggest that the mechanism by which public reporting affects improvement is more likely to lie with concerns about reputation than with concerns about market share. Moreover, the concern about reputation is well founded, for consumers exposed to the public report were more likely than others to accurately remember the results. Hospitals have other reasons to be concerned about their reputations. Although professional pride is likely a motivating factor, more concrete financial issues could also be influenced by changes in hospital reputation. For example, a downturn in reputation could affect a hospitals ability to raise funds. Charitable donations are an important source of income for not-for-profit hospitals. Alliance-area hospitals receive on average $250,000 a year from fund-raising activities.18 Reputation concerns could also affect how hospital boards allocate future resources and set priorities. Because of the tight alignment between hospitals and physician groups in this market, poor performance scores could disrupt or threaten these PHO arrangements; this could affect market share and ultimately the bottom line. Unique features of public report. What is different about this public report, and why have other studies of public reporting not shown similar results? One possible reason is that the public report was widely disseminated in the community and designed so that consumers could easily evaluate it. For example, the report highlighted high- and low-performing hospitals by rank-ordering them on performance. This can have a powerful effect on reputation, as we saw in this investigation. However, most public reports are complex and difficult to interpret and therefore do little to enhance or threaten institutional reputations. The implication is that public reports, if implemented in the way that the QualityCounts was designed and disseminated, could have a strong positive impact on quality improvement efforts. Study limitations. The hospital markets we studied have unique characteristics, including the tight alignment between physician groups and hospitals, so the findings should be interpreted within the context of these characteristics. These physician-hospital alignments could help to explain the hospitals lack of concern about market-share changes and the fact that no market shifts were observed. That is, because of this tight alignment between physicians and hospitals, hospitals are aware that for patients to change hospitals, they would have to take the unlikely step of changing physicians. This suggests that in markets where this high degree of alignment is not present, a public report could raise concerns about both reputation and market share, motivating improvements through both of these pathways. That is, in less aligned markets, both market share and reputation could be activated by a well-designed and widely disseminated public performance report.
Judith Hibbard (jhibbard{at}uoregon.edu) is a professor of health care policy; Jean Stockard, a professor; and Martin Tusler, a data analyst, in the Department of Planning, Public Policy, and Management, University of Oregon, in Eugene. The authors thank the Robert Wood Johnson Foundations Changes in Health Care Financing and Organization (HCFO) Initiative for financial support and the Alliance for its participation.
This article has been cited by other articles:
eLetters:Read all eLetters
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||