|
The Public Versus The World Health Organization On Health System Performance
The World Health Organization (WHO) ranked health systems in 191 countries based on measures developed by public health experts. This paper compares the WHO rankings for seventeen industrialized countries with the perceptions of their citizens. The results show little relationship between WHO rankings and the satisfaction of the citizens who experience these health systems. The health systems of some top WHO performers are rated poorly by their citizens, including the low-income and elderly. The two rated most highly by the public rank at the bottom of the WHO ratings. These findings suggest that both public and expert views should be considered in international rankings.
On 21 June 2000 the World Health Organization (WHO), one of the most respected United Nations agencies, issued a report that ranked health systems in 191 countries across the globe. This study, which attracted extensive media coverage, was the first by any organization to undertake the huge task of rating the health systems of countries as disparate as Oman, San Marino, Andorra, France, and the United States on a common set of measures. But it was not only the prominence of the agency delivering the report and the scale of the undertaking that drew media attention. Ranked highest on WHO’s measure of overall health system performance were some national health systems that had not previously been seen as successful models, while other, more highly respected systems ranked much lower. For example, among industrialized countries, France, Italy, and Spain were seen as the top health system performers. Canada, Germany, Sweden, and Denmark, which are often cited as models by some experts, ranked lower. The United States ranked thirty-seventh among all countries and last among industrialized countries. Greece, Portugal, and Ireland, which spent much less of their national resources on health, ranked much higher on WHO’s scale than did the United States, the world’s biggest health spender. According to a press release issued by WHO, the rating of the 191 countries was based on five composite indicators: Overall level of population health; health inequalities (or disparities) within the population; overall level of health system responsiveness (a combination of patient satisfaction and how well the system acts); distribution of responsiveness with the population (how well people of varying economic status find that they are served by the health system); and the distribution of the health system’s financial burden within the population (who pays the costs).1 To many readers of this press release, the use of phrases such as those in italics above suggest that patients, the poor, or the general population in each country were interviewed to establish these particular measures of health system performance. In fact, no citizens or patients were interviewed. Rather, the report relied on a survey of public health experts, many of whom did not reside in the countries whose responsiveness to patients and the poor they were rating. In background papers for the June 2000 report, WHO argued that interviewing experts is a much more appropriate way to elicit this information than is interviewing the people who live in these countries. WHO made the following assertions: How well a health system performs is too complex for the general public to understand; it involves basic knowledge of public health issues that are not well understood; and patients’ perceptions are too narrow to fully assess a health system.2 Even on the issue of how responsive a health system is to individuals’ expectations for how health care should be provided to them, WHO prefers expert judgments because they are seen as being more reliable.3 Few would disagree that many measures of health system performance should come from those who have expert knowledge and data. But the real issue is whether citizens’ views and experiences should carry any weight in assessing a nation’s health system performance, or whether this domain belongs strictly to experts. This question has arisen in debates in other areas between experts and those in roles that make them accountable to the general public. Both in economics and in assessments of the overall performance of nations, experts believe that they have the better measures. However, those in publicly accountable roles believe that they cannot disregard the perceptions of persons who experience the economy and day-to-day life in their nation. As a result, in putting together reports about a nation’s economy or overall performance, agencies regularly include public perceptions as well as expert measures. If in fact citizens’ views are used, a second issue arises: whether to include only measures of individual experiences with the health system, or to broaden the inquiry to include also the public’s overall assessment of the system. Data about individuals’ experience, such as waiting times for elective surgery, are important. But overall public assessments give a composite measure of individual experiences, plus an assessment of what is going on in the health system more broadly. In the WHO report, neither individual experiences nor overall public satisfaction with health systems is used. A third issue is whether people can distinguish between their own health care and what goes on in the health system as whole. Prior research has shown that people can, in fact, make such distinctions. For example, in a 1998 five-country study, a majority of citizens in each nation reported that their health care systems had enough wrong with them to require major changes. At the same time, the majority of the citizens also rated as good or better the overall physician and hospital care they received during the prior year.4 The last issue often raised regarding the use of public opinion in system assessments is that such attempts measure satisfaction with the citizen’s own country as a whole, not with the health system as a unique entity. This has been shown in prior research not to be the case. In a five-year period during which Canadians expressed a higher level of satisfaction with their health care system than Americans did with theirs, Americans were consistently more satisfied than Canadians were with their country’s overall direction. Moreover, satisfaction with country and satisfaction with the health system do not necessarily remain parallel. In Canada the percentage of the public that said that their health care system worked pretty well and needed only minor changes fell from 56 percent in 1988 to 20 percent in 1998.5 Meanwhile, Canadians’ satisfaction with their country’s overall direction actually increased during this period, from 41 percent in 1989 to 50 percent in 1998.6 What would the WHO report have said if it had included a ranking of health system performance drawn from the experiences and perceptions of citizens? We try to give a partial answer by looking at seventeen countries for which data were available. All seventeen are industrialized Western countries with democratically elected governments, similar cultures, and complex health systems and are among the countries that spend the most on health care and advanced medical technologies. For these countries we compare rankings of citizens’ satisfaction with their own health care systems to the rankings awarded by the WHO expert panels. We also look at the views of vulnerable populations within these countries—the poor and the elderly—to see if they differ from the perspectives of their populations as a whole or from the WHO ratings. While the WHO report provided multiple measures of each country’s health system performance, two aggregate summary measures—overall system performance (the measure that received the most attention) and overall health system attainment—are central to our analysis. For each measure, the relative ranking of each country is compared with the satisfaction measures expressed by the citizens in that country. Also, we compare citizens’ views with one of WHO’s specific measures, system responsiveness, because it is the measure WHO used instead of overall public satisfaction.7
Sources of data. The data presented are derived primarily from three sources. The first is the Eurobarometer 49, a series of general population surveys in fifteen European countries.8 These national random-probability surveys were conducted for the European Commission by INRA (Europe) between 7 April and 27 May 1998. Because some European countries lack universal telephone coverage, interviews were conducted in person. The European Commission does not release response rates for the Eurobarometer polls. The surveys were conducted as part of a broader Eurobarometer survey series that routinely measures the views of citizens of the European Community on a number of issues. A standardized questionnaire was used in each country and was administered in the appropriate national language(s). Our comparisons focus on a single question: "In general, would you say you are very satisfied, fairly satisfied, neither satisfied nor dissatisfied, fairly dissatisfied, or very dissatisfied with the way health care runs in your country?" The first two response categories were combined to give a measure of the percentage satisfied in each country. The second source of data is two surveys developed by the Harvard School of Public Health and conducted by International Communications Research (ICR) in Canada and the United States. The surveys were conducted by telephone with randomly selected adults in the United States, 4–8 October 2000 (n = 1,007), and Canada, 5–11 October 2000 (n = 1,500). Questions were asked on nationwide omnibus surveys regularly used by news organizations in each country. Like the Eurobarometer surveys, these ongoing surveys do not provide response rates. Because the vast majority of Canadian (96 percent) and U.S. (94 percent) households have telephones, this mode of interview provides an accurate picture of public views. The surveys, using the identical wording of the Eurobarometer questions, were administered in English in the United States and English and French in Canada. In the analysis of data from both survey sources, the poor are defined as those from households with incomes in the country’s lowest quartile. The elderly are persons age sixty-five or older. The third source is the data for each scale of health system performance, which was derived from the WHO’s World Health Report 2000—Health Systems: Improving Performance. The health systems of the seventeen countries are ranked—with the best as 1, the worst as 17—according to their relative positions in the WHO study. The public satisfaction measures drawn from the European Commission surveys and the Harvard School of Public Health/ICR surveys were compared to the re ranked three measures provided by WHO: (1) overall health system attainment; (2) overall health system performance; and (3) responsiveness index of the health system. Rank-order correlation (Spearman’s rho) was calculated to provide statistical information on each pair of rankings being compared. WHO methods. Briefly, these three measures were developed by WHO using the following methodological approaches. More detailed information is available in the report. Overall system attainment. The system attainment indicator is an absolute measure of how well a country has done in achieving the five goals of a health system, defined by WHO experts. This composite measure of achievement sums WHO’s five health system goals into a single overall measure based on weights (25 percent level of health, 25 percent distribution of health, 12.5 percent level of responsiveness, 12.5 percent distribution of responsiveness, and 25 percent fairness of financial contribution). These weights were derived from an Internet-based survey on health system preferences with 1,006 public health practitioners from more than 125 countries. Half were WHO staff members; the other half were WHO Web site visitors who volunteered to participate in the survey. Overall system performance. The WHO overall system performance rating is an efficiency measure, which compares how short each country’s actual overall health system performance fell of the ideal maximum level it ought to have attained if the system operated efficiently, given its health spending and educational attainment. The WHO report describes it as a method of comparing how efficiently countries translate expenditures into health system outcomes. Among the various methods used by WHO to evaluate health systems, the "overall system performance" rating of the 191 member states has received the most attention and, as a result, is presented first in each of our analyses.9 Responsiveness of health systems. The WHO measure of responsiveness is an indicator of how the system performs in nonhealth aspects. According to the report, the measure intends to gauge whether or not a system meets its "population’s expectations of how it should be treated by providers of prevention, care or non-personal services."10 The responsiveness scores are constructed based on a survey of 1,791 key informants (a network of fifty or more informants in each country) in thirty-five selected countries. These experts were asked to evaluate their health system regarding seven elements of responsiveness: dignity, autonomy, confidentiality (jointly termed "respect of persons"), prompt attention, quality of basic amenities, access to social support networks, and choice of care providers (encompassed by the term "client orientation"). The scores on each component were combined into a composite score for responsiveness based on weights derived from the previously mentioned Internet-based survey. No informants were interviewed on this measure in 156 of the countries included in these ratings. Based on the survey results of these key informants from thirty-five countries, the responsiveness of the health systems in each of the other 156 countries was estimated using statistical estimation techniques.11
Overall health system performance. There appears to be little relationship between the WHO ranking on this measure and views of the citizens of each of these countries (Exhibit 1  ). Some of the differences between the two evaluations are striking. Among the seventeen countries in this comparison, Italy is ranked second by WHO. But only 20 percent of its citizens say they are satisfied with their health care system. This low citizen rating of the Italian health care system is not a new phenomenon. Three surveys during the past decade have shown that Italians rate their system poorly.12 At the other extreme, Denmark is ranked sixteenth in the WHO overall performance measure, yet 91 percent of Danish citizens say they are satisfied with their health system. Once again, those who live in that country have a different perception than the WHO experts have.
Spain and Finland provide two more examples of difference between the public and WHO experts. Spain ranks third among these industrialized countries using the WHO measure of overall system performance but thirteenth in satisfaction as rated by its own citizens. This relatively low ranking is consistent with findings of an earlier survey conducted for the Spanish parliamentary commission in the early 1990s.13 Finland, on the other hand, ranks fifteenth according to WHO but second in terms of its citizens’ levels of public satisfaction. France, the top performer on the WHO measure, ranked sixth in satisfaction by its own citizens. On either rating scale, the health care system in the United States performs poorly. On the public satisfaction measure, it ranks fourteenth; on the WHO measure, seventeenth. However, U.S. citizens are much more satisfied with their health system than are Italians, whose system ranks second in performance on the WHO ratings. Overall system attainment. As with the first measure, there is little correspondence between public satisfaction and the expert views of system attainment as measured by WHO. The top two countries in the WHO rankings are Sweden and Luxembourg, which rank tenth and fifth, respectively, in public satisfaction. The two poorest performers on WHO’s system attainment measure are Portugal and Ireland. In fact, Portugal ranks sixteenth in public satisfaction, but Ireland ranks eighth, nowhere near the bottom. Responsiveness of the system. This WHO measure of how responsive a health system is to the non health needs of its citizens seems to better reflect citizen satisfaction in each nation than the other two WHO aggregate measures. Yet there are large discrepancies here as well. The United States ranks number one on this measure but fourteenth in citizen satisfaction. Finland ranks eleventh on the WHO measure but second on citizens’ own ratings. Canada ranks fifth on responsiveness but twelfth on citizen satisfaction.
In examining the WHO rankings on the responsiveness measure, we were struck by how closely the WHO expert panel’s ratings seem to mirror the per capita health expenditures of the seventeen industrialized countries. Per capita health spending and a country’s expert-based responsiveness rating are highly correlated with each other (r = 0.93) (Exhibit 2
However, while more spending may lead to higher ratings on this measure, it does not necessarily produce more satisfied citizens. Nations that spend much less than the average for industrialized countries reported lower levels of citizen satisfaction with their health systems (Greece, Portugal, Italy, and Spain). But, beyond that point, other factors are important in determining whether a country’s health system meets its citizens’ expectations. Clearly, WHO’s responsiveness measures, which are based on experts’ views, are not capturing other concerns the public may have.
The poor and the elderly.
One of the specific WHO concerns was with how well people of varying economic status are served by their health systems. In our analysis we examined this by looking the views of two vulnerable populations, the poor and elderly, each of these seventeen countries (Exhibit 3
As we found previously for the general population, WHO expert measures do not correspond well with the views of the poorer citizens in these countries. The two countries whose poor citizens are most satisfied with their health care system are, once again, Denmark and Finland. These countries are rated near the bottom of both the WHO overall system performance scale and overall system attainment scale. By contrast, in Italy and Spain, two of WHO’s top rated countries, only 22 percent and 47 percent of poorer citizens are satisfied with their current health system. The poor in the United States, which ranks last in system performance and eleventh in system attainment, reported a higher level of satisfaction than did the poor in Italy, WHO’s second-rated country. This U.S. figure falls just slightly below that of Spain, WHO’s third-rated nation. When it comes to the views of its elderly, the pattern remains the same. The top two countries from the citizens’ perspective are Denmark and Finland, while once again, Italy ranks low. However, here the United States does better. The elderly are the one population in the United States that has guaranteed universal coverage. American elders are more satisfied with their health system than are citizens in seven of these other countries, all of which are rated higher in overall health system performance. The WHO measure of responsiveness also does not closely mirror the view of the poor and elderly in many instances. Finland ranks second based on the views of its poor but eleventh in WHO’s responsiveness rankings. At the other end, Canada ranks fourteenth based on the views of its poorer and older citizens but fifth on the WHO responsiveness measure. The United States ranks first in WHO responsiveness but thirteenth and tenth in the level of satisfaction expressed by its poor and elderly, respectively.
This divergence in views between WHO experts and the public on the performance of health systems raises fundamental questions. In democratic societies, what importance should be placed on taking into account the views of the general public in assessing how well the health system works? Clearly, there is a gap in what is seen as important by one group versus the other.14 The public is likely to see health systems as meaning health care delivery systems. A well-functioning health system from the public’s perspective involves what physicians, hospitals, emergency rooms, ambulances, health centers, insurance agencies, and government officials do in the way of providing and financing health care services. Public health experts see better health systems as those that improve health by encouraging better diets; reducing environmental risks, violence, injuries, and tobacco use; and improving water supplies and living conditions. Rating these latter domains may be an area where expert opinion should predominate because many of the issues are less visible to average citizens. But the health care system is something different. Most families have direct experience with it. It is discussed widely in the media, and citizens have many contacts with health professionals. Here, the general public has a basis for informed opinion, particularly in areas relating to barriers to care, financial difficulties, unresponsive health system bureaucracies, shortages of services, lack of respect for patients by professionals and administrators, inefficiencies, and corruption within the health care system. Citizens in each country are in as good or a better position to assess these issues in their own countries as experts are. Although we recognize that some countries in the Third World may have difficulties in conducting population-based surveys, we feel that not taking public views into account is a serious shortcoming in this international effort.
Robert Blendon is a professor of health policy and political analysis at the Harvard School of Public Health and Kennedy School of Government; Minah Kim is a doctoral fellow in health policy at Harvard University; John Benson is the managing director of the Harvard Opinion Research Program at the Harvard School of Public Health. This work was supported in part by the Commonwealth Fund and the Robert Wood Johnson Foundation. The views expressed are solely those of the authors, and no official endorsement by either foundation is intended or should be inferred.
This article has been cited by other articles:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Top
Data And Methods
Results
