This Article Extract Reprint (PDF) Submit a response to this article Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services E-mail this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to My Personal Archive Download to Citation Manager Reprints & Permissions Citing Articles Citing Articles via HighWire Citing Articles via ISI Web of Science (10) Citing Articles via Google Scholar Google Scholar Articles by Murray, C. J.L. Articles by Valentine, N. Search for Related Content PubMed PubMed Citation Articles by Murray, C. J.L. Articles by Valentine, N. Related Collections International Issues

Cross-National Comparisons

PERSPECTIVE

People’s Experience Versus People’s Expectations

Comparing Apples With Oranges

 
Two out of three of Blendon’s comparisons seem to be mixing apples with oranges. Satisfaction with the way health care runs in a country is not conceptually comparable with overall health system performance or attainment and only partly comparable to responsiveness. First, WHO defines the health system broadly to include all actors, institutions, and resources that undertake actions whose primary intent is to improve health. This broad definition includes medical care but also includes efforts to decrease tobacco consumption or to reduce highway fatalities through better car design. We suspect that those who responded to the Blendon question were more likely reflecting their satisfaction with personal medical care and not the broader health system.

Second, health system performance as measured by WHO is the efficiency with which health system resources are used to achieve socially valued outcomes. It is a value for-money concept in which the achievements of a health system are compared to what would be possible given the level of spending on the health system and other non–health system factors. The survey question used by Blendon does not ask respondents about efficiency, so why is the comparison meaningful?

Third, even taking into account the difference between the health system and the medical care system, satisfaction with the way health care runs does not seem comparable with the WHO measure of overall health system goal attainment. This measure captures the levels of health, health inequalities, system responsiveness, and who bears the burden of financing the health system. Satisfaction with the way health care runs is unlikely to adequately reflect mortality rates (the respondents are all alive) or inequalities in health and may not even capture individual health status. Perhaps it is most influenced by fairness in financial contribution and responsiveness of the health system. Blendon does compare satisfaction with responsiveness but not with fairness in financial contribution. The remainder of this response focuses on the comparison of system responsiveness and satisfaction with one’s health care system.

The Role Of Expectations

Top Comparing Apples With Oranges The Role Of Expectations WHO's Strategy To Measure... Enhancing Comparability Of... Concluding Comments NOTES

 
Satisfaction with one’s health care system compares a person’s assessment of the health care that is available with his or her expectations for health care. If two individuals differ in their satisfaction, it may be because of differences in their perception of what health care does, or in their expectations for health care, or both. A good illustration of the impact of expectations was provided in a recent survey in five countries.² In the United States, Canada, New Zealand, Australia, and the United Kingdom, 49.5 percent, 40.2 percent, 29.6 percent, 27.3 percent, and 13.3 percent of respondents, respectively, had spent more than fifteen minutes with the doctor at their most recent visit. On the other hand, satisfaction rates for respondents were lowest in the country that reported the most time spent with the doctor. The lowest rate of satisfaction was reported in the United States (75.5 percent), compared with higher percentages in Canada (82.8 percent), the United Kingdom (83 percent), Australia (84.8 percent), and New Zealand (84.8 percent). Satisfaction surveys do not necessarily reflect variations in how people are actually treated by the system.

This is, in fact, reflected in Blendon’s own data. A reanalysis of Exhibits 1 and 2 shows that satisfaction with the health care system is higher among the poor populations than the nonpoor in nine of seventeen countries. For example, satisfaction among the poor in the United Kingdom was 67 percent versus 54 percent among the nonpoor; in the United States, it was 45 percent versus 38 percent.³ Rapidly changing expectations may also explain the substantial variability in the responses over time to questions on satisfaction. For example, in Spain the percentage satisfied was 21 percent in 1991, 61 percent in 1995, and 44 percent in 1996.⁴ It is difficult to imagine that the system changed so dramatically from year to year.

On the other hand, responsiveness as a concept is meant to reflect an individual’s actual experience with the health system in specific domains. In the most recent WHO household survey instrument on responsiveness, we included eight domains: dignity, autonomy, confidentiality, information, prompt attention, access to social support networks, quality of basic amenities, and choice. At the conceptual level, comparisons of responsiveness should be unaffected by differences in expectations. For example, if waiting time for diagnosis and treatment is longer in Country A than in Country B, then, ceteris paribus, the responsiveness of the system is lower in A, regardless of whether people are more or less satisfied with waiting time in A or B. Satisfaction may be important for health service managers and providers, but it is not a meaningful basis for comparisons over time or across countries.

WHO’s Strategy To Measure Responsiveness

Top Comparing Apples With Oranges The Role Of Expectations WHO's Strategy To Measure... Enhancing Comparability Of... Concluding Comments NOTES

 
Blendon and colleagues have misunderstood WHO’s strategy for measuring responsiveness. They claim several times that we propose to measure responsiveness by asking experts and not the public. This is incorrect. Our measurement strategy has been clearly defined and depends primarily on using a standardized survey instrument with demonstrated psychometric properties in nationally representative random sample surveys to measure individuals’ actual experience of health systems in the eight domains.⁵ Because of the near-complete lack of comparable survey data on any domains of responsiveness, WHO, with the collaboration of the U.S. Agency for Healthcare Research and Quality (AHRQ), developed a household survey instrument that was field-tested in the Philippines, Tanzania, and Colombia. Because of the limitations of resources and time, a variant of this instrument was used with key informants to provide some empirical data for the World Health Report 2000. A revised form of this instrument has now been fielded in nationally representative household surveys using face-to-face interviews in forty-one countries and self-administered postal surveys in another thirty. In fact, WHO is working with INRA to use the WHO survey instrument in the Eurobarometer sample frame. These surveys are all scheduled to be completed by 1 May 2001 and will provide a much-improved empirical basis for comparing the responsiveness of different health systems.

Enhancing Comparability Of Survey Results

Top Comparing Apples With Oranges The Role Of Expectations WHO's Strategy To Measure... Enhancing Comparability Of... Concluding Comments NOTES

 
Apart from the conceptual difference between responsiveness of a health system and satisfaction with it, there is an additional reason to challenge the comparison. In using results across countries, Blendon and colleagues are assuming that the response categories to the question such as "fairly satisfied" or "very satisfied" have the same meaning in all populations. There is now an extensive psychometric literature on survey instrument design that highlights the importance of differential item functioning—a technical term for when response categories such as "fairly satisfied" have different meanings to different groups of individuals.⁶ Because language is so often used differently, we believe that it is an essential part of good instrument development to empirically ascertain the meaning of response categories on a particular question in each population.

For this reason, the WHO survey module on responsiveness includes a series of vignettes that can be used to establish the meaning of various response categories. For example, one survey question on responsiveness is "Now, overall, how would you rate your experience of getting prompt attention at the health services in the last 12 months?" and the response categories are very good, good, moderate, bad, and very bad. On a sub-sample, we also provide respondents with a series of vignettes of health system interactions and ask them to rate that experience using the same response categories. For example, "Stan fell down from a ladder and broke his leg one evening. He had to be taken to the district hospital, about 10 miles away (15 km), in a private car. He had to wait for an hour in the hospital for the surgeon to arrive and could be operated [on] only the next day. How would you rate Stan’s experience of getting prompt attention?" The set of responses on the vignettes can then be used to identify and correct for differences in the use of language in different cultural settings.⁷

The same approach is being used in the WHO survey module to measure individual health. Our experience to date with this approach suggests wide variation in the meaning of response categories. Even very concrete questions and response categories such as "some difficulty seeing someone you know across the road" can have very different meanings in different populations. In this case, the findings have also been validated using vision tests. We strongly suspect that cross-national comparisons of satisfaction with the way health care runs are hampered by the lack of comparability of the response categories.

Concluding Comments

Top Comparing Apples With Oranges The Role Of Expectations WHO's Strategy To Measure... Enhancing Comparability Of... Concluding Comments NOTES

 
We emphasize here three points. First, Blendon and colleagues attempt to portray the ongoing work of WHO to measure the performance of the world’s health systems as an exercise in expert judgment that does not capture the experience of the public. In fact, all five of the measures of the goals of health systems are constructed from household survey data of one form or another.

Second, in many cases, good survey data were unavailable for the World Health Report 2000 so that estimates had to be used. Catalyzed by the broad interest generated by health systems performance assessment, future WHO assessments will be informed by the wave of nationally representative random sample surveys under way in many countries and will be reported in 2002.

Third, we believe that responsiveness is a much more meaningful, measurable, and comparable construct than "satisfaction with the way health care runs." The data on satisfaction presented by Blendon and colleagues lead us to conclude that the poor in the United States and the United Kingdom get better health care than the nonpoor. We doubt this is true. Satisfaction measures are profoundly influenced by expectations. Performance assessment should reflect the reality of people’s experiences—in terms of their health, their interactions with the health care system, and the financial burden they bear to pay for that system—not simply their expectations.

Editor's Notes

 
Christopher Murray is executive director ad interim of the Evidence and Information for Policy Cluster at the World Health Organization. Kei Kawabata is the coordinator in the same cluster for Designing Policy Options: Financing, Resource Allocation, Regulation, and Legislation team. Nicole Valentine is a health economist on the team.

NOTES

Top Comparing Apples With Oranges The Role Of Expectations WHO's Strategy To Measure... Enhancing Comparability Of... Concluding Comments NOTES

 

1. A. De Silva and N. Valentine, "A Framework for Measuring Responsiveness," GPE Discussion Paper Series no. 32 (Geneva: World Health Organization, 2000); D.B. Evans et al., "The Comparative Efficiency of National Health Systems in Producing Health," British Medical Journal (forthcoming); C. Darby et al., "Strategies to Measure Responsiveness," GPE Discussion Paper Series no. 23 (Geneva: WHO, 2000); and R. Sadana et al., "Comparative Analyses of More than 50 Household Surveys on Health Status," GPE Discussion Paper Series no. 15 (Geneva: WHO, 2000).
2. K. Donelan et al., "The Cost of Health System Change: Public Discontent in Five Nations," Health Affairs (May/June 1999): 206–216.
3. We calculated these percentages using the total percentage satisfied and the percentage of the poor satisfied, as reported by Blendon and colleagues.
4. R.J. Blendon et al., "Spain’s Citizens Assess Their Health Care System," Health Affairs (Fall 1991): 216–228; and E. Mossialos, "Citizens’ Views on Health Care Systems in the Fifteen Member States of the European Union," Health Economics (March–April 1997): 109–116.
5. Darby et al., "Strategies to Measure Responsiveness."
6. W. Groot, "Adaptation and Scale of Reference Bias in Self-Assessments of Quality of Life," Journal of Health Economics 19, no. 3 (2000): 403–420; J.B. Bjoner, S. Kreiner, and J.E. Ware, "Differential Item Functioning in the Danish Translation of the SF-36," Journal of Clinical Epidemiology 51, no. 11 (1998): 1189–1202; and R.D. Hays, L.S. Morales, and S.P. Reise, "Item Response Theory and Health Outcomes Measurement in the Twenty-first Century," Medical Care (September 2000): ii28–ii42.
7. C.J.L. Murray et al., "Enhancing Cross-Population Comparability of Survey Results," GPE Discussion Paper Series no. 35 (Geneva: WHO, 2000).

                      What's this?

This article has been cited by other articles:

				S. L. Appleton-Knapp and K. A. Krentler Measuring Student Expectations and Their Effects on Satisfaction: The Importance of Managing Student Expectations Journal of Marketing Education, December 1, 2006; 28(3): 254 - 264. [Abstract] [PDF]

				P. J. Devereux and B. A. Weisbrod Does "Satisfaction" with Local Public Services Affect Complaints (Voice) and Geographic Mobility (Exit)? Public Finance Review, March 1, 2006; 34(2): 123 - 147. [Abstract] [PDF]

				M. S. Garfinkel, D. Sarewitz, and A. L. Porter A Societal Outcomes Map for Health Research and Policy Am J Public Health, March 1, 2006; 96(3): 441 - 446. [Abstract] [Full Text] [PDF]

				Task Force 1 Writing Group, L. A. Green, R. Graham, B. Bagley, C. M. Kilo, S. J. Spann, S. P. Bogdewic, and J. Swanson Task Force 1. Report of the Task Force on Patient Expectations, Core Values, Reintegration, and the New Model of Family Medicine Ann. Fam. Med, March 1, 2004; 2(suppl_1): S33 - S50. [Abstract] [Full Text] [PDF]

				J. Koffman, I. J Higginson, and N. Donaldson Symptom severity in advanced cancer, assessed in two ethnic groups by interviews with bereaved family members and friends J R Soc Med, January 1, 2003; 96(1): 10 - 16. [Abstract] [Full Text] [PDF]

				J. S. Coyne and P. Hilsenrath The World Health Report 2000: Can Health Care Systems Be Compared Using a Single Measure of Performance? Am J Public Health, January 1, 2002; 92(1): 30 - 33. [Full Text] [PDF]