Health Affairs, 23, no. 3 (2004): 247-256 doi: 10.1377/hlthaff.23.3.247 © 2004 by Project HOPE	New Online   Obama's Success & Next Steps   3 Health Policy Options for President Obama   A Hybrid Proposal   Bipartisan Compromise

This Article Abstract Figures Only Reprint (PDF) Submit a response to this article Alert me when this article is cited Alert me when Comments are posted Alert me if a correction is posted Services E-mail this article to a friend Similar articles in this journal Similar articles in ISI Web of Science Similar articles in PubMed Alert me to new issues of the journal Add to My Personal Archive Download to Citation Manager Reprints & Permissions Citing Articles Citing Articles via ISI Web of Science (32) Citing Articles via Google Scholar Google Scholar Articles by Kerr, E. A. Articles by Asch, S. M. Search for Related Content PubMed PubMed Citation Articles by Kerr, E. A. Articles by Asch, S. M. Related Collections Chronic Care Public Opinion Quality Of Care

DataWatch

Profiling The Quality Of Care In Twelve Communities: Results From The CQI Study

Abstract

 
Health care quality falls far short of its potential nationally. Because care is delivered locally, improvement strategies should be tailored to community needs. This analysis from the Community Quality Index (CQI) study reports on a comprehensive examination of how effectively care is delivered in twelve metropolitan areas. We find room for improvement in quality overall and in dimensions of preventive, acute, and chronic care in all of these communities; no community was consistently best or worst on the various dimensions. Having concrete estimates of the extent of the gap in performance should stimulate community-based quality improvement efforts.

For redesign to occur, information on quality should be presented at a level where action can be taken. Thus, reports on hospital and health plan performance may stimulate actions within organizations. But these reports, while important, might not affect care for the majority of the population. For example, although Health Plan Employer Data and Information Set (HEDIS) reports cover approximately 80 percent of people enrolled in health maintenance organizations (HMOs), only 28 percent of Americans belong to an HMO.⁴ Steven Jencks and colleagues recently published a report on quality of care for Medicare beneficiaries at the state level.⁵ However, the work of John Wennberg and others suggests that there is likely to be considerable variation within these states.⁶ Having information specific to a local market is critical for making choices about how to allocate public and private resources most effectively. Communities can act to improve health and quality of care in areas such as immunizations, screening for and prevention of chronic disease through education, housing, employer-based efforts, collaborations with academic institutions and health centers, and public safety initiatives.⁷ No comprehensive, community-based assessments of health care quality have been reported to date.

Here we report on a comprehensive and clinically detailed examination of the quality of care in twelve metropolitan areas, based on analyses from the Community Quality Index (CQI) study. We focus specifically on the effectiveness of care—that is, the extent to which care is delivered consistent with science-based knowledge about preventing death and disability. This is one of the six domains of system performance recommended for evaluation by the IOM.⁸

Data And Methods

Top Editor's Notes Data And Methods Findings Discussion NOTES

 
The CQI is a collateral study of the Community Tracking Study (CTS), conducted by the Center for Studying Health System Change (HSC), which uses periodic surveys and site visits to track changes in the health care system in sixty metropolitan areas that, along with a national sample, are representative of the United States.⁹ In the twelve communities we studied, the CTS collects enough information to characterize those health care markets. The CQI uses the RAND Quality Assessment (QA) Tools system to assess quality of care in these communities, evaluating all inpatient and outpatient care received by adults over a two-year period for thirty acute and chronic conditions as well as preventive care.¹⁰

Participants. The twelve communities in the CQI are representative of metropolitan areas with more than 200,000 population (Boston, Cleveland, Greenville [South Carolina], Indianapolis, Lansing, Little Rock, Miami, Newark, Orange County [California], Phoenix, Seattle, and Syracuse). The structure of health care markets in these areas varies in ways that have been associated with differences in the use of health care services (Exhibit 1).¹¹

Because of the complex, multistage nature of the design, several response-rate calculations are provided. Among the 20,028 adults in the starting sample, 2,091 (10 percent) were ineligible primarily because they had left the area. Among the 17,937 remaining, 13,275 (74 percent) participated in the telephone health history, including 863 (7 percent) who had no provider visits in the previous two years. Among the 12,412 participants with visits, 10,404 (84 percent) agreed orally to provide access to their medical records. We obtained written consent from 7,528 (61 percent of those with provider visits). We received at least one record for 6,712 (89 percent) of those returning consent forms. These people are included in the results reported here (37 percent of the eligible sample). On average, these respondents were 45.5 years old and had 13.7 years of education; 60 percent were women; and 81 percent were white.

RAND’s QA Tools. QA Tools for adults contains 439 indicators across thirty conditions and preventive care. The indicators were developed by RAND staff and validated by four multispecialty expert panels, using the RAND/UCLA modified Delphi method.¹² The conditions selected represent 52 percent of the reasons adults use ambulatory care and 46 percent of the reasons adults are hospitalized.¹³

Data collection. To improve the efficiency and reliability of data collection, we developed computer-assisted abstraction software on a Microsoft Visual Basic 6.0 platform. We abstracted charts separately for each provider of each patient. Average inter-rater reliability (kappa statistic) was substantial to almost perfect at three levels: patients’ conditions (kappa = 0.83), eligibility for indicators (kappa = 0.76), and passing or failing indicators (kappa = 0.80).¹⁴

Creating the scores. For each indicator of quality, we specified the combination of variables necessary to determine whether each respondent was eligible for (yes/no) and received (yes/no or proportion) the recommended care. Next we created aggregate scores in four categories: (1) overall quality of care; (2) type of care (preventive, acute, chronic); (3) dimensions of preventive care (preventing sexually transmitted diseases [STDs] and HIV, adult immunizations, substance abuse counseling, cancer screening, and preventing chronic disease through screening [for example, blood pressure and weight measurement]); and (4) care for selected chronic conditions (hypertension, diabetes, cardiac, pulmonary, and depression).

To produce aggregate scores, we divided all instances in which recommended care was delivered by the number of times recommended care was required (a combination of the number of people eligible and the number of indicators in a category). The aggregate scores can be interpreted as the proportion of recommended care that is delivered. Using data from the CTS, we adjusted the scores for nonresponse, weighting participants to be representative of the population from which they were drawn. Quality of care may be influenced by the demographic characteristics of a community as well as structural community factors. To remove the effect of community demographics on quality, we standardized the communities by reweighting our respondents to a common reference population with distributions for age, sex, education, and race similar to the overall U.S. population.

Findings

Top Editor's Notes Data And Methods Findings Discussion NOTES

 
Overall quality and quality by type of care. We found similar levels of performance overall across the twelve communities, and all scores reflected important deficits in the provision of basic care. Overall quality ranged from 51 percent in Little Rock to 59 percent in Seattle (Exhibit 2). Overall performance in Seattle was statistically significantly higher than in Greenville, Little Rock, Newark, and Orange County (p < .001). We found that the quality of preventive, acute, and chronic care was similar in each of the communities. On average, residents were receiving 50 to 60 percent of recommended care.

View larger version (25K): [in this window] [in a new window]   EXHIBIT 2 Quality Deficits Found In All Types Of Care And In All Twelve Community Quality Index Study Communities

View larger version (33K): [in this window] [in a new window]   EXHIBIT 3 Variations In Quality Among Twelve Community Quality Index Study Communities, By Dimension Of Preventive Care

View larger version (28K): [in this window] [in a new window]   EXHIBIT 4 Variations In Quality Among Twelve Community Quality Index Study Communities, By Chronic Condition

Top Editor's Notes Data And Methods Findings Discussion NOTES

Some readers may be surprised to learn that performance was not better in areas with outstanding medical institutions. Our analysis examined average care for adults from an entire metropolitan area, rather than care received from a specific facility, health care system, or doctor. Perceptions of quality in some cities may be driven by beliefs about the performance of benchmark doctors and institutions. However, the experience of a broad cross-section of patients in those communities may be more variable. Similarly, even though performance on diabetes quality measures has been improving in some health plans, as documented by HEDIS reporting, we found large quality deficits at the community level for diabetes care.¹⁵ When we take into account the fact that there are wide variations in performance across health plans and that most community members are not in plans that participate in HEDIS reporting, our observation of poor community performance is less surprising. Unfortunately, our data do not allow us to assess care at the provider or system levels. The dearth of readily available, detailed clinical information makes such investigations both expensive and time-consuming. The release of community-level data may stimulate demand for such information.

Universally poor performance should not diminish enthusiasm for quality improvement at the community level. Health care is delivered locally, with regional health plans, medical groups, and public health clinics often sharing patients, either now or in the future. Therefore, the lack of variation in overall quality should serve as a wake-up call to all communities to examine their own quality of care and determine how they might be able to approach local quality improvement initiatives. As part of this examination, communities can also choose to come together and tackle some problems at the regional or state level. For example, in Minnesota forty-three medical groups and hospitals are collaborating to accelerate the adoption of best clinical practices, resulting in uniform practice guidelines for all six Minnesota health plans.¹⁶ In New York State, monitoring and public reporting of mortality rates after coronary artery bypass graft (CABG) surgery contributed to a decline in operative mortality.¹⁷

Further, our analysis of dimensions of preventive and chronic disease care demonstrated variability at the community level and could help communities articulate the need for quality improvement efforts. The availability of a concrete estimate of the size of the problem may be more compelling than a general belief that a problem exists. Baseline information also facilitates assessments of the effects of interventions to improve care.

Some of the variation in delivery of preventive care may reflect differences in the reimbursement rules for these services. However, these results may also reflect the success of community-based interventions for immunizations. In one community, for example, the county health department developed an active outreach program for public employees, inviting retirees to return to their former place of work for influenza and pneumoccocal vaccines.¹⁸ Such programs could be established by local health insurers and public health departments for other services. For example, community-based smoking cessation programs could target all neighborhood residents, regardless of insurance status. Screening and counseling programs for HIV, tuberculosis, and other infectious diseases could be offered by public and community health centers.

Because quality in most areas of care was uniformly poor, we cannot draw definite conclusions about the effects of structural and financial characteristics on quality. One would need to study a larger number of communities (probably more than 100) to draw specific conclusions about the relationship between market structure and quality. Further, the financial characteristics presented in Exhibit 1, while important in determining access to care, may not be critical components once access is achieved. This finding is consistent with previous studies that have found no relationship between variations in the rates of services used (such as procedures and hospital admissions) and the clinical appropriateness of those services, and it also underscores the IOM’s conclusion that fundamental system resdesign will be necessary.¹⁹

Nonetheless, the design of quality improvement programs must take the needs of the population into account. For example, in Miami, where 23 percent of people under age sixty-five are uninsured and 25 percent are living below poverty, programs will need to have a large outreach component and public funding. Conversely, programs in the Boston area, with only 8 percent uninsured and 10 percent living below poverty, may rely more on the private health care system.

Study limitations. Because our analytic sample included 37 percent of eligible adults, the results are likely to be biased, but the direction of that bias is not clear. In particular, our participants were limited to those who had at least one visit to a health care provider in the previous two years, which may lead us to underestimate quality deficits related to underuse.

The study relied primarily on medical records to evaluate quality, which may lead some to question whether the results reflect only problems with documentation. This issue has been examined in a few studies that conclude that medical record–based quality scores may be five to ten percentage points lower than scores based on vignettes or patient report.²⁰ These same studies find some over-reporting by patients and in medical records. We used the health history interview to offset some of this effect; in general, including self-reported data improved scores.

Concluding comments. Although each community will need to grapple with how to improve quality of care locally, it is clear that considerable room exists for improvement in health care delivery at the community level. So how will this "quality chasm" be bridged? The answers, as suggested by community-based quality initiatives, do not lie entirely within the medical care system.²¹ Large employers could provide leadership to improve chronic disease care in outpatient settings as they have done to reduce medical errors in hospitals.²² Health care systems, with or without financial incentives, could initiate quality improvement initiatives in collaboration with the community, to improve provision of preventive care.²³ Community-based education and outreach efforts could activate patients to demand improved quality across many dimensions and to identify their own care requirements.²⁴ For example, several organizations have developed guides to help patients assess whether they are getting needed care for selected health problems.²⁵

There are many laudable community coalitions and initiatives throughout the country, but few have meaningful baseline or postintervention data on the processes and outcomes of health care to be able to answer whether or not their initiatives improve quality of care.²⁶ The data that are available are generally not collected using standardized methods that allow system-to-system or provider-to-provider comparisons to be made. Our study shows that the processes of health care can be measured at the community level. While our study cannot explore all of the reasons for poor care, the approach we used to assess quality (the QA Tools system) could be applied to assessing other communities or health delivery systems. Advances in the routine availability of detailed clinical data would improve such efforts.

Those who believe that there is no room for improvement in the quality of care delivery in their markets should test those beliefs. Objective assessments of quality are essential for identifying priorities for action, galvanizing support for such interventions, and allowing others to benefit from assessing the effectiveness of these interventions.

Editor's Notes

Top Editor's Notes Data And Methods Findings Discussion NOTES

 
Eve Kerr (ekerr{at}umich.edu) is a research scientist, Ann Arbor Veterans Affairs (VA) Center for Practice Management and Outcomes Research; and assistant professor, Department of Internal Medicine, University of Michigan. Elizabeth McGlynn is associate director, RAND Health, Santa Monica, California. John Adams is senior statistician at RAND; Joan Keesey is senior programmer there. Steven Asch is associate director, HSRPD Center of Excellence, VA Greater Los Angeles Healthcare System; a natural scientist, RAND Health; and associate professor of medicine, David Geffen School of Medicine, University of California, Los Angeles.

This study was funded by the Robert Wood Johnson Foundation (Maureen Michael, James Knickman, Robert Hughes). Eve Kerr and Steven Asch were supported by Veterans Affairs Health Services Research and Development career development awards. We are grateful for the collaboration of Paul Ginsburg at the Center for Studying Health System Change and Richard Strauss at Mathematica Policy Research. RAND’s Survey Research Group (Josephine Levy, Laural Hill) recruited participants and obtained telephone interview data. The design of the data collection tool, hiring and training nurse abstractors, and overseeing the data collection process was undertaken by several RAND research nurses (Peggy Wallace, Karen Ricci, Belle Griffin). Liisa Hiatt was the project manager; analytic support was provided by Jennifer Hicks, Alison DeCristofaro, and David Klein. The data collection software was developed by Vector Research Inc. (Kevin Dombkowski).

NOTES

Top Editor's Notes Data And Methods Findings Discussion NOTES

 

1. E.A. McGlynn et al., "The Quality of Health Care Delivered to Adults in the United States," New England Journal of Medicine 348, no. 26 (2003): 2635–2645.[Abstract/Free Full Text]
2. See, for example, M.A. Schuster, E.A. McGlynn, and R.H. Brook, "How Good Is the Quality of Health Care in the United States?" Milbank Quarterly 76, no. 4 (1998): 517–563; E.A. McGlynn and R.H. Brook, "Keeping Quality on the Policy Agenda," Health Affairs 20, no. 3 (2001): 82–90; [Abstract/Free Full Text]Institute of Medicine, Crossing the Quality Chasm: A New Health System for the Twenty-first Century (Washington: National Academies Press, 2001); and M.R. Chassin and R.W. Galvin, "The Urgent Need to Improve Health Care Quality: Institute of Medicine National Roundtable on Health Care Quality," Journal of the American Medical Association 280, no. 11 (1998): 1000–1005.[Abstract/Free Full Text]
3. IOM, Crossing the Quality Chasm.
4. U.S. Centers for Disease Control and Prevention, "Health Maintenance Organizations (HMOs) and Enrollment, According to Model Type, Geographic Region, and Federal Program: United States, Selected Years 1976–2001," www.cdc.gov/nchs/data/hus/tables/2002/02hus132.pdf (12 February 2004); and National Committee for Quality Assurance, The State Of Health Care Quality 2003, www.ncqa.org/Communications/State%20Of%20Managed%20Care/SOHCREPORT2003.pdf (12 February 2004).
5. S.F. Jencks, E.D. Huff, and T. Cuerdon, "Change in the Quality of Care Delivered to Medicare Beneficiaries, 1998–1999 to 2000–2001," Journal of the American Medical Association 289, no. 3 (2003): 305–312.[Abstract/Free Full Text]
6. Center for the Evaluative Clinical Sciences, Dartmouth Medical School, Dartmouth Atlas of Health Care, www.dartmouthatlas.org (12 February 2004).
7. J. Flower, "Healthier Communities: A Compendium of Best Practices," www.well.com/user/bbear/hc_compendium.html (8 May 2003); D.M. Levine et al., "A Partnership with Minority Populations: A Community Model of Effectiveness Research," Ethnic Disparities 2, no. 3 (1992): 296–305; and D.M. Levine, D.M. Becker, and L.R. Bone, "Narrowing the Gap in Health Status of Minority Populations: A Community-Academic Medical Center Partnership," American Journal of Preventative Medicine 8, no. 5 (1995): 319–323.
8. IOM, Crossing the Quality Chasm.
9. For more details on the Community Tracking Study, see P. Kemper et al., "The Design of the Community Tracking Study: A Longitudinal Study of Health System Change and Its Effects on People," Inquiry 33, no. 2 (1996): 195–206; [ISI][Medline] E. Schaefer and J.D. Reschovsky, "Are HMO Enrollees Healthier than Others? Results from the Community Tracking Study," Health Affairs 21, no. 3 (2002): 249–258; [Abstract/Free Full Text] D.A. Draper et al., "The Changing Face of Managed Care," Health Affairs 21, no. 2 (2002): 11–23; and C.S. Lesser and P.B. Ginsburg, "Update on the Nation’s Health Care System: 1997–1999," Health Affairs 19, no. 6 (2000): 206–216.[Abstract]
10. E.A.McGlynn et al., Quality of Care for Women: A Review of Selected Clinical Conditions and Quality Indicators (Santa Monica, Calif.: RAND, 2000); S. Asch et al., Quality of Care for Oncologic Conditions and HIV: A Review of Selected the Literature and Quality Indicators (Santa Monica, Calif.: RAND, 2002); E.A. Kerret al., Quality of Care for Cardiopulmonary Conditions: A Review of the Literature and Quality Indicators (Santa Monica, Calif.: RAND, 2000); E.A. Kerr et al., Quality of Care for General Medical Conditions: A Review of the Literature and Quality Indicators (Santa Monica, Calif.: RAND, 2000); and E.A. McGlynn et al., "The Quality of Health Care Delivered to Adults in the United States: Technical Appendix," www.rand.org/health/mcglynn_appa.pdf (12 February 2004).
11. Center for Studying Health System Change, "HSC Community Reports for 2001," 3 February 2004, www.hschange.org/index.cgi?func=pubs&what=5&page=Community2 (12 February 2004).
12. R.H.Brook, "The RAND/UCLA Appropriateness Method," in Clinical Practice Guideline Development: Methodology Perspectives, ed. K.A. McCormick, S.R. Moore, and R.A. Siegel (Rockville, Md.: U.S. Public Health Service, 1994), 59–70.
13. Kerr et al., Quality of Care for General Medical Conditions.
14. J.R. Landis and G.G. Koch, "The Measurement of Observer Agreement for Categorical Data," Biometrics 33, no. 1 (1977): 159–174.[CrossRef][ISI][Medline]
15. NCQA, The State of Health Care Quality 2003.
16. Ibid.
17. M. Chassin, E. Hannan, and B. DeBuono, "Benefits and Hazards of Reporting Medical Outcomes Publicly," New England Journal of Medicine 334, no. 6 (1996): 394–398.[Free Full Text]
18. JanetOlszewski, Michigan Department of Community Health, personal communication, 3 January 2002.
19. A. Siu et al., "Inappropriate Use of Hospitals in a Randomized Trial of Health Insurance Plans," New England Journal of Medicine 315, no. 20 (1986): 1259–1266;[Abstract] and E. McGlynn et al., "A Comparison of the Appropriateness of Coronary Angiography and Coronary Artery Bypass Graft Surgery between Canada and New York State," Journal of the American Medical Association 272, no. 12 (1994): 934–940.[Abstract]
20. J. Luck and J.W. Peabody, "Using Standardized Patients to Measure Physicians’ Practice: Validation Study Using Audio Recordings," British Medical Journal 325, no. 7366 (2002): 679;[Abstract/Free Full Text] and J.W. Peabody et al., "Comparison of Vignettes, Standardized Patients, and Chart Abstraction," Journal of the American Medical Association 283, no. 13 (2000): 1715–1722.[Abstract/Free Full Text]
21. Coalition for Healthier Cities and Communities, "About the Coalition," www.hospitalconnect.com/healthycommunities/about_what.html (8 May 2003).
22. Leapfrog Group, "The Leapfrog Group," www.leapfroggroup.org (8 May 2003).
23. Kaleida Health, "Community Health and Wellness: A Brief Inventory of Community Collaborations," www.kaleidahealth.org/community/collaborations.html (12 February 2004).
24. Robert Wood Johnson Foundation, "Patient Education and Consumer Activation in Chronic Disease," July 2002, www.rwjf.org/publications/publicationsPdfs/patient_education.pdf (8 May 2003).
25. See, for example, Pfizer Corporation, "Health Resources: Older Adult Health," www.pfizer.com/do/mn_health_older1.html (12 February 2004); Agency for Healthcare Research and Quality, "Improving Health Care Quality: A Guide for Patients and Families," www.ahrq.gov/consumer/qntlite (8 May 2003); and American Diabetes Association, "Take Care of Yourself, Take Care of Your Diabetes," www.diabetes.org/main/uedocuments/ADA-EngWeb.pdf (8 May 2003).
26. Flower, "Healthier Communities."

                      What's this?