|
P U B L I C H E A L T H
P R E P A R E D N E S S
W E B E X C L U S I V E
2 June 2004
Local Variation In Public Health Preparedness:
Lessons From California
Even in California—one of
the best-prepared states—much work remains
to ensure preparedness for a public health emergency.
By Nicole Lurie, Jeffrey Wasserman,
Michael Stoto, Sarah Myers, Poki Namkung,
Jonathan Fielding, and Robert Burciaga Valdez
ABSTRACT:
Since September 2001 Congress has allocated approximately $3 billion
to strengthen the public health infrastructure. To achieve this goal, the U.S.
Centers for Disease Control and Prevention (CDC) allocates funding to states,
which distribute funds to local jurisdictions. Evidence-based measures to assess
public health preparedness are lacking. We used an expert-panel process to develop
performance measures, based on the ten essential public health services. We
developed and conducted tabletop exercises in California to evaluate preparedness
to detect and respond to a hypothetical smallpox outbreak based on those measures.
There was wide variation of readiness in California. While the sources of variation
are often different, common infrastructure gaps need to be addressed.
Since the 1988 institute of medicine report, The Future of Public Health,
the “disarray” of the U.S. public health system has been broadly acknowledged.1
The impressive achievements of public health during the past century and their
accompanying improvements in longevity created a sense of complacency about
the underlying public health infrastructure, which has deteriorated markedly
during the past twenty-five years. The U.S. Centers for Disease Control and
Prevention (CDC) declared in 2001 that the U.S. public health infrastructure
remains “structurally weak in nearly every area.”2
Not until the 11 September 2001 terrorist attacks and the subsequent anthrax
attacks were any large-scale investments made in the public health infrastructure.
Since then Congress has allocated approximately $3 billion over a three-year
period to strengthen the infrastructure, largely through programs administered
by the CDC that are focused on public health agencies.3
A companion program, administered by the Health Resources and Services Administration
(HRSA), addresses hospitals’ preparedness. Although these programs were
intended to improve public health preparedness, especially the ability to detect
and respond to a bioterrorist attack, an additional goal was to improve public
health systems’ ability to address other threats to health more generally,
particularly those related to infectious diseases. The CDC’s Bioterrorism
Supplemental Funding Cooperative Agreement was developed to “ensure that
all state and local public health organizations have a strong infrastructure
and are prepared to respond to bioterrorism, outbreaks of infectious diseases,
and other public health threats and emergencies through comprehensive planning,
training, and evaluation.”4 Funds from this
program are allocated to states, which in turn are responsible for passing funds
along to local jurisdictions.
In the first two years of this investment, in response to a set of mandates
(critical benchmarks) from the CDC that are intended to strengthen public health
preparedness, state and local public health agencies have developed preparedness
plans. Assessing their progress has been hampered by several factors, including
the different ways that public health is organized. There are approximately
3,000 local health departments, with varying size, structure, relationship to
their state agencies, scope of responsibility, and constituent population.
Although there is broad agreement about some functional responsibilities of
public health, there are widespread differences in the specific roles played
by individual state and local public health departments. Nearly all would agree
that public health agencies should be prepared to deal with a large-scale infectious
disease outbreak or a bioterrorist event. However, there is little agreement
regarding key elements of preparedness or how to measure them. Many elements
in previously proposed measures are not easily measurable, and the evidence
base underpinning most of the elements is weak.
The absence of well-established measures of performance for public health systems
is not unique to preparedness. Although limited performance measurement for
local public health departments dates back to the 1940s, it has not been as
widely accepted in public health as in personal health.5
The field of measurement in the quality of medical care has been growing since
the 1960s; despite its shortcomings, it is still more firmly established than
performance measurement in public health.6
The Trust for America’s Health (TFAH), a not-for-profit public health advocacy
group, used an expert-panel process to identify an initial set of key indicators
with which to judge the preparedness of state public health departments. TFAH
recently published its first statewide rankings based on these indicators and
found wide state-to-state variation in the status of the public health infrastructure.7
California was one of four states in these rankings’ top tier.
In late 2002 RAND was asked to identify gaps in California’s public health
infrastructure, focusing first on public health preparedness against infectious
disease outbreaks. Because there are no standards for what the public health
infrastructure should look like or established methods for its assessment, we
developed and applied a combination of methods to examine public health preparedness
in California. We chose the framework of the ten essential public health services
(EPHS) for our work because these were developed by the public health community
(Public Health Functions Steering Committee, 1994) and endorsed in the recent
IOM report, The Future of the Public’s Health in the Twenty-first Century.8
This framework does not specify programmatic activities; rather, a given public
health program is likely to have activities associated with each EPHS. In this
paper we briefly describe how we tried to measure performance on each of the
essential public health services and the methods we developed and used to assess
local public health agencies’ preparedness. We also report our key findings.
Study Methods
Identifying an interim set
of measures. The
study protocol was approved by RAND’s Human Subjects Protection Committee.
First, we reviewed existing measures of public health preparedness. This review
documented both the lack of consensus regarding how to measure public health
preparedness and the lack of evidence to support most of the performance measures.9
We identified the measures common to more than one instrument and added to them
the Critical Benchmarks for preparedness developed by the CDC and HRSA. We organized
these measures according to the EPHS to which they were most related. We used
an expert-panel process to assess the importance and feasibility of measurement
for each item. The nine-member panel comprised content experts in public health
measurement, organization and finance, community relationships, media, occupational
health, and emergency response, and it included representatives from the medical
and public health practice communities. Panelists rated both importance and
feasibility of measurement on a 1–10 scale, with 10 being the most important.
We eliminated measures scoring 7 or below on importance and retained measures
scoring 5 or greater for feasibility, because these cutoffs represented natural
break points in the scoring for each component. We used these as an “interim”
set of measures, designated as such because widespread agreement about a final
set of measures will require a more robust evidence base. These measures helped
guide the site visits and exercises described below. Representative measures
appear in the first column of Exhibit
1.
Study sample.
We focused our assessments on the local level.10
We requested participation from eight jurisdictions covering the spectrum along
several dimensions: geography (urban, rural, and border status), population
size (large, medium, and small), demographics (especially ethnicity and socioeconomic
status), and organizational status (independent, contracts public health functions
back to the state, and receives funding directly from the CDC). Seven agreed
to participate. Together these seven jurisdictions contained 39 percent of the
state population.
Site visits.
We conducted two-day site visits in each jurisdiction. On the first day we interviewed
key informants, including the public health director; health officer; bioterrorism
coordinator (if one existed); director of public health nursing; senior epidemiologist
(if one existed); fiscal manager; local political official; representatives
of the clinical community; as well as senior officials from police, fire, and
emergency medical services (EMS) departments. These interviews provided information
about how the system was organized and financed, preparedness plans and their
progress to date, and the challenges involved in developing a public health
preparedness plan amid other health department responsibilities. Interviews
were semistructured using a standardized protocol, and interviewees were assured
confidential handling of their responses.
Exercises.
We developed and pretested tabletop exercises that were specifically designed
to assess how jurisdictions were able to fulfill the EPHS, focusing on the measures
endorsed by our expert panel. We conducted the exercises on the second day of
the site visit.
In each jurisdiction, exercises were facilitated by two people, who followed
a semistructured protocol. A third person took detailed notes. Public health
directors in jurisdictions were asked to participate in the exercises and to
invite other participants, based on the nature of their communities. They received
guidelines regarding the types of people to be invited (health officer; bioterrorism
coordinator; public health nurse; lab director; representatives from local hospitals,
doctors, and local disaster relief agencies; minority-serving community organizations;
elected local officials; school administrators; and representatives from fire,
police, and EMS), but individual invitations were left to their discretion.
Participants were assured that neither they nor their jurisdictions would be
individually identified without permission.
The exercises consisted of three steps. In the first step, participants were
confronted by three unrelated case reports that might be consistent with smallpox,
in the context of a heightened national terrorism alert. They were asked how
they would respond to the case reports, begin an investigation, collect and
transport biologic samples from patients to the appropriate laboratories, involve
other partners as they deemed necessary, activate an emergency operations center,
and communicate with the public and political authorities. In the second step,
participants were confronted with confirmed cases, increasing numbers of sick
people, up to 2,000 exposed people, and a panicked population. They were asked
about isolation, quarantine and its enforcement, legal authority to act, plans
for vaccination, delivering care to those in need, crowd control, and ongoing
public communication. In each step, after participants responded to the specific
problems posed by the scenario, they were asked to discuss how the scenario
related to their infrastructure more broadly.
In step three, participants were asked to assess (on a 1–10 scale) the
group’s response to the exercise events and the state of their current
public health infrastructure, using questions that addressed the EPHS. They
were then asked to rate “where they are” and “where they want
to be” for each dimension. The difference, or gap, between exercise and
desired performance was used as a springboard for discussion about what resources
would be required to fill the gap and the obstacles to filling it. The director
of the health department received a written after-action report that documented
the group’s responses during the exercise and included an assessment of
strengths and areas for improvement.
Analysis.
At the completion of all of the exercises, three members of the study team reviewed
each after-action report and rated the level of preparedness of each jurisdiction
on a 1–10 scale for each EPHS, using the interim set of performance measures
as a guide. Rater agreement was high; raters never disagreed with one another
by more than one point for any EPHS. Raters then identified which of the interim
performance measures seemed most relevant to assessing performance, and a “short
set” of indicators was developed. Two additional raters then reviewed the
after-action reports again and scored them using this short set. They again
agreed within one point on all but one EPHS; the disagreement was related to
a measure about syndromic surveillance and was not directly used in evaluating
performance.
Results
Site visits yielded information about the organization of public health activities
in the jurisdiction and how they are financed, and they highlighted areas for
further exploration that came up during the exercises. Conversely, exercises
yielded information that was not apparent during the site visits, particularly
with regard to how the system and its key participants would respond to a specific
threat. Both exercise performance and information gleaned from site visits were
used in our assessments of preparedness.
Variations.
We found that each jurisdiction has done considerable planning since receipt
of the CDC and HRSA grants; interviewees and exercise participants could point
to progress made in increasing preparedness or in strengthening particular aspects
of their infrastructure, or both. Despite this, the exercises revealed wide
variation in the level of preparedness. Raters judged two of the seven counties
to be well prepared to respond to a scenario of the magnitude described in the
exercise and one to be particularly poorly prepared. Several factors seemed
to account for the high ratings in two counties. Both had strong leadership
and had worked to develop the leadership potential of others in their departments,
including incident-command training. Both had confidence and experience in communicating
with the public and the media and in working with the law enforcement community,
and both had participated in multiple exercises over the past several years.
Neither size nor urban-rural status clearly differentiated one jurisdiction’s
performance from that of another, although larger counties clearly have more
flexibility and personnel to commit to preparedness activities. The sample size
is too small to draw other conclusions about which attributes are associated
with better preparedness. Columns 2 and 3 of Exhibit
1 provide examples of local public health agencies’ responses on a
sample of performance measures associated with each EPHS.
With regard to monitoring health status in a community (EPHS 1), we were struck
by the general lack of recent community health assessment and relatively incomplete
information about the distribution and demographics of potentially vulnerable
or underserved populations. In some jurisdictions, representatives from police
and fire departments appeared to have better knowledge of vulnerable populations
than the health department had; in some, these entities also had stronger relationships
with community leaders from various racial/ethnic groups. Various efforts are
under way to improve disease surveillance in most jurisdictions, but most were
limited to increased outreach to health care providers. No jurisdiction has
implemented a comprehensive surveillance system. As revealed in the exercises,
the jurisdictions varied in how long it would take for three suspicious cases
to come to the attention of public health officers and for their health departments
to realize that the cases were related.
Health departments varied dramatically in their ability to rapidly alert the
physician and hospital community to a potential outbreak (EPHS 2). There was
much variation in modalities for beginning active surveillance, with methods
ranging from blast fax and e-mail for reaching local hospitals quickly to relying
on informal relationships between the public health director or health officer
and hospital medical leadership to facilitate communication. Only one jurisdiction
had the ability to rapidly contact most practicing physicians outside of a hospital
context. Another jurisdiction seemed fundamentally uncertain about the basics
of beginning an investigation.
In the area of communication and education (EPHS 3), jurisdictions were split
regarding when they would first communicate with the public about a potential
outbreak. Some would notify the public as soon as they began to investigate
a suspicious case; others would wait until a diagnosis was confirmed (days later)
to hold a press conference. One health department can communicate health information
in nine languages, while another is not prepared to communicate in any language
except English.
There was wide variation in understanding of public health legal authority,
especially with regard to quarantine and its enforcement. Many health departments
reported that they did not have access to legal counsel with public health knowledge
(EPHS 6).
Specific to smallpox, there was considerable variation in the fundamental approach
to beginning an epidemiological investigation, beliefs about whom to vaccinate
and when, vaccine efficacy, whether epidemiologists or vaccinators had to wait
until their vaccination “took” before they could contact potential
cases, and whether healthy people should stay home or go about normal activities
once one or more smallpox cases were confirmed in the community. There was wide
variation in understanding which lab could perform which type of test to diagnose
smallpox. There was similar variation in knowledge about where to locate information
that could be used to communicate about smallpox to the public and the medical
practice community, and there were large differences of opinion regarding when,
either legally or practically, the jurisdiction would hand over responsibility
for managing the public health crisis to the state health department or the
CDC.
Similarities.
There were also some similarities across the jurisdictions we studied. The implications
of the current nurse shortage are substantial when it came to staffing in a
public health emergency. Participants noted that many public health nurses also
worked at one or more local hospitals or nursing homes and could only be in
one place (if they came to work at all) in an emergency. A similar situation
existed for law enforcement personnel; in some jurisdictions, most hospital
security guards are off-duty police officers. In addition, many police officers
are also part of National Guard units. It seems likely that these jurisdictions
would have a serious shortage of law enforcement personnel during an emergency
(EPHS 6, 7, and 8). Additionally, in all but one health department, a key public
health function was dependent on a single person who was very close to retirement.
Hiring freezes imposed by state and local budget crises and bureaucratic hiring
processes compound staff shortages in every site. Finally, the lack of a pipeline
for epidemiologists, lab personnel, and public health nurses means that even
if funding were adequate, there are not sufficient numbers of qualified people
available to be hired (EPHS 8).
Most jurisdictions had similar types of needs. Many were allocating scarce resources,
often working on their own, to fill needs that are likely common to most jurisdictions.
Examples include developing programs to train nurses assigned to other functions
to help with an epidemiological investigation; rewriting of laboratory procedures
for processing samples or conducting diagnostic tests; and developing emergency
response plans, policies, and procedures. An overarching need was a statewide
information system that could be used to receive automated reports from hospital
and commercial laboratories; manage a public health emergency, including mapping,
managing, and monitoring the status of contact tracing and other investigative
activities; and administration and monitoring of vaccination or prophylaxis.
Costs.
Preparedness efforts appeared to be associated with sizable unintended costs.
In nearly all cases, public health directors described needing to reassign staff
with responsibilities for other key functions (such as teen pregnancy prevention
or sexually transmitted disease [STD] contact tracing) to preparedness activities,
particularly in preparing the smallpox plan. Several health directors reported
that they are contemplating curtailing or suspending key public health activities—including
tuberculosis (TB) control, STD contact tracing, childhood immunization, and
pregnancy prevention activities—or have already done so. Directors reported
that these decisions were made in part because of the higher priority accorded
the CDC’s bioterrorism mandates but also because of state and local budget
cuts and overall workforce shortages. It is not clear when the downstream consequences
of these programmatic changes will be manifest as population health problems.
Discussion
We found wide variation in the level of public health preparedness across a
representative sample of local public health jurisdictions in California. These
findings are consistent with those of TFAH, which indicated much state-to-state
variation in preparedness. They are cause for concern, for several reasons.
First, they describe much unevenness in preparedness in a state that some consider
to be in the top tier for preparedness. In addition, they indicate that the
degree of protection afforded by the public health system is highly dependent
on where one lives, and not merely on whether one chooses an urban or rural
lifestyle. Variations can indicate poor agreement on the standards for quality
performance.
Local variation.
Variation in medical care has been documented at statewide, regional, and local
levels. Large regional variations in care are thought to indicate waste and
inefficiency and suggest ample room for improvement in the cost, efficiency,
and quality of medical care. Our findings suggest a similar phenomenon in public
health. Although we found evidence of progress in preparedness as a result of
the recent investment, the infrastructure in local public health jurisdictions
varied widely—with regard to not only preparedness but also basic elements
such as strategic planning, community health assessment, environmental control,
communicable disease surveillance, workforce development, and emphasis on chronic
diseases versus other public health threats.
In the medical care delivery system, recognition of variation was a major impetus
in the development of consensus measures of health system performance, including
sophisticated methods for case-mix adjustment and the development and use of
solid indicators of quality of care. We believe that public health is ripe for
a similar movement. Whether at the local or state level, the public is entitled
to know how well their public health system is performing. Absent such information,
it is difficult to know how much to invest, and whom to hold accountable for
results. We are a long way from that goal in public health, but the lessons
from the quality improvement movement, whether in personal health care, airline
safety, or automobile manufacturing, suggest that tools exist with which to
get started.
Inefficiency.
Two other findings from our study deserve mention. First, we were repeatedly
struck during our site visits and exercises by tremendous inefficiencies. Often
each local jurisdiction was spending scarce resources to fill a need, such as
developing training programs for public health nurses to learn how to investigate
an outbreak, that was common to all jurisdictions in the state. In light of
the high fixed costs associated with many public health functions, small health
jurisdictions are particularly disadvantaged in this regard. As a result, it
is probably not realistic to expect small counties to ever be sufficiently prepared
for a major bioterrorist event, absent more regional approaches. For many functions,
not just those related to preparedness, it was apparent that some sort of regionalization
and sharing of resources could increase efficiency.
Local health officials reported varying levels of support from state health
officials in developing their preparedness programs. The state adapted the guidance
it received from the CDC (benchmarks and “critical capacities”) into
a local guidance document. Health jurisdictions were required to apply to the
state to receive funds, with funds awarded based on a fixed amount plus a population-based
increment. In addition to this “formal guidance,” some local stakeholders
reported a fair amount of informal communication with knowledgeable state staff
and with the work groups they assembled to implement the CDC grants. Others
indicated that they did not receive adequate technical assistance.
Strong leadership at national, state, and local levels will be required to bring
about needed transformation of the public health system. In most jurisdictions,
there is opportunity for great improvement within reasonable resource constraints.
For example, staffing to conduct an outbreak investigation could be done the
way it has been done for years or could take advantage of new technologies (such
as Web-supported call centers) and even use personnel other than public health
nurses (such as community members or first responders) to obtain basic information
from the affected population. Such redesign could ultimately free up needed
resources that could be reinvested in other critical areas of public health.
Undermining other priorities.
Second, we were disturbed by repeated reports from local health departments
that the combination of local budget cuts and the need to shift key personnel
from other areas to preparedness efforts has meant that they have had to curtail
or eliminate important public health programs. Although these reports are only
anecdotal at this point, similar situations have been noted in other states.11
This underlines both potential unintended consequences of the recent investments
and the need to study this in more detail and remain vigilant for problems associated
with developing only one area of the system. It will likely take several years
before the consequences of these programmatic changes create public health emergencies
themselves.
Study limitations.
Our study has obvious limitations. There are no agreed-upon, evidence-based
standards or measures for public health preparedness, so we used an expert-panel
methodology to guide us toward those standards. This methodology has not been
validated for use in public health.12 The use of
tabletop exercises to measure public health preparedness is new, and the methods
have not been validated. Participants in our exercises were limited to those
invited by the public health director and may not reflect the true array of
stakeholders in the communities’ preparedness efforts.
Because we conducted our exercises in only seven jurisdictions, we cannot ascertain
the generalizability of our findings. However, we have several reasons to believe
that it is quite high. The jurisdictions were broadly representative of all
state health jurisdictions, in terms of size, location, minority populations,
and per capita public health spending. The sampled jurisdictions cover nearly
two-fifths of the state’s population. The ways in which California’s
public health system is organized and financed suggests that the state, by and
large, treats local health jurisdictions equally. Finally, and perhaps most
importantly, we found that the marginal information added by the last two site
visits and exercises was low, which suggests that we were unlikely to learn
much more from conducting additional exercises in many other jurisdictions.
We remain uncertain about what the appropriate number of exercises might be.
We expect that our findings are broadly applicable beyond California. Although
California is the most populous state, it is not alone in its need to deal with
border issues, vulnerable populations, language diversity, or multiple organizational
structures for public health, as well as the panoply of public health threats
faced by all Americans.13
We found wide variation
in the public health infrastructure and the level of preparedness across California’s
local public health jurisdictions. Variation of this magnitude suggests important
opportunities for improved performance, greater efficiency, and clearer standards.
Our finding that two jurisdictions seemed highly prepared by most measures suggests
that there may be exemplary practices that could be shared with other jurisdictions
and other states.
This research was supported by the California Endowment and Kaiser Permanente.
The views expressed here do not necessarily reflect those of the funders. The
authors appreciate the help of the numerous people who participated in site
visits and tabletop exercises, as well as the advice of the expert panel: Raymond
J. Baxter, Maria Campbell Casey, Mark Ghilarducci, Warner Hudson, Brian Johnston,
Donna Knutson, Sally Lehrman, Jack Lewin, Steve Shortell, and Bernard Turnock.
Brad Austin and Paul Halvorson participated as observers. In addition, the authors
appreciate the help of Steven Asch in organizing the expert-panel process and
in reviewing the manuscript; David Mussington and Roger Molander for help with
developing the exercises; Vanessa Solomon for help with the exercises; and Rebecca
McLaughlin for help with preparing the manuscript.
NOTES
1. Institute of Medicine, The Future of Public Health
(Washington: National Academies Press, 1 January 1988).
2. U.S. Centers for Disease Control and Prevention, Public
Health’s Infrastructure—A Status Report (Atlanta: CDC, 2001),
iii.
3. CDC, Notes to Annual Financial Statements, 30 September
2002, www.cdc.gov/fmo/CFOReport2002/NotestoAnnualFinancialStatements.pdf
(12 December 2003).
4. CDC, “Continuation Guidance for Cooperative Agreement
on Public Health Preparedness and Response for Bioterrorism—Budget Year
Four,” 2 May 2003,
www.bt.cdc.gov/planning/continuationguidance/pdf/guidance_intro.pdf
(20 November 2003).
5. American Public Health Association, Committee on Administrative
Practice, Evaluation Schedule for Use in the Study and Appraisal of Community
Health Programs (New York: APHA, 1947).
6. S.F. Derose et al., “Public Health Quality Measurement:
Concepts and Challenges,” Annual Review of Public Health 23 (2002):
1–21.
7. Trust for America’s Health, “Ready or Not? Protecting
the Public’s Health in the Age of Bioterrorism,” December 2003, healthyamericans.org/state/bioterror/Bioterror.pdf
(12 January 2004).
8. Public Health Functions Steering Committee, “Public
Health in America,” Fall 1994, www.health.gov/phfunctions/public.htm
(8 October 2003); and IOM, The Future of the Public’s Health in the
Twenty-first Century (Washington: National Academies Press, 11 November
2002), 31.
9. S. Asch et al., “Performance Measures for Public Health
Preparedness: A Review” (Unpublished paper, February 2004).
10. Each of California’s fifty-eight counties has a health
department, along with three cities (Berkeley, Long Beach, and Pasadena). The
three city health departments operate independent of the corresponding county
health departments. Los Angeles receives funding directly from the CDC.
11. B.J. Turnock, Public Health Preparedness at a Price:
Illinois (New York: Century Foundation, 2003).
12. P.G. Shekelle, M.R. Chassin, and R.E. Park, “Assessing
the Predictive Validity of the RAND/UCLA Appropriateness Method Criteria for
Performing Carotid Endarterectomy,” International Journal of Technology
Assessment in Health Care 14, no. 4 (1998): 707–727.
13. National Association of City and County Health Officers,
“NACCHO Survey Examines State/Local Health Department Relationships,”
Research Brief no. 2, October 1998,
www.naccho.org/downloadfile2.cfm?filenamex=General65.pdf
(24 May 20040).
Nicole Lurie (lurie{at}rand.org)
is a senior natural scientist and the Paul O'Neill Alcoa Professor at RAND in
Arlington, Virginia. Jeffrey Wasserman is a senior policy researcher at RAND
in Santa Monica, California. Michael Stoto is a senior statistical scientist
and Sarah Myers, an associate social research scientist, at RAND in Arlington.
Poki Namkung is health officer for the City of Berkeley (California). Jonathan
Fielding is director of public health and the health officer for the Los Angeles
(California) County Department of Health Services. Robert Valdez is a senior
health scientist at RAND (Arlington).
DOI: 10.1377/hlthaff.W4.341
©2004 Project HOPE–The People-to-People Health Foundation, Inc.
|
