|
D A T A W A T C H :
P U B L I C H E A L T H R E S P O N S E
30 August 2005
Public Health Response
To Urgent Case Reports
A structured test reveals gaps
in response times
in nineteen communities across the country.
by David J. Dausey, Nicole
Lurie, and Alexis Diamond
ABSTRACT:
We
evaluated the ability of local public health agencies (LPHAs) to meet a preparedness
standard set by the U.S. Centers for Disease Control and Prevention (CDC): to
receive and respond to urgent case reports of communicable diseases twenty-four
hours a day, seven days a week. We found substantial variability in performance
and in the systems in place to respond to such reports. Development and implementation
of measures of public health agencies’ performance are crucial to improving
public health preparedness and, ultimately, to assuring the agencies’
accountability.
Local public health agencies (LPHAs) are one of the country’s first lines
of defense during disease outbreaks. In the wake of September 11 and the subsequent
anthrax attacks, strengthening LPHAs’ ability to detect and respond to
bioterrorism and naturally occurring disease outbreaks has been a national priority.1
Nevertheless, LPHA surveillance and reporting systems designed to receive case
reports from health care and laboratory workers in the field tend to rely upon
fragmented communication networks and paper-based case-reporting mechanisms.2
Such reports are LPHAs’ primary source of information for detecting disease
outbreaks and initiating preemptive public health responses. Policies and procedures
for reporting to these systems can differ within and across states, which makes
it difficult to coordinate and consolidate real-time information across public
health agencies at federal, state, and local levels.
In 2003, to encourage public health agencies to move in this direction, the
U.S. Centers for Disease Control and Prevention (CDC) issued guidance to these
agencies to clarify their responsibilities.3 This
guidance contained four primary recommendations: (1) LPHAs should have a single,
well-publicized telephone number to receive all urgent case reports because
telephones are the simplest, quickest, and most direct method of communication.
(2) They also should have a phone triage protocol to process urgent case reports.
(3) They should be capable of receiving urgent case reports twenty-four hours
a day, seven days a week. (4) They should be able to have a trained public health
professional respond to urgent case reports within thirty minutes of receiving
the report.
This guidance, as well as broad guidance related to public health emergency
preparedness, was not accompanied by tools or methods for LPHAs to use to evaluate
and test their systems to ensure that they could comply. We developed a methodology
to evaluate LPHAs’ ability to receive and respond to urgent case reports
based on the CDC’s recommendations and used it to evaluate the performance
of nineteen LPHAs across the country.
Study Methods
Study sites and population.
We identified and invited twenty metropolitan-area LPHAs to participate in a
test of their telephone response systems. We first identified these twenty LPHAs
by constructing a matrix of agency sizes, structures, and broadly defined regions
of the country. We identified twenty health departments that would provide a
roughly even distribution across these characteristics and invited each to participate
in a test of their system. One LPHA declined to participate, resulting in a
sample of nineteen LPHAs in eighteen different states.
Our goal in these tests, conducted February–November 2004, was to reach
an “action officer,” a person at the LPHA responsible for responding
to urgent case reports. Action officers might include physicians, nurses, epidemiologists,
bioterrorism coordinators, and infection control practitioners.
Test design and procedures.
Study procedures were approved by the RAND institutional review board. The director
of each health department provided informed consent and agreed not to inform
staffers of the planned tests. We promised each director that we would not publicly
reveal the list of participating sites or information about agencies’
performance. At the end of the testing period, each LPHA director received a
written summary of his or her agency’s responses, along with deidentified
information on the performance of the other participating LPHAs.
Aside from the health director, no other LPHA personnel were told that the test
was taking place, so tests were completely unannounced. A trained caller contacted
a participating LPHA on six to ten separate occasions over the course of four
months, claiming to be a doctor or nurse at a local health care facility calling
about an urgent case. We included at least one call during normal daytime hours,
during lunch hour, at the end of a workday, after hours on a weekday, and during
the weekend.
After reaching the action officer, the caller stated that the call was a test
and that no further action regarding the subject of the call was required. Action
officers were then asked four questions about their LPHA’s triage procedures
for handling urgent case reports: “How does your LPHA normally triage
case reports?” “What information would you collect from a caller
with an urgent case?” “What next steps would you take after receiving
this urgent case report?” “Who (if anyone) would you notify if you
received an urgent case report?”
After calls were completed, action officers were asked not to discuss the test
with coworkers until overall testing was finished. Because testing involved
multiple calls to each participating LPHA, some action officers received test
calls more than once. To reduce the possibility of having our test discovered
by caller ID systems or the callback numbers left for action officers, we used
disposable cell phones programmed with phone numbers corresponding to the local
area code of the LPHA tested.
In a subset of calls, we presented respondents with a simulated case and asked
for advice before revealing that the call was a test. These calls simulated
cases of botulism, anthrax, smallpox, and bubonic plague. To avoid inadvertently
provoking a public health response or media leak, we provided no clinical information
until we reached the action officer, and in every case we indicated that the
call was a test before it was terminated.
The number of calls placed to each LPHA can be justified by assuming two categories
of response for LPHAs. The first is a “high-response” category,
where responding is a binary variable (for example, responded within thirty
minutes or did not) and the probability of response is greater than 0.7. The
second is a “low-response” category, where the probability of response
is less than 0.3.
Distinguishing between these groups was straightforward for LPHAs that did or
did not respond to several consecutive calls. For example, the probability that
an LPHA in the low-response group would respond to six consecutive calls is
very small (p = .001). Conversely, the probability that an LPHA in
the high-response group would not respond to five consecutive calls is also
small
(p = .0025). Therefore, we terminated testing if an LPHA responded
to six consecutive calls or did not respond to five. We chose to stop at five
calls instead of six for LPHAs not responding to consecutive calls because we
felt the need to contact these LPHAs as soon as possible to alert them to their
problems in responding.
For LPHAs that responded to calls intermittently, placing ten calls per LPHA
allowed us to discern which LPHAs were low or high responders. Specifically,
LPHAs that responded to at least six of ten calls fell into the high-response
distribution, whereas those that responded to fewer than six of ten calls fell
into the low-response distribution.
Outcomes.
We determined whether the LPHA had a single telephone number for calls during
business hours and calls after hours by searching the phone book and the Internet
and by calling directory assistance. We also noted when recorded messages instructed
the caller to use a different number after hours.
We measured the length of time from the initiation of a call until contact with
an action officer and the percentage of calls that were “warm transfers”
(calls transferred immediately to the action officer), as opposed to callbacks
(calls that ended when our callers left a message and waited for the call to
be returned). In addition, we recorded information obtained during the calls,
including the questions asked and instructions provided by action officers and
a description of the mechanisms each health department used to triage calls.
Study Results
All LPHA directors reported to us at the outset that they had a system in place
to receive and respond to emergency case reports twenty-four hours a day, seven
days a week. We placed a total of 143 calls to 19 participating LPHAs (mean
= 7.5 calls) over the study period. At the time of testing, 47 percent of LPHAs
(nine) in our sample had multiple phone numbers for receiving case reports.
Exhibit
1 describes the health departments and their responses. Response times for
calls that were returned ranged from under a minute to 2,470 minutes (mean =
55 minutes); 42 percent of LPHAs (eight) returned all calls within thirty minutes,
while 21 percent (four) returned all calls within fifteen minutes.4
The longest response times, on average, occurred at the end of the workday,
during the evening, and on weekends (means, in minutes, were 99, 61, and 102,
respectively). Four LPHAs (21 percent) received one or more calls that were
not returned.
Three LPHAs did not respond to the first five calls they received. In one of
these instances, we agreed to suspend the tests until the LPHA in question attempted
to resolve its problems. We reinstituted testing once this had been done, and
performance was improved, although not perfect. Eight LPHAs responded to the
first six calls received. The remaining eight LPHAs received ten calls apiece,
and all of them responded to at least six calls.
One state had two participating LPHAs, both of which reported that the state
health department was responsible for receiving calls after hours and forwarding
these calls to an appropriate LPHA action officer. We tested this system by
placing four calls to this state public health agency after hours and recording
how long it took until we received a response from the appropriate LPHA action
officer. In all instances, the response times for these calls greatly exceeded
thirty minutes.
The percentage of calls handled with warm transfers in each LPHA ranged from
25 percent to 100 percent of calls. Only two LPHAs returned all calls within
thirty minutes and handled them with warm transfers. Among LPHAs that responded
within thirty minutes, 69 percent of calls were handled with a warm transfer;
among the rest, half provided warm transfers.
There was also much heterogeneity across LPHAs with regard to procedures for
handling urgent calls. For example, one LPHA had a cellular telephone passed
among staff after hours to receive urgent case reports. Several LPHAs had answering
services to receive urgent case reports, while others provided a pager number
for contact. Many LPHAs had established protocols and call schedules. In most
cases, procedures for handling calls differed depending on whether or not the
call was received during regular business hours.
In some jurisdictions, the LPHA handled calls during the day and the state health
department handled them on weekends. In one complex jurisdiction, a city health
department addressed problems that were gastrointestinal in nature, while the
county health department addressed respiratory complaints.
In 58 percent of LPHAs (eleven), after-hours calls were handled by an answering
service. In 11 percent (two), a recorded message provided instruction or requested
that the caller leave a message. Of those with answering machines, 25 percent
of calls were returned within thirty minutes; 10 percent took more than ten
hours to be returned.
Respondents varied widely in their handling of simulated cases. Several action
officers asked relevant questions about the location of the patient and about
clinical details, but in several cases the responses were troubling. For example,
among respondents presented with a case report from an emergency department
physician of a patient presenting with “pustules on the face, arms, and
legs with lesions in the same stage of development,” none suggested isolation
of the patient or advised the caller to use personal protective equipment. Similarly,
when presented with a case consistent with botulism, one action officer responded,
“You’re right, it does sound like botulism. I wouldn’t worry
too much if I were you.” In response to classic symptoms of bubonic plague,
the action officer told the caller not to worry and to “go back to bed”
because no similar cases had been reported that day.
Discussion
Since 2001, many resources have been invested in strengthening the public health
infrastructure. At the same time, the CDC and the Department of Homeland Security
have each promulgated guidelines and begun to develop standards for public health
preparedness. The public and policymakers rightly want to know whether the country
is becoming more prepared to deal with a public health emergency and how preparedness
can be improved, yet there are almost no objective measures with which to answer
this question. Most reports on the state of preparedness have been based on
state-level checklists and self-reports.
We undertook this study as part of a broader effort to determine whether components
of a public health agency’s performance could be measured objectively.
In this case, there is a widely acknowledged performance benchmark: the ability
to receive and respond to urgent case reports around the clock, every day. For
many other proposed elements of public health emergency preparedness, clear
standards have not yet been articulated, nor are there reliable and valid methods
for measuring them. The methods we developed for testing performance for this
standard are described in detail in an “operations manual” prepared
for this purpose.5
We found wide variation in performance with respect to the standard, including
variation in the availability of phone numbers, the procedures used to triage
calls, and the way in which simulated case reports were handled. Only two LPHAs
consistently met the standard of responding to all calls within thirty minutes
and handling calls with warm transfers. However, we were encouraged to find
that 91 percent of all calls responded to were responded to within thirty minutes. In the
U.S. health care system, large variations in use or procedures raise questions
about quality, efficiency, and safety. In public health systems, such variations
ought to raise similar concerns. They may also reflect the multi-tier, complex
organizational arrangements under which LPHAs operate. For example, in some
states, public health is highly centralized, while in others, LPHAs operate
under home rule. Such variation thwarts attempts to standardize protocols and
procedures and raises the likelihood that there will be delays or errors in
reporting, outbreak detection, and disease control. It also calls into question
the purported ability of the U.S. public health system overall to mount a consistent,
concerted surveillance effort or respond to mass threats.
Early attempts to respond to regional variation in the case of quality and safety
in personal health care focused on practice guidelines, which ultimately became
locally tailored to achieve a balance of usefulness and buy-in. However, regardless
of the clinical issue or geographic location, performance measurement was an
essential step in reducing variation and achieving improvement in quality. The
fields of quality measurement and improvement in health care are now decades
old.
Although there have been previous attempts to measure performance in public
health, most have highlighted improvement as an essential component of assessment.
Improving public health preparedness will similarly require the development
of guidelines and standards, as well as measurement of how well they are attained.
Lessons learned from the personal health care system and aspects of both the
public and private sectors should facilitate the process of quality improvement
in public health.
One of the expressed expectations for the investment in public health preparedness
was that it would improve the public health infrastructure more broadly. Although
large-scale public health emergencies occur infrequently, health departments
routinely conduct activities that are essential to preparedness, including responding
to phone calls, fielding case reports, investigating worrisome cases, and communicating
with the public. Measuring how well they perform these functions is a critical
step toward improving the quality of our public health systems.
One additional benefit of the focus on accountability for the investment in
public health preparedness is the degree to which it highlights the need to
measure other aspects of public health agency performance. We note, however,
that many of the outcomes of importance for public health are beyond the sole,
direct control of public health agencies, which creates additional challenges
for the relationship between performance and accountability.
The CDC intends to require measurement of public health preparedness as part
of its cooperative agreements with state health departments. We note, however,
that in this case, we were able to use an objective measure to assess a very
clear standard. Such standards are rare; performance standards and measures
are urgently needed to assess and improve other aspects of public health preparedness
and should extend beyond state agencies to the local level.
We are encouraged that “improvement” is one of the goals of the
2005–06 CDC public health preparedness guidance. This will likely require
not only a greatly increased emphasis on performance measurement but the application
of other quality improvement methods and processes. It may also require additional
resources.
Our study has several important limitations. Participating health departments
were not chosen at random and might not be representative of LPHAs throughout
the entire United States. We suspect that the variability in LPHAs is even greater
than that seen in our sample. Second, in some smaller jurisdictions, we reached
the same on-call action officer more than once, and it is possible that in these
cases, the LPHA personnel might have realized that they were participating in
our study. Furthermore, one jurisdiction has a weekly practice of reviewing
all calls and how they were handled, so it is likely that some action officers
learned about the testing through that process.
Third, there are no established standards for the advice health departments
should provide callers. Therefore, we cannot comment conclusively about the
appropriateness of the responses we received. Rather, we have provided examples
of some potentially problematic responses. Finally, our goal was to develop
an easy-to-use testing method. We did not compare this method with other forms
of assessment, such as checklists or computer-generated call-downs, although
we expect that prior to our tests, all participating LPHAs would have self-reported
that they had a system in place for receipt and response to calls twenty-four
hours a day, seven days a week.
The tests we describe in this paper are easy to implement and can be one component
of a quality measurement and improvement system. We are encouraged that some
of the LPHAs we worked with made substantial changes to their call and triage
systems after receiving the results of our test calls. Additionally, one health
department pilot-tested a draft of the “operations manual” for conducting
such tests and made changes to its systems based on their findings.
These findings should not be interpreted to imply that the investment in the
public health infrastructure to date has been in vain. In part, because of the
complexity and magnitude of the task, it has taken several years for resources
to reach the local level and to have measurable impacts.
Strengthening the public health infrastructure is essential not only for bioterrorism
preparedness, but for addressing a wide range of health threats, from pandemic
influenza to chronic disease. Achieving that goal will require the measurement
and continuous improvement of key public health processes. Using call responsiveness
as an example, we have shown that measurement is possible and that there is
room for improvement across a wide range of health departments.
This study was supported by a contract from the U.S. Department of Health
and Human Services (282-00-0005-T11). The authors appreciate the support of
William Raub and Lara Lamprecht at HHS and the contributions of the health departments
and their staffs in participating in the study and responding to test calls.
They also appreciate the assistance of Rebecca McLaughlin in preparing the manuscript
for publication.
NOTES
1. Joseph M. Henderson, director, Office of Terrorism Preparedness
and Emergency Response, U.S. Centers for Disease Control and Prevention, testimony
before the Subcommittee on Emergency Preparedness and Response, House Select
Committee on Homeland Security, 24 September 2003, www.hhs.gov/asl/testify/t030924.html
(12 May 2005).
2. M.L. Popovich et al., “A Framework for Creating a Public
Health Bioterrorism Surveillance System,” White Paper no. WHP018A, October
2001, www.stchome.com/white_papers/WHP018A.pdf
(3 August 2005).
3. U.S. Centers for Disease Control and Prevention, “Improving
Surveillance Infrastructure for Terrorism Detection: The Eight-Cities Project
Resource Materials,” 7 April 2004, www.cdc.gov/epo/dphsi/8city.htm
(3 August 2005).
4. See Supplemental Exhibit 1, online at content.healthaffairs.org/cgi/content/full/hlthaff.w5.412/DC2.
5. D.J. Dausey et al., Tests to Evaluate Public Health Disease
Reporting Systems in Local Public Health Agencies, RAND Health Pub. no.
TR-260-DHHS, 2005, www.rand.org/publications/TR/TR260
(18 July 2005).
David Dausey (david_dausey{at}rand.org)
is an associate policy researcher at RAND in Pittsburgh, Pennsylvania. Nicole
Lurie is senior natural scientist and the Paul O'Neill Alcoa Professor at RAND
in Arlington, Virginia. Alexis Diamond is a doctoral candidate at the Center
for Basic Research in the Social Sciences, Harvard University, in Cambridge,
Massachusetts.
DOI: 10.1377/hlthaff.w5.412
©2005 Project HOPE–The People-to-People Health
Foundation, Inc.
|
