This Article Abstract Submit a response to this article Services E-mail this article to a friend Alert me to new issues of the journal

D A T A W A T C H :
P U B L I C  H E A L T H  R E S P O N S E
 
30 August 2005  
  

Public Health Response 
To Urgent Case Reports

A structured test reveals gaps in response times
in nineteen communities across the country.


by David J. Dausey, Nicole Lurie, and Alexis Diamond


ABSTRACT:

We evaluated the ability of local public health agencies (LPHAs) to meet a preparedness standard set by the U.S. Centers for Disease Control and Prevention (CDC): to receive and respond to urgent case reports of communicable diseases twenty-four hours a day, seven days a week. We found substantial variability in performance and in the systems in place to respond to such reports. Development and implementation of measures of public health agencies’ performance are crucial to improving public health preparedness and, ultimately, to assuring the agencies’ accountability.

Local public health agencies (LPHAs) are one of the country’s first lines of defense during disease outbreaks. In the wake of September 11 and the subsequent anthrax attacks, strengthening LPHAs’ ability to detect and respond to bioterrorism and naturally occurring disease outbreaks has been a national priority.¹ Nevertheless, LPHA surveillance and reporting systems designed to receive case reports from health care and laboratory workers in the field tend to rely upon fragmented communication networks and paper-based case-reporting mechanisms.² Such reports are LPHAs’ primary source of information for detecting disease outbreaks and initiating preemptive public health responses. Policies and procedures for reporting to these systems can differ within and across states, which makes it difficult to coordinate and consolidate real-time information across public health agencies at federal, state, and local levels.

In 2003, to encourage public health agencies to move in this direction, the U.S. Centers for Disease Control and Prevention (CDC) issued guidance to these agencies to clarify their responsibilities.³ This guidance contained four primary recommendations: (1) LPHAs should have a single, well-publicized telephone number to receive all urgent case reports because telephones are the simplest, quickest, and most direct method of communication. (2) They also should have a phone triage protocol to process urgent case reports. (3) They should be capable of receiving urgent case reports twenty-four hours a day, seven days a week. (4) They should be able to have a trained public health professional respond to urgent case reports within thirty minutes of receiving the report.
This guidance, as well as broad guidance related to public health emergency preparedness, was not accompanied by tools or methods for LPHAs to use to evaluate and test their systems to ensure that they could comply. We developed a methodology to evaluate LPHAs’ ability to receive and respond to urgent case reports based on the CDC’s recommendations and used it to evaluate the performance of nineteen LPHAs across the country.

Study Methods

Study sites and population. We identified and invited twenty metropolitan-area LPHAs to participate in a test of their telephone response systems. We first identified these twenty LPHAs by constructing a matrix of agency sizes, structures, and broadly defined regions of the country. We identified twenty health departments that would provide a roughly even distribution across these characteristics and invited each to participate in a test of their system. One LPHA declined to participate, resulting in a sample of nineteen LPHAs in eighteen different states.

Our goal in these tests, conducted February–November 2004, was to reach an “action officer,” a person at the LPHA responsible for responding to urgent case reports. Action officers might include physicians, nurses, epidemiologists, bioterrorism coordinators, and infection control practitioners.

Test design and procedures. Study procedures were approved by the RAND institutional review board. The director of each health department provided informed consent and agreed not to inform staffers of the planned tests. We promised each director that we would not publicly reveal the list of participating sites or information about agencies’ performance. At the end of the testing period, each LPHA director received a written summary of his or her agency’s responses, along with deidentified information on the performance of the other participating LPHAs.

Aside from the health director, no other LPHA personnel were told that the test was taking place, so tests were completely unannounced. A trained caller contacted a participating LPHA on six to ten separate occasions over the course of four months, claiming to be a doctor or nurse at a local health care facility calling about an urgent case. We included at least one call during normal daytime hours, during lunch hour, at the end of a workday, after hours on a weekday, and during the weekend.

After reaching the action officer, the caller stated that the call was a test and that no further action regarding the subject of the call was required. Action officers were then asked four questions about their LPHA’s triage procedures for handling urgent case reports: “How does your LPHA normally triage case reports?” “What information would you collect from a caller with an urgent case?” “What next steps would you take after receiving this urgent case report?” “Who (if anyone) would you notify if you received an urgent case report?”

After calls were completed, action officers were asked not to discuss the test with coworkers until overall testing was finished. Because testing involved multiple calls to each participating LPHA, some action officers received test calls more than once. To reduce the possibility of having our test discovered by caller ID systems or the callback numbers left for action officers, we used disposable cell phones programmed with phone numbers corresponding to the local area code of the LPHA tested.

In a subset of calls, we presented respondents with a simulated case and asked for advice before revealing that the call was a test. These calls simulated cases of botulism, anthrax, smallpox, and bubonic plague. To avoid inadvertently provoking a public health response or media leak, we provided no clinical information until we reached the action officer, and in every case we indicated that the call was a test before it was terminated.

The number of calls placed to each LPHA can be justified by assuming two categories of response for LPHAs. The first is a “high-response” category, where responding is a binary variable (for example, responded within thirty minutes or did not) and the probability of response is greater than 0.7. The second is a “low-response” category, where the probability of response is less than 0.3.
Distinguishing between these groups was straightforward for LPHAs that did or did not respond to several consecutive calls. For example, the probability that an LPHA in the low-response group would respond to six consecutive calls is very small (p = .001). Conversely, the probability that an LPHA in the high-response group would not respond to five consecutive calls is also small
(p = .0025). Therefore, we terminated testing if an LPHA responded to six consecutive calls or did not respond to five. We chose to stop at five calls instead of six for LPHAs not responding to consecutive calls because we felt the need to contact these LPHAs as soon as possible to alert them to their problems in responding.

For LPHAs that responded to calls intermittently, placing ten calls per LPHA allowed us to discern which LPHAs were low or high responders. Specifically, LPHAs that responded to at least six of ten calls fell into the high-response distribution, whereas those that responded to fewer than six of ten calls fell into the low-response distribution.

Outcomes. We determined whether the LPHA had a single telephone number for calls during business hours and calls after hours by searching the phone book and the Internet and by calling directory assistance. We also noted when recorded messages instructed the caller to use a different number after hours.

We measured the length of time from the initiation of a call until contact with an action officer and the percentage of calls that were “warm transfers” (calls transferred immediately to the action officer), as opposed to callbacks (calls that ended when our callers left a message and waited for the call to be returned). In addition, we recorded information obtained during the calls, including the questions asked and instructions provided by action officers and a description of the mechanisms each health department used to triage calls.

Study Results

All LPHA directors reported to us at the outset that they had a system in place to receive and respond to emergency case reports twenty-four hours a day, seven days a week. We placed a total of 143 calls to 19 participating LPHAs (mean = 7.5 calls) over the study period. At the time of testing, 47 percent of LPHAs (nine) in our sample had multiple phone numbers for receiving case reports.

Exhibit 1 describes the health departments and their responses. Response times for calls that were returned ranged from under a minute to 2,470 minutes (mean = 55 minutes); 42 percent of LPHAs (eight) returned all calls within thirty minutes, while 21 percent (four) returned all calls within fifteen minutes.⁴ The longest response times, on average, occurred at the end of the workday, during the evening, and on weekends (means, in minutes, were 99, 61, and 102, respectively). Four LPHAs (21 percent) received one or more calls that were not returned.

Three LPHAs did not respond to the first five calls they received. In one of these instances, we agreed to suspend the tests until the LPHA in question attempted to resolve its problems. We reinstituted testing once this had been done, and performance was improved, although not perfect. Eight LPHAs responded to the first six calls received. The remaining eight LPHAs received ten calls apiece, and all of them responded to at least six calls.

One state had two participating LPHAs, both of which reported that the state health department was responsible for receiving calls after hours and forwarding these calls to an appropriate LPHA action officer. We tested this system by placing four calls to this state public health agency after hours and recording how long it took until we received a response from the appropriate LPHA action officer. In all instances, the response times for these calls greatly exceeded thirty minutes.

The percentage of calls handled with warm transfers in each LPHA ranged from 25 percent to 100 percent of calls. Only two LPHAs returned all calls within thirty minutes and handled them with warm transfers. Among LPHAs that responded within thirty minutes, 69 percent of calls were handled with a warm transfer; among the rest, half provided warm transfers.

There was also much heterogeneity across LPHAs with regard to procedures for handling urgent calls. For example, one LPHA had a cellular telephone passed among staff after hours to receive urgent case reports. Several LPHAs had answering services to receive urgent case reports, while others provided a pager number for contact. Many LPHAs had established protocols and call schedules. In most cases, procedures for handling calls differed depending on whether or not the call was received during regular business hours.
In some jurisdictions, the LPHA handled calls during the day and the state health department handled them on weekends. In one complex jurisdiction, a city health department addressed problems that were gastrointestinal in nature, while the county health department addressed respiratory complaints.

In 58 percent of LPHAs (eleven), after-hours calls were handled by an answering service. In 11 percent (two), a recorded message provided instruction or requested that the caller leave a message. Of those with answering machines, 25 percent of calls were returned within thirty minutes; 10 percent took more than ten hours to be returned.

Respondents varied widely in their handling of simulated cases. Several action officers asked relevant questions about the location of the patient and about clinical details, but in several cases the responses were troubling. For example, among respondents presented with a case report from an emergency department physician of a patient presenting with “pustules on the face, arms, and legs with lesions in the same stage of development,” none suggested isolation of the patient or advised the caller to use personal protective equipment. Similarly, when presented with a case consistent with botulism, one action officer responded, “You’re right, it does sound like botulism. I wouldn’t worry too much if I were you.” In response to classic symptoms of bubonic plague, the action officer told the caller not to worry and to “go back to bed” because no similar cases had been reported that day.

Discussion

Since 2001, many resources have been invested in strengthening the public health infrastructure. At the same time, the CDC and the Department of Homeland Security have each promulgated guidelines and begun to develop standards for public health preparedness. The public and policymakers rightly want to know whether the country is becoming more prepared to deal with a public health emergency and how preparedness can be improved, yet there are almost no objective measures with which to answer this question. Most reports on the state of preparedness have been based on state-level checklists and self-reports.

We undertook this study as part of a broader effort to determine whether components of a public health agency’s performance could be measured objectively. In this case, there is a widely acknowledged performance benchmark: the ability to receive and respond to urgent case reports around the clock, every day. For many other proposed elements of public health emergency preparedness, clear standards have not yet been articulated, nor are there reliable and valid methods for measuring them. The methods we developed for testing performance for this standard are described in detail in an “operations manual” prepared for this purpose.⁵

We found wide variation in performance with respect to the standard, including variation in the availability of phone numbers, the procedures used to triage calls, and the way in which simulated case reports were handled. Only two LPHAs consistently met the standard of responding to all calls within thirty minutes and handling calls with warm transfers. However, we were encouraged to find that 91 percent of all calls responded to were responded to within thirty minutes. In the U.S. health care system, large variations in use or procedures raise questions about quality, efficiency, and safety. In public health systems, such variations ought to raise similar concerns. They may also reflect the multi-tier, complex organizational arrangements under which LPHAs operate. For example, in some states, public health is highly centralized, while in others, LPHAs operate under home rule. Such variation thwarts attempts to standardize protocols and procedures and raises the likelihood that there will be delays or errors in reporting, outbreak detection, and disease control. It also calls into question the purported ability of the U.S. public health system overall to mount a consistent, concerted surveillance effort or respond to mass threats.

Early attempts to respond to regional variation in the case of quality and safety in personal health care focused on practice guidelines, which ultimately became locally tailored to achieve a balance of usefulness and buy-in. However, regardless of the clinical issue or geographic location, performance measurement was an essential step in reducing variation and achieving improvement in quality. The fields of quality measurement and improvement in health care are now decades old.

Although there have been previous attempts to measure performance in public health, most have highlighted improvement as an essential component of assessment. Improving public health preparedness will similarly require the development of guidelines and standards, as well as measurement of how well they are attained. Lessons learned from the personal health care system and aspects of both the public and private sectors should facilitate the process of quality improvement in public health.

One of the expressed expectations for the investment in public health preparedness was that it would improve the public health infrastructure more broadly. Although large-scale public health emergencies occur infrequently, health departments routinely conduct activities that are essential to preparedness, including responding to phone calls, fielding case reports, investigating worrisome cases, and communicating with the public. Measuring how well they perform these functions is a critical step toward improving the quality of our public health systems.

One additional benefit of the focus on accountability for the investment in public health preparedness is the degree to which it highlights the need to measure other aspects of public health agency performance. We note, however, that many of the outcomes of importance for public health are beyond the sole, direct control of public health agencies, which creates additional challenges for the relationship between performance and accountability.

The CDC intends to require measurement of public health preparedness as part of its cooperative agreements with state health departments. We note, however, that in this case, we were able to use an objective measure to assess a very clear standard. Such standards are rare; performance standards and measures are urgently needed to assess and improve other aspects of public health preparedness and should extend beyond state agencies to the local level.

We are encouraged that “improvement” is one of the goals of the 2005–06 CDC public health preparedness guidance. This will likely require not only a greatly increased emphasis on performance measurement but the application of other quality improvement methods and processes. It may also require additional resources.

Our study has several important limitations. Participating health departments were not chosen at random and might not be representative of LPHAs throughout the entire United States. We suspect that the variability in LPHAs is even greater than that seen in our sample. Second, in some smaller jurisdictions, we reached the same on-call action officer more than once, and it is possible that in these cases, the LPHA personnel might have realized that they were participating in our study. Furthermore, one jurisdiction has a weekly practice of reviewing all calls and how they were handled, so it is likely that some action officers learned about the testing through that process.

Third, there are no established standards for the advice health departments should provide callers. Therefore, we cannot comment conclusively about the appropriateness of the responses we received. Rather, we have provided examples of some potentially problematic responses. Finally, our goal was to develop an easy-to-use testing method. We did not compare this method with other forms of assessment, such as checklists or computer-generated call-downs, although we expect that prior to our tests, all participating LPHAs would have self-reported that they had a system in place for receipt and response to calls twenty-four hours a day, seven days a week.

The tests we describe in this paper are easy to implement and can be one component of a quality measurement and improvement system. We are encouraged that some of the LPHAs we worked with made substantial changes to their call and triage systems after receiving the results of our test calls. Additionally, one health department pilot-tested a draft of the “operations manual” for conducting such tests and made changes to its systems based on their findings.

These findings should not be interpreted to imply that the investment in the public health infrastructure to date has been in vain. In part, because of the complexity and magnitude of the task, it has taken several years for resources to reach the local level and to have measurable impacts.

Strengthening the public health infrastructure is essential not only for bioterrorism preparedness, but for addressing a wide range of health threats, from pandemic influenza to chronic disease. Achieving that goal will require the measurement and continuous improvement of key public health processes. Using call responsiveness as an example, we have shown that measurement is possible and that there is room for improvement across a wide range of health departments.

This study was supported by a contract from the U.S. Department of Health and Human Services (282-00-0005-T11). The authors appreciate the support of William Raub and Lara Lamprecht at HHS and the contributions of the health departments and their staffs in participating in the study and responding to test calls. They also appreciate the assistance of Rebecca McLaughlin in preparing the manuscript for publication.

NOTES

1. Joseph M. Henderson, director, Office of Terrorism Preparedness and Emergency Response, U.S. Centers for Disease Control and Prevention, testimony before the Subcommittee on Emergency Preparedness and Response, House Select Committee on Homeland Security, 24 September 2003, www.hhs.gov/asl/testify/t030924.html (12 May 2005).
2. M.L. Popovich et al., “A Framework for Creating a Public Health Bioterrorism Surveillance System,” White Paper no. WHP018A, October 2001, www.stchome.com/white_papers/WHP018A.pdf (3 August 2005).
3. U.S. Centers for Disease Control and Prevention, “Improving Surveillance Infrastructure for Terrorism Detection: The Eight-Cities Project Resource Materials,” 7 April 2004, www.cdc.gov/epo/dphsi/8city.htm (3 August 2005).
4. See Supplemental Exhibit 1, online at content.healthaffairs.org/cgi/content/full/hlthaff.w5.412/DC2.
5. D.J. Dausey et al., Tests to Evaluate Public Health Disease Reporting Systems in Local Public Health Agencies, RAND Health Pub. no. TR-260-DHHS, 2005, www.rand.org/publications/TR/TR260 (18 July 2005).

David Dausey (david_dausey{at}rand.org) is an associate policy researcher at RAND in Pittsburgh, Pennsylvania. Nicole Lurie is senior natural scientist and the Paul O'Neill Alcoa Professor at RAND in Arlington, Virginia. Alexis Diamond is a doctoral candidate at the Center for Basic Research in the Social Sciences, Harvard University, in Cambridge, Massachusetts.

DOI: 10.1377/hlthaff.w5.412
©2005 Project HOPE–The People-to-People Health Foundation, Inc.