This Article Abstract Submit a response to this article Services E-mail this article to a friend Alert me to new issues of the journal

P E R S P E C T I V E : 
F D A  R E V I E W
W E B  E X C L U S I V E   

30 January 2004  
  

Defending Submission-Year Analyses 
Of New Drug Approvals

A counterargument for why analysis of the year of submission,
not the year of approval, is the best way of assessing the effects
of FDA staffing levels.

By Daniel P. Carpenter

ABSTRACT:

In response to the critique of Mary Olson, Daniel Carpenter, on behalf of his coauthors, addresses the issue of analysis based on the year a new drug is submitted for Food and Drug Administration (FDA) approval, not the year it is approved. Both substantive knowledge of the FDA drug review process and sound social science theory favor submission-year averaging. The history and bureaucratic mechanics of the Center for Drug Evaluation and Review (CDER) conform to the author’s assumption. The statistical theory of optimal experimentation also points to the beginning of review as the locus for effects upon decisions.

My coauthors and I thank Mary Olson for her valuable criticism of our paper.¹ In other studies, I have arrived at conclusions that are much in agreement with hers, both theoretically and empirically.² I must nonetheless dissent from her research conclusions and policy suggestions here.

Olson does not provide new data or analysis but instead aims criticism at the method that my coauthors and I used in our analysis: submission-year averaging. Put simply, the question is this: When assessing the effect of any variable upon regulatory approval times, should we attach its effects to the year in which the drug finished the new drug application (NDA) review process (Olson’s assumption), or the year in which it started the NDA review process (our assumption)? Our strong belief is that both substantive knowledge of the Food and Drug Administration (FDA) drug review process and sound social science theory strongly favor submission-year averaging. Here I offer two considerations in support of this point.

First, the history and actual bureaucratic mechanics of the Center for Drug Evaluation and Review (CDER) conform much more to our assumption than to Olson’s. The crucial decisions in CDER drug review—the priority rating given to the application, the composition of the review team, procedures under which review and evaluation of the evidence will proceed, and many others—are made at the beginning, not the end of NDA review. This conclusion is supported in recent journalistic studies of the FDA.³ It is also supported in numerous interviews I have conducted with FDA officials and pharmaceutical company officials, and by analysis of hundreds of actual NDA reviews conducted at the agency.

Second, the statistical theory of optimal experimentation also points to the beginning of review as the locus for effects upon decisions.⁴ To see this, consider the metaphor of CDER staff as an “investment” in a particular review, the same way that a pharmaceutical firm might allocate extra staff to a set of research and development (R&D) projects. To take the metaphor a step further, suppose we seek to unearth a relationship between the firm’s “stock” of human capital and the speed with which its R&D projects proceed. (This is essentially what my coauthors and I were doing in our paper.) In most any setting governed by rational, scientific behavior, exogenous additions to the company’s stock of staff would exercise their effects at the beginning of the R&D project, not at the end. Indeed, a large literature on stochastic decision theory in mathematical statistics and finance renders this point consistently and unequivocally.⁵ In this way, our results imply a high degree of scientific and technical rationality at CDER.

Olson expresses two concerns about approval-year averaging. She first claims that some of the drugs submitted in a given year (say, 1995) may be influenced by developments subsequent to the submission year. This is a helpful point, but note first that Olson’s approval-year averaging commits the same kind of error, with results that are arguably worse. Among the new molecular entities (NMEs) approved in 1995, for instance, one was submitted in 1988, four in 1991, three in 1992, eight in 1993, ten in 1994, and only six in 1994. Fully half of the 1995-approved NMEs, then, were submitted two or more years before 1995. It is, as a result, almost impossible to generate valid inferences from a sample where the “treatment” has been assigned ex post in this fashion.

One way of assessing Olson’s claim is to decompose each drug’s review time into months and then assign CDER staff aggregates to the specific month that the drug is under review. In this way, each drug can “experience” several different levels of staffing at CDER, and the analyst can proceed agnostically in the debate between approval-year averaging and submission-year averaging. In unpublished analyses that I have reported on the “FDA Project” page of my Web site, I report estimates from these “time-varying covariates” duration model estimations.⁶ The essential result is that even when we accept Olson’s criticism and assign CDER staff aggregates separately to each month that the drug was under review, CDER staff has a significant and negative relationship with expected approval time, and controlling for CDER staff, no such robust relationship is observed between the drug user-fee law, the Prescription Drug User Fee Act (PDUFA) of 1992, and drug approval times.

Olson’s general lament is that a drug’s year of submission is affected by the firm, but this is largely irrelevant for our study. Firms generally have incentives to develop drugs as quickly as possible. Yet even if we accepted the hypothesis that firms’ submissions were endogenous, the implication would be that firms submit more drugs (especially those with expected longer reviews) during years with more CDER staff, which would imply that our statistical equations underestimate the effect of resources upon review times.

Two other statistical issues are worth discussing here. First, our statistical methods are better matched to the nature of the drug review data (nonnegative duration data in which the unit of analysis is the NDA) than Olson’s time-series analyses. Olson’s analyses are subject to ecological regression bias; ours are not.⁷

Second, it is likely the case that both Olson’s study and ours suffer from omitted variable bias, as administrative workload is excluded.⁸ This matters because if we had a good measure of administrative workload, this measure would have increased over the past twenty or thirty years (as almost any study of the FDA would suggest). The implication of this result is crucial: The exclusion of a good measure of workload results in an underestimate of the effect of resources upon approval times, both in our study and in Olson’s.

Finally, it is worth noting that even if some version of Olson’s incentives argument holds true, it must be because staff resources are consequential in achieving administrative efficiency. In Olson’s argument, if the FDA does not meet performance goals, the agency will be “punished” in the sense that funding from user fees will dry up.⁹ Yet what else is the “punishment” in the PDUFA statutes but denial of the staff resources that FDA officials have so long sought? In other words, the incentives scheme works only if FDA officials value the additional staff and the acceleration effects that they bring.

In short, Olson’s argument is neither theoretically, statistically, nor historically compelling. Occam’s razor, sound theory, and the weight of the evidence point to a different assessment. The more CDER staff hired, the lower the approval time. This relationship, which has held over half a century and is statistically robust in repeated statistical analyses of drug review times, has now been quantified for use by policy analysts.

NOTES

1. M.K. Olson, “Explaining Reductions in FDA Drug Review Times: PDUFA Matters,” Health Affairs, 29 January 2004, content.healthaffairs.org/cgi/content/abstract/hlthaff.w4.s1 (29 January 2004); and D.P. Carpenter et al., “Approval Times for New Drugs: Does the Source of Funding for FDA Staff Matter?” Health Affairs, 17 December 2003, content.healthaffairs.org/cgi/content/abstract/hlthaff.w3.618 (28 January 2004).
2. D.P. Carpenter, “Groups, the Media, Agency Waiting Costs, and FDA Drug Approval,” American Journal of Political Science 46, no. 3 (2002): 490– 505.
3. P.J. Hilts, Protecting America’s Health: The FDA, Business, and One Hundred Years of Regulation (New York: Alfred A. Knopf, 2003), chap. 15.
4. Olson’s recent research on the trade-off between quicker approvals and adverse reactions lends support to an optimal stopping interpretation of drug review. M.K. Olson, “Pharmaceutical Policy Change and the Safety of New Drugs,” Journal of Law and Economics 45, no. 2, part 2 (2002): 615– 642.
5. A.N. Shiryaev, Optimal Stopping Rules (New York: Springer-Verlag, 1970); and A. Dixit and R. Pindyck, Investment under Uncertainty (Princeton, N.J.: Princeton University Press, 1994).
6. D. Carpenter, “The FDA Project,” people.hmdc.harvard.edu/~dcarpent/fdaproject.html (28 January 2004).
7. G. King, A Solution to the Ecological Inference Problem: Reconstructing Individual Behavior from Aggregate Data (Princeton, N.J.: Princeton University Press, 1997). See Carpenter, “Groups, the Media,” for further discussion of these methods.
8. In some unpublished statistical analyses and in Olson’s forthcoming work, the number of NDA submissions in a year has been used as a proxy for workload, but this is poorly correlated with the lumping of tasks by Congress upon the FDA (for instance, the provision of the 1962 Kefauver-Harris Amendments, or the “twenty-one new federal laws, enacted [between 1979 and 1987] giving the agency additional responsibilities, without any budget increases to finance the work,” in Hilts, Protecting America’s Health, 239). See M.K. Olson, “Managing Delegation in the FDA: Reducing Delay in New-Drug Review,” Journal of Health Politics, Policy and Law (forthcoming); and M.K. Olson, “Examining the Determinants of New Drug Review: Considering Alternative Approaches,” Journal of Health Politics, Policy and Law (forthcoming).
9. I find it difficult to believe that politicians can truly commit themselves to such a scheme, but let us assume such a credible commitment is possible, for the sake of argument. See also M.K. Olson, “Regulatory Reform and Bureaucratic Responsiveness to Firms: The Impact of User Fees in the FDA,” Journal of Economics and Management Strategy 9, no. 3 (2000): 363–395.

Daniel Carpenter (dcarpenter{at}latte.harvard.edu) is a professor of government at Harvard University and fellow-in-residence at the Center for Advanced Study in the Behavioral Sciences, Stanford University.

Please click here to read a related perspective by Mary Olson.

DOI: 10.1377/hlthaff.W4.S3
©2004 Project HOPE–The People-to-People Health Foundation, Inc.