This Article Abstract Figures Only Reprint (PDF) Submit a response to this article Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services E-mail this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to My Personal Archive Download to Citation Manager Reprints & Permissions Citing Articles Citing Articles via HighWire Citing Articles via ISI Web of Science (34) Citing Articles via Google Scholar Google Scholar Articles by Fuchs, V. R. Articles by Sox, H. C. Search for Related Content PubMed PubMed Citation Articles by Fuchs, V. R. Articles by Sox, H. C., Jr. Related Collections Pharmaceuticals Physicians Research And Technology

Value Of Innovation

Physicians’ Views Of The Relative Importance Of Thirty Medical Innovations

Abstract

 
In response to a mail survey, 225 leading general internists provided their opinions of the relative importance to patients of thirty medical innovations. They also provided information about themselves and their practices. Their responses yielded a mean score and a variability score for each innovation. Mean scores were significantly higher for innovations in procedures than in medications and for innovations to treat cardiovascular disease than for those to treat other diseases. The rankings were similar across subgroups of respondents, but the evaluations of a few innovations were significantly related to physicians’ age. The greatest variability in response was usually related to the physician’s patient mix.

The efficacy and safety of most innovations have been studied through randomized clinical trials. In addition, there have been numerous-attempts to calculate the cost-effectiveness of specific interventions for well-defined clinical conditions.² There does not, however, seem to be any systematic information about different innovations’ relative importance to patients. Because no patient has direct experience with more than a subset of technologies, it is not possible for patients themselves to make comparisons across a larger set. Similarly, physicians who specialize are not able to compare different technologies that are applied to a wide variety of health problems. Primary care physicians, who see the effects of many different interventions on their patients, are probably in the best position to make such comparisons. This paper presents the results of an initial effort to provide an assessment of thirty innovations through a survey of leading general internists.

Design Of The Study

Top Design Of The Study Results Of The Survey Discussion NOTES

 
The innovations. Our study focuses on thirty major innovations, a number large enough to permit meaningful comparisons across innovations but small enough to be manageable by respondents. The innovations were chosen by an electronic search of the Journal of the American Medical Association and the New England Journal of Medicine for the past twenty-five years, based on the frequency with which the innovations were the principal focus of published articles. An adjustment was made to include more recent innovations that could have been the subject of articles for only a few years. We modified and supplemented the resulting list according to our judgment concerning the clinical and economic importance of particular innovations. We do not claim that the thirty innovations chosen for the survey are unambiguously the most important ones of the past thirty years; the survey invited respondents to suggest omitted innovations that they thought were particularly important. Only 2 percent did so, and no omitted innovation was mentioned by more than one physician.

The survey population. We chose the survey population from two sources. First, the governors of the sixty-five U.S. chapters of the American College of Physicians–American Society of Internal Medicine (ACP-ASIM) were asked to nominate two to four physicians whom they considered to be among the leading general internists in their chapters. Second, a list of best physicians compiled by Castle Connolly Medical Ltd. was used, choosing those in the category "primary care–internal medicine." Among other qualifications, physicians on the Castle Connolly list were those whom other physicians said they would choose for their own family care.³ Physicians who graduated from medical school after 1980 were excluded because it was thought important to have physicians who had an extensive opportunity to observe the effects of innovations on patients and some familiarity with interventions that predated the innovations. Also excluded were physicians who spent less than half of their work time in face-to-face patient care as a generalist.

The questions. Outside medicine, a variety of strategies have been used to determine the relative importance of different innovations: surveys of experts; counts of citations to patents; and economic revenues, profits, or consumer surplus.⁴ There is no consensus regarding the best approach. In this study the survey instrument (Exhibit 1) listed the thirty innovations in alphabetical order, and each respondent was asked to consider how adverse the effect on their patients would be if the innovation were not available. They were asked to select the five to seven innovations have the most adverse effects and the probably have the least adverse on the absolute efficacy or effectiveness abstract, but on the benefit provided by PSA is prostate-specific antigen. SSRIs are selective serotonin reuptake inhibitors. CPK is an innovation relative to the best alternative intervention.⁵ Respondents were asked to consider the innovation’s likely effect on length and quality of life, taking into account the proportion of patients in their practice that would be affected.

Results Of The Survey

Top Design Of The Study Results Of The Survey Discussion NOTES

 
Response rate. The involvement of the ACP-ASIM and the fact that the survey was only one page long helped to produce an excellent response rate: a mailing of 387 yielded 274 replies (73 percent). This compares very favorably with most surveys of physicians and far surpasses the 50 percent response to a 1996 survey of leading economists.⁶ Of the 274 replies, thirty physicians ruled themselves as ineligible because they didn’t spend the requisite time in face-to-face patient care as a generalist; twelve replies could not be used because they were not marked correctly; and seven replies arrived too late to be included. Thus, the statistical analyses are based on 225 replies.

Characteristics of survey respondents. Respondents were predominantly male and considerably older than the average American physician, and almost half practiced with fewer than five physician colleagues (Exhibit 2). Almost 60 percent said that they devoted more than half of their practice time to patients age sixty-five and older, but only one-fourth devoted more than 5 percent of their time to Medicaid patients. All four regions of the United States were well represented, with a heavy concentration in very large metropolitan areas. The respondents were not and were never intended to be a representative sample of all U.S. physicians. It is their experience, distinction among their peers, and active involvement in patient care that make their views credible and important.

To determine whether innovations that have their effect primarily on length rather than quality of life are evaluated more highly, seven internists who did not participate in the survey and were blind to the results rated each of the innovations on a ten-point scale where ten indicated "length of life only" and one indicated "quality of life only." A mean rating for each innovation was calculated, and the innovations were divided into two categories of fifteen innovations each, according to mean rating. The survey respondents evaluated innovations that primarily affect length of life somewhat higher than they did those that primarily affect quality of life (mean scores are 0.558 versus 0.483, respectively).

The order in which the innovations were listed on the survey did not appear to affect the evaluations. The sum of ranks for the fifteen innovations listed first, on the left-hand side of the page, was 234; the sum of ranks for the second fifteen, on the right-hand side of the page, was 231. The allocation of "mosts" and "leasts" was also not related to the order of questions. The first fifteen innovations produced almost half of the former (48.4 percent) and slightly more than half of the latter (52.7 percent).

One reassuring result of the survey is the fact that various subgroups of respondents all had similar rankings. The coefficients of rank correlation measure the extent of similarity between any two groups of physicians. If the rankings are identical, the coefficient is 1.0. If the rankings are opposite, the coefficient is –1.0, and if the rankings are unrelated, the coefficient is 0.0. The correlation in our study was always above 0.90 and usually above 0.95 (Exhibit 4). None of the coefficients were significantly different from 1.0. In short, respondents in different locations and different kinds of practices all tended to provide similar assessments of the thirty innovations. The lowest correlation was between male and female physicians (0.907), but this is at least partly explained by the small number of female respondents (twenty). The standard deviation of the mean score of females is 0.07, which implies that there is probably considerable random variation in their rankings. The comparisons based on physician’s age and on percentage of practice time spent with Medicaid patients also show below-average correlations. Worthy of note is the very high correlation (0.98) between the first half of responses and the second half. Some survey researchers believe that a significant difference in rankings between early and late respondents suggests that the nonrespondents might differ even more.

View larger version (16K): [in this window] [in a new window] EXHIBIT 7 Mean Scores Of Selected Innovations, By Age Of Physician, 2001

Top Design Of The Study Results Of The Survey Discussion NOTES

Complexity of survey question. We are well aware of the complexity of the question we posed to the survey population. It is often extremely difficult to rank alternative treatments for the same medical problem because possible differences in mortality, complications, side effects, relief of symptoms, and functional improvements must be considered simultaneously. Comparisons of interventions for different medical problems are even more difficult. Nevertheless, we believe that a start must be made because the information gathered by this survey raises interesting questions and could stimulate research on issues such as continuing physician education, the deployment of medical resources, and investment in research and development.

Strong consensus. The high response rate to our survey of leading general internists indicates that physicians are willing to give their views regarding the relative importance to their patients of different medical innovations. Their responses form a systematic pattern, with most innovations receiving scores significantly higher or lower than would be expected by chance. Moreover, subgroups of the population all show similar patterns of response. This strong consensus could be the result of similar observations of the effects of these innovations on patients or similar exposure to the medical literature, or both. It would be of interest to know whether rankings by other generalists or by specialists would be similar to those reported here and whether the level of consensus would be as great.

Applications of the rankings. Practice style. One possible application of the rankings would be in the evaluation of physicians’ uses of innovations. If some physicians make consistently above-average use of innovations with low ranking (unexplained by patient-mix) or below-average use of high-ranking innovations, a closer examination of their practice patterns might be warranted.

Quality assessment. The rankings might also contribute to an expanded approach to quality assessment. The National Committee for Quality Assurance (NCQA) seems to use strong evidence of efficacy as a prime criterion for choosing its quality measures. This policy encourages health plans to promote the use of innovations that improve health outcomes in clinical trials. The rankings in Exhibit 3 could provide another form of evidence, that of perceived value to patients, which is, or ought to be, an important aspect of quality.

Research and development. The rankings also could help to inform policies concerning research and development. Innovations in diagnostic and surgical procedures tended to receive significantly higher rankings than innovations in medications, although there were some exceptions. The importance that the internists ascribed to innovations in diagnostic and surgical procedures highlights the need to understand the scientific and technological foundations of advances in medicine. It may be that such advances are more dependent on research in physics, engineering, and related fields than on "medical" research narrowly defined.

Cardiovascular treatments. Innovations designed for the treatment of cardiovascular disease received significantly higher ratings than did other innovations. This probably reflects both the high incidence of cardiovascular disease (the leading cause of death in the United States) and the greater efficacy of new cardiovascular procedures and medications relative to innovations that address other major diseases such as malignant neoplasms.

Outliers. Although the rankings are similar across subgroups of respondents, evaluations of a few innovations show considerable variability. In most cases, high variability is related to differences among physicians in patient mix. For instance, the evaluation of HIV testing and treatment is strongly positively related to the percentage of practice time devoted to Medicaid patients. Two other innovations with high variability scores, cataract extraction with lens implant and PSA testing, got relatively low evaluations from physicians who spend less than 25 percent of their practice time with patients age sixty-five and older.

Areas for future research. There are a few innovations whose mean scores vary considerably with the age of the physician. Reasons for these variations should be explored. For example, older physicians may not think that the new antidepressants are as valuable as their younger colleagues do because they do not diagnose depression as frequently. Alternatively, they may make this diagnosis as frequently but think that the older antidepressants are as good or almost as good as the newer ones. A more highly focused study than this broad survey could test these hypotheses.

Another promising area for research is the diffusion of innovations. Did the innovations with high mean scores diffuse more rapidly than those with low scores? What are the factors that help or hinder the diffusion process? Also of interest are innovations’ economic consequences. For example, is the ranking based on respondents’ evaluations similar to a ranking based on expenditures? Are there some innovations that generate a higher level of spending than would be predicted by their ranking in the survey? If there are, what accounts for the difference?

Finally, health policy specialists need to pay more attention to the distributive consequences of innovations. How do the benefits vary among different groups defined by age, sex, ethnicity, education, income, and other characteristics? Are the distributive consequences the same for innovations with high and low rankings? This initial effort to obtain information about the relative importance to patients of thirty innovations should provide a stimulus for addressing many questions about innovations relevant to health policy.

Editor's Notes

 
Victor Fuchs is the Henry J. Kaiser Jr. Professor Emeritus at Stanford University and a research associate at the National Bureau of Economic Research. Harold Sox was a professor of medicine at Dartmouth, where he chaired the Department of Medicine. In 2001 he became the editor of Annals of Internal Medicine.

The authors thank Sarah Rosen for outstanding research assistance, Byron Wm. Brown Jr. and Persi Diaconis for statistical advice, Claire Gilchrist and Rossannah Reeves for administering the survey, and numerous colleagues at Dartmouth and Stanford for helpful comments on many issues. Victor Fuchs is also pleased to acknowledge financial support from the Robert Wood Johnson and Henry J. Kaiser Family Foundations.

NOTES

Top Design Of The Study Results Of The Survey Discussion NOTES

 

1. See, for example, V.R. Fuchs, "Economics, Values, and Health Care Reform," American Economic Review (March 1996): 1–24.
2. T.O. Tengs et al., "Five Hundred Life-Saving Interventions and Their Cost-Effectiveness," Risk Analysis 15, no. 3 (1995): 369–390.
3. Castle Connolly Medical Ltd. is an Internet-based source of information about health care providers. For its selection procedures, see <www.castleconnolly.com>.
4. See, for example, Z.J. Acs and D.B. Audretsch, Innovation and Small Firms (Cambridge, Mass.: MIT Press, 1990); F.M. Scherer, "The Size Distribution of Profits from Innovation," in Economics and Econometrics of Innovation, ed. D. Encaoua et al. (Amsterdam: Kluwer Publishing, 2000); and D. Harhoff et al., "Citation Frequency and the Value of Patented Inventions," Review of Economics and Statistics 81, no. 3 (1999): 511–515.
5. This corresponds to the economists’ concept of "marginal" or "incremental" benefit, a concept deemed most relevant for analytical purposes.
6. V.R. Fuchs, A.B. Krueger, and J.M. Poterba, "Economists’ Views about Parameters, Values, and Policies: Survey Results in Labor and Public Economics," Journal of Economic Literature (September 1998): 1387–1425. To be sure, the survey sent to the economists was much longer and more complicated.

                      What's this?

This article has been cited by other articles:

				P. Varkey, A. Horne, and K. E. Bennet Innovation in Health Care: A Primer American Journal of Medical Quality, September 1, 2008; 23(5): 382 - 388. [Abstract] [PDF]

				J. A. Ruiz and G. M. Glazer The State of Radiology in 2006: Very High Spatial Resolution but No Visibility Radiology, October 1, 2006; 241(1): 11 - 16. [Full Text] [PDF]

				J. K. Iglehart The new era of medical imaging--progress and pitfalls. N. Engl. J. Med., June 29, 2006; 354(26): 2822 - 2828. [Full Text] [PDF]

				A S K Dzik-Jurasz Pelvic malignancy: integrating form and function Br. J. Radiol., October 1, 2005; 78(Special_Issue_2): S86 - S93. [Abstract] [Full Text] [PDF]

				T. Bodenheimer High and Rising Health Care Costs. Part 2: Technologic Innovation Ann Intern Med, June 7, 2005; 142(11): 932 - 937. [Abstract] [Full Text] [PDF]

				M. E. Liebeskind, P. H. Arger, A. Liebeskind, K. Maston, and C. Langlotz Using Sonography to Examine Adult Patients at an Academic Medical Center: Have Usage Patterns Changed with the Expansion of Managed Care? Am. J. Roentgenol., December 1, 2002; 179(6): 1395 - 1399. [Abstract] [Full Text] [PDF]

				S. Chan The Importance of Strategy for the Evolving Field of Radiology Radiology, September 1, 2002; 224(3): 639 - 648. [Abstract] [Full Text] [PDF]

				A. M. Garber, C. Mulrow, and H. C. Sox Articles on Diagnostic Tests: A New Series and an Invitation to Authors Ann Intern Med, August 20, 2002; 137(4): 288 - 289. [Full Text] [PDF]