|
|
| Home | Current Issue | Archives | Topic Collections | Search | Blog | Subscribe | Contact Us | Help |
Value Of Innovation |
Physicians’ Views Of The Relative Importance Of Thirty Medical Innovations
| Abstract |
|---|
In response to a mail survey, 225 leading general internists provided their opinions of the relative importance to patients of thirty medical innovations. They also provided information about themselves and their practices. Their responses yielded a mean score and a variability score for each innovation. Mean scores were significantly higher for innovations in procedures than in medications and for innovations to treat cardiovascular disease than for those to treat other diseases. The rankings were similar across subgroups of respondents, but the evaluations of a few innovations were significantly related to physicians’ age. The greatest variability in response was usually related to the physician’s patient mix.
During the past thirty years an unprecedented number of innovations have had great clinical and economic importance for U.S. medicine. New medications, new diagnostic techniques, and new surgical procedures have helped millions of patients to live longer, better-quality lives. At the same time, leading health economists believe that technological advance is the major cause of rising expenditures.1 The need to compare the value to patients of new technologies with their effect on spending is a major source of tension among physicians, hospitals, patients, insurance companies, and government policymakers.
The efficacy and safety of most innovations have been studied through randomized clinical trials. In addition, there have been numerous-attempts to calculate the cost-effectiveness of specific interventions for well-defined clinical conditions.2 There does not, however, seem to be any systematic information about different innovations’ relative importance to patients. Because no patient has direct experience with more than a subset of technologies, it is not possible for patients themselves to make comparisons across a larger set. Similarly, physicians who specialize are not able to compare different technologies that are applied to a wide variety of health problems. Primary care physicians, who see the effects of many different interventions on their patients, are probably in the best position to make such comparisons. This paper presents the results of an initial effort to provide an assessment of thirty innovations through a survey of leading general internists.
| Design Of The Study |
|---|
 Top Design Of The Study Results Of The SurveyDiscussionNOTES |
|---|
The innovations. Our study focuses on thirty major innovations, a number large enough to permit meaningful comparisons across innovations but small enough to be manageable by respondents. The innovations were chosen by an electronic search of the Journal of the American Medical Association and the New England Journal of Medicine for the past twenty-five years, based on the frequency with which the innovations were the principal focus of published articles. An adjustment was made to include more recent innovations that could have been the subject of articles for only a few years. We modified and supplemented the resulting list according to our judgment concerning the clinical and economic importance of particular innovations. We do not claim that the thirty innovations chosen for the survey are unambiguously the most important ones of the past thirty years; the survey invited respondents to suggest omitted innovations that they thought were particularly important. Only 2 percent did so, and no omitted innovation was mentioned by more than one physician.
The survey population. We chose the survey population from two sources. First, the governors of the sixty-five U.S. chapters of the American College of Physicians–American Society of Internal Medicine (ACP-ASIM) were asked to nominate two to four physicians whom they considered to be among the leading general internists in their chapters. Second, a list of best physicians compiled by Castle Connolly Medical Ltd. was used, choosing those in the category "primary care–internal medicine." Among other qualifications, physicians on the Castle Connolly list were those whom other physicians said they would choose for their own family care.3 Physicians who graduated from medical school after 1980 were excluded because it was thought important to have physicians who had an extensive opportunity to observe the effects of innovations on patients and some familiarity with interventions that predated the innovations. Also excluded were physicians who spent less than half of their work time in face-to-face patient care as a generalist.
The questions.
Outside medicine, a variety of strategies have been used to determine the relative importance of different innovations: surveys of experts; counts of citations to patents; and economic revenues, profits, or consumer surplus.4 There is no consensus regarding the best approach. In this study the survey instrument (Exhibit 1
) listed the thirty innovations in alphabetical order, and each respondent was asked to consider how adverse the effect on their patients would be if the innovation were not available. They were asked to select the five to seven innovations have the most adverse effects and the probably have the least adverse on the absolute efficacy or effectiveness abstract, but on the benefit provided by PSA is prostate-specific antigen. SSRIs are selective serotonin reuptake inhibitors. CPK is an innovation relative to the best alternative intervention.5 Respondents were asked to consider the innovation’s likely effect on length and quality of life, taking into account the proportion of patients in their practice that would be affected.
|
Physicians also were asked to provide information about their age and sex; the number of physicians in their practice; and the percentage of practice time devoted to patients age sixty-five or older, to male patients, and to Medicaid patients. Geographical location was inferred from physicians’ ZIP codes. A cover letter, sent on ACP-ASIM stationery, emphasized our intention to include only physicians who spent at least half of their time in face-to-face patient care as a generalist and promised anonymity to the respondents. The initial mailing was sent during the week of 20 January 2001. Physicians who did not respond to the initial survey were sent a follow-up letter five weeks later.
| Results Of The Survey |
|---|
| TopDesign Of The StudyResults Of The SurveyDiscussionNOTES |
|---|
Response rate. The involvement of the ACP-ASIM and the fact that the survey was only one page long helped to produce an excellent response rate: a mailing of 387 yielded 274 replies (73 percent). This compares very favorably with most surveys of physicians and far surpasses the 50 percent response to a 1996 survey of leading economists.6 Of the 274 replies, thirty physicians ruled themselves as ineligible because they didn’t spend the requisite time in face-to-face patient care as a generalist; twelve replies could not be used because they were not marked correctly; and seven replies arrived too late to be included. Thus, the statistical analyses are based on 225 replies.
Characteristics of survey respondents.
Respondents were predominantly male and considerably older than the average American physician, and almost half practiced with fewer than five physician colleagues (Exhibit 2). Almost 60 percent said that they devoted more than half of their practice time to patients age sixty-five and older, but only one-fourth devoted more than 5 percent of their time to Medicaid patients. All four regions of the United States were well represented, with a heavy concentration in very large metropolitan areas. The respondents were not and were never intended to be a representative sample of all U.S. physicians. It is their experience, distinction among their peers, and active involvement in patient care that make their views credible and important.
|
Mean score. The mean score for an innovation was calculated by assigning a value of 1.0 if the innovation was selected as having a most adverse effect if it were unavailable, a value of 0.0 if it was placed in the least category, and 0.5 if it was neither most nor least (Exhibit 3
). Inasmuch as respondents chose somewhat more innovations in the "most" category than in the "least" category, the mean for all thirty innovations is 0.520, with a standard deviation of 0.02. (Scores above 0.56 or below 0.48 are statistically significant at the 
95 percent confidence level.) The most important innovation by a considerable margin is magnetic resonance imaging (MRI) and computed tomography (CT) scanning, and the innovation whose absence would have the least adverse effect on patients is bone marrow transplant.
|
Innovations that take the form of medications have a statistically significantly lower mean score (0.473) than do the diagnostic innovations (0.570) or the surgical innovations (0.582). When the innovations disease group, those that are used to treat are clustered by cardiovascular disease have a significantly higher mean score (0.625) than do those used for the treatment of malignant neoplasms (0.497) or other disease groups (0.490).
To determine whether innovations that have their effect primarily on length rather than quality of life are evaluated more highly, seven internists who did not participate in the survey and were blind to the results rated each of the innovations on a ten-point scale where ten indicated "length of life only" and one indicated "quality of life only." A mean rating for each innovation was calculated, and the innovations were divided into two categories of fifteen innovations each, according to mean rating. The survey respondents evaluated innovations that primarily affect length of life somewhat higher than they did those that primarily affect quality of life (mean scores are 0.558 versus 0.483, respectively).
The order in which the innovations were listed on the survey did not appear to affect the evaluations. The sum of ranks for the fifteen innovations listed first, on the left-hand side of the page, was 234; the sum of ranks for the second fifteen, on the right-hand side of the page, was 231. The allocation of "mosts" and "leasts" was also not related to the order of questions. The first fifteen innovations produced almost half of the former (48.4 percent) and slightly more than half of the latter (52.7 percent).
One reassuring result of the survey is the fact that various subgroups of respondents all had similar rankings. The coefficients of rank correlation measure the extent of similarity between any two groups of physicians. If the rankings are identical, the coefficient is 1.0. If the rankings are opposite, the coefficient is –1.0, and if the rankings are unrelated, the coefficient is 0.0. The correlation in our study was always above 0.90 and usually above 0.95 (Exhibit 4). None of the coefficients were significantly different from 1.0. In short, respondents in different locations and different kinds of practices all tended to provide similar assessments of the thirty innovations. The lowest correlation was between male and female physicians (0.907), but this is at least partly explained by the small number of female respondents (twenty). The standard deviation of the mean score of females is 0.07, which implies that there is probably considerable random variation in their rankings. The comparisons based on physician’s age and on percentage of practice time spent with Medicaid patients also show below-average correlations. Worthy of note is the very high correlation (0.98) between the first half of responses and the second half. Some survey researchers believe that a significant difference in rankings between early and late respondents suggests that the nonrespondents might differ even more.
|
Variability in evaluations. Although the physicians’ assessments were similar regardless of geographic location, personal characteristics, or type of practice, there was some variability for certain innovations (Exhibit 5
). The variability score for each innovation was calculated by summing across respondents the square of the difference between the individual respondent’s value (that is, 1.0, 0.5, or 0.0) and the mean score of the innovation, multiplying by 4 and dividing by N. This variability score has a potential range from zero (all responses identical) to 1.0 (half of the respondents choosing "most" and half choosing "least"). The greatest variability is for human immunodeficiency virus (HIV) testing and treatment; three other innovations—cataract extraction, prostate-specific antigen (PSA) testing, and opioids—also show high variability.
|
Through correlation and regression analysis (detailed tables with tests of statistical significance are available upon request), we explored the possibility that high variability was related to differences among respondents in the characteristics of their patients (Exhibit 6
). We see that the mean score for HIV testing and treatment rises of Medicaid patients increases. Physicians sharply as the percentage with no Medicaid patients gave it a mean score of only 0.37, while physicians who devoted at least 15 percent of their practice time to Medicaid patients gave it a mean score of 0.60. Also notable is the decrease in mean score for HIV testing and treatment as the percentage of patients age sixty-five and older increases. The score for cataract extraction and lens implant is positively related to both percentage of Medicaid and percentage of elderly patients, but the relationship is less clear-cut than between HIV and percentage of Medicaid patients. The PSA score is much lower if the percentage of Medicaid patients is 6 or higher; the relationship to percentage of elderly patients is mixed. The evaluation of opioids does not show a strong relation to any of the patient characteristics.
|
Data analyses also revealed that the evaluation of several innovations varied significantly with the age of the physician (Exhibit 7
). We see that the mean score of the new antidepressants declines very sharply with age. Physicians age fifty or under gave this innovation a score of 0.746, while physicians over age sixty evaluated it at 0.578. The newer gastrointestinal drugs (proton pump inhibitors and H2 blockers) and bone densitometry also show a significant decline in mean score between younger and older physicians. Ultrasonography (including echocardiography) shows the reverse pattern: The mean score rises with physician’s age.
|
| Discussion |
|---|
| TopDesign Of The StudyResults Of The SurveyDiscussionNOTES |
|---|
Most studies of medical innovations are conducted by specialists; they focus on only one innovation at a time; and an innovation’s effects are usually judged in the light of goals set by researchers. In this survey, generalists were asked to compare many innovations simultaneously and, to the best of their ability, consider the effects of the innovations on their patients.
Complexity of survey question. We are well aware of the complexity of the question we posed to the survey population. It is often extremely difficult to rank alternative treatments for the same medical problem because possible differences in mortality, complications, side effects, relief of symptoms, and functional improvements must be considered simultaneously. Comparisons of interventions for different medical problems are even more difficult. Nevertheless, we believe that a start must be made because the information gathered by this survey raises interesting questions and could stimulate research on issues such as continuing physician education, the deployment of medical resources, and investment in research and development.
Strong consensus. The high response rate to our survey of leading general internists indicates that physicians are willing to give their views regarding the relative importance to their patients of different medical innovations. Their responses form a systematic pattern, with most innovations receiving scores significantly higher or lower than would be expected by chance. Moreover, subgroups of the population all show similar patterns of response. This strong consensus could be the result of similar observations of the effects of these innovations on patients or similar exposure to the medical literature, or both. It would be of interest to know whether rankings by other generalists or by specialists would be similar to those reported here and whether the level of consensus would be as great.
Applications of the rankings. Practice style. One possible application of the rankings would be in the evaluation of physicians’ uses of innovations. If some physicians make consistently above-average use of innovations with low ranking (unexplained by patient-mix) or below-average use of high-ranking innovations, a closer examination of their practice patterns might be warranted.
Quality assessment.
The rankings might also contribute to an expanded approach to quality assessment. The National Committee for Quality Assurance (NCQA) seems to use strong evidence of efficacy as a prime criterion for choosing its quality measures. This policy encourages health plans to promote the use of innovations that improve health outcomes in clinical trials. The rankings in Exhibit 3
could provide another form of evidence, that of perceived value to patients, which is, or ought to be, an important aspect of quality.
Research and development. The rankings also could help to inform policies concerning research and development. Innovations in diagnostic and surgical procedures tended to receive significantly higher rankings than innovations in medications, although there were some exceptions. The importance that the internists ascribed to innovations in diagnostic and surgical procedures highlights the need to understand the scientific and technological foundations of advances in medicine. It may be that such advances are more dependent on research in physics, engineering, and related fields than on "medical" research narrowly defined.
Cardiovascular treatments. Innovations designed for the treatment of cardiovascular disease received significantly higher ratings than did other innovations. This probably reflects both the high incidence of cardiovascular disease (the leading cause of death in the United States) and the greater efficacy of new cardiovascular procedures and medications relative to innovations that address other major diseases such as malignant neoplasms.
Outliers. Although the rankings are similar across subgroups of respondents, evaluations of a few innovations show considerable variability. In most cases, high variability is related to differences among physicians in patient mix. For instance, the evaluation of HIV testing and treatment is strongly positively related to the percentage of practice time devoted to Medicaid patients. Two other innovations with high variability scores, cataract extraction with lens implant and PSA testing, got relatively low evaluations from physicians who spend less than 25 percent of their practice time with patients age sixty-five and older.
Areas for future research. There are a few innovations whose mean scores vary considerably with the age of the physician. Reasons for these variations should be explored. For example, older physicians may not think that the new antidepressants are as valuable as their younger colleagues do because they do not diagnose depression as frequently. Alternatively, they may make this diagnosis as frequently but think that the older antidepressants are as good or almost as good as the newer ones. A more highly focused study than this broad survey could test these hypotheses.
Another promising area for research is the diffusion of innovations. Did the innovations with high mean scores diffuse more rapidly than those with low scores? What are the factors that help or hinder the diffusion process? Also of interest are innovations’ economic consequences. For example, is the ranking based on respondents’ evaluations similar to a ranking based on expenditures? Are there some innovations that generate a higher level of spending than would be predicted by their ranking in the survey? If there are, what accounts for the difference?
Finally, health policy specialists need to pay more attention to the distributive consequences of innovations. How do the benefits vary among different groups defined by age, sex, ethnicity, education, income, and other characteristics? Are the distributive consequences the same for innovations with high and low rankings? This initial effort to obtain information about the relative importance to patients of thirty innovations should provide a stimulus for addressing many questions about innovations relevant to health policy.
| Editor's Notes |
|---|
Victor Fuchs is the Henry J. Kaiser Jr. Professor Emeritus at Stanford University and a research associate at the National Bureau of Economic Research. Harold Sox was a professor of medicine at Dartmouth, where he chaired the Department of Medicine. In 2001 he became the editor of Annals of Internal Medicine.
The authors thank Sarah Rosen for outstanding research assistance, Byron Wm. Brown Jr. and Persi Diaconis for statistical advice, Claire Gilchrist and Rossannah Reeves for administering the survey, and numerous colleagues at Dartmouth and Stanford for helpful comments on many issues. Victor Fuchs is also pleased to acknowledge financial support from the Robert Wood Johnson and Henry J. Kaiser Family Foundations.
| NOTES |
|---|
| TopDesign Of The StudyResults Of The SurveyDiscussionNOTES |
|---|
- See, for example, V.R. Fuchs, "Economics, Values, and Health Care Reform," American Economic Review (March 1996): 1–24.
- T.O. Tengs et al., "Five Hundred Life-Saving Interventions and Their Cost-Effectiveness," Risk Analysis 15, no. 3 (1995): 369–390.
- Castle Connolly Medical Ltd. is an Internet-based source of information about health care providers. For its selection procedures, see <www.castleconnolly.com>.
- See, for example, Z.J. Acs and D.B. Audretsch, Innovation and Small Firms (Cambridge, Mass.: MIT Press, 1990)
; F.M. Scherer, "The Size Distribution of Profits from Innovation," in Economics and Econometrics of Innovation, ed. D. Encaoua et al. (Amsterdam: Kluwer Publishing, 2000) ; and D. Harhoff et al., "Citation Frequency and the Value of Patented Inventions," Review of Economics and Statistics 81, no. 3 (1999): 511–515. - This corresponds to the economists’ concept of "marginal" or "incremental" benefit, a concept deemed most relevant for analytical purposes.
- V.R. Fuchs, A.B. Krueger, and J.M. Poterba, "Economists’ Views about Parameters, Values, and Policies: Survey Results in Labor and Public Economics," Journal of Economic Literature (September 1998): 1387–1425. To be sure, the survey sent to the economists was much longer and more complicated.
- T.O. Tengs et al., "Five Hundred Life-Saving Interventions and Their Cost-Effectiveness," Risk Analysis 15, no. 3 (1995): 369–390.
This article has been cited by other articles:
 |
|
 |
|
  D. I. Bulas, M. J. Goske, K. E. Applegate, and B. P. Wood Image Gently: Why We Should Talk to Parents About CT in Children Am. J. Roentgenol., May 1, 2009; 192(5): 1176 - 1178. [Full Text] [PDF]
|
|
|
|
|
|
R. B. Gunderman and P. J. Mehta Continue Drawing Back the Curtain: Educating the Community About Radiology Am. J. Roentgenol., May 1, 2009; 192(5): 1388 - 1390. [Full Text] [PDF]
|
|
|
|
 |
|
 R. Smith-Bindman, D. L. Miglioretti, and E. B. Larson Rising Use Of Diagnostic Medical Imaging In A Large Integrated Health System Health Aff., November 1, 2008; 27(6): 1491 - 1502. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. L. Schur and M. L. Berk Views On Health Care Technology: Americans Consider The Risks And Sources Of Information Health Aff., November 1, 2008; 27(6): 1654 - 1664. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 E. A. Zerhouni Major Trends in the Imaging Sciences: 2007 Eugene P. Pendergrass New Horizons Lecture Radiology, November 1, 2008; 249(2): 403 - 409. [Full Text] [PDF]
|
|
|
|
 |
|
 P. Varkey, A. Horne, and K. E. Bennet Innovation in Health Care: A Primer American Journal of Medical Quality, September 1, 2008; 23(5): 382 - 388. [Abstract] [PDF]
|
|
|
|
|
|
J. A. Ruiz and G. M. Glazer The State of Radiology in 2006: Very High Spatial Resolution but No Visibility Radiology, October 1, 2006; 241(1): 11 - 16. [Full Text] [PDF]
|
|
|
|
 |
|
 J. K. Iglehart The new era of medical imaging--progress and pitfalls. N. Engl. J. Med., June 29, 2006; 354(26): 2822 - 2828. [Full Text] [PDF]
|
|
|
|
 |
|
 A S K Dzik-Jurasz Pelvic malignancy: integrating form and function Br. J. Radiol., October 1, 2005; 78(Special_Issue_2): S86 - S93. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 T. Bodenheimer High and Rising Health Care Costs. Part 2: Technologic Innovation Ann Intern Med, June 7, 2005; 142(11): 932 - 937. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. E. Liebeskind, P. H. Arger, A. Liebeskind, K. Maston, and C. Langlotz Using Sonography to Examine Adult Patients at an Academic Medical Center: Have Usage Patterns Changed with the Expansion of Managed Care? Am. J. Roentgenol., December 1, 2002; 179(6): 1395 - 1399. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. Chan The Importance of Strategy for the Evolving Field of Radiology Radiology, September 1, 2002; 224(3): 639 - 648. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. M. Garber, C. Mulrow, and H. C. Sox Articles on Diagnostic Tests: A New Series and an Invitation to Authors Ann Intern Med, August 20, 2002; 137(4): 288 - 289. [Full Text] [PDF]
|
|
| Home | | | Current Issue | | | Archives | | | Topic Collections | | | Search | | | Blog | | | Subscribe | | | Contact Us | | | Help |
|
© 2001-2001 Project
HOPE–The People-to-People Organization |
||||||||||||||||