Health Affairs, 25, no. 5 (2006): w412-w419 (Published online 5 September 2006) doi: 10.1377/hlthaff.25.w412 © 2006 by Project HOPE	New Online   Senate Health Reform Bill   Rewarding Providers   Public Option Policy Brief   Health Reform & Abortion   Delivery System Reform

This Article Abstract Reprint (PDF) Submit a response to this article Comments: View responses Alert me when this article is cited Alert me when Comments are posted Alert me if a correction is posted Services E-mail this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to My Personal Archive Download to Citation Manager Reprints & Permissions Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Galvin, R. Search for Related Content PubMed PubMed Citation Articles by Galvin, R.

Web Exclusives

INTERVIEW

Pay-For-Performance: Too Much Of A Good Thing? A Conversation With Martin Roland

Abstract

 
As the United States moves down the road of pay-for-performance (P4P), concerns about unintended consequences are foremost in the minds of policymakers. Initial results from the world’s most ambitious P4P program, the United Kingdom’s Quality and Outcomes Framework (QOF), indicate that while quality improvements exceeded expectations, so too did the amount of funds paid out, straining the National Health Service (NHS) budget. Martin Roland, one of the leading U.K. health services researchers and an adviser to the QOF, gives his views on what went right and what went wrong, and he offers his advice to the United States about using financial incentives to improve quality.

Martin Roland: I think the response has been, to some extent, dismay by health authorities [organizations that run the NHS on the local level] because the amount they’ve needed to pay over the budgeted amount is essentially unplanned-for. The response of the physicians has been that it’s not surprising that they scored as well as they did, and they have argued that care was always better than people thought it was.

The Quality And Outcomes Framework In Context

Top The Quality And Outcomes... The Data Challenge Unintended Consequences Going Forward Lessons Learned Advice To The United... NOTE

 
Galvin: Studies of quality performance in the U.K. before the framework was put in place showed overall guideline adherence that was more in the 60–80 percent range. How do you account for the marked improvement?

Roland: I think it’s important to put this scheme in context. In 1998 the government introduced a wide-ranging strategy of quality improvement, reflecting what the published literature tells us about what makes doctors change behavior. This is that you need multiple interventions, and they need to be repeated. And that’s exactly what the government did, as its strategy involved the development of national guidelines for major diseases, a process called Clinical Governance [a range of quality improvement procedures in physicians’ offices], and a national inspection process. These interventions were introduced with a wide range of financial and non-financial incentives—principally nonfinancial ones, such as feeding back information to doctors on their performance and in some cases making that information public.

Galvin: How often was it made public? And do you think that this led to quality improvement?

Roland: To the public outside of the medical community, probably less than 10 percent of the time. But it was made public within the medical community in quite a high proportion of the Primary Care Trusts [PCTs]. And I think that making it public within the medical community was probably very effective.

Galvin: In retrospect, it does look like quality was already on a significant upswing even before P4P was instituted.

Roland: We know that it was, because we published data comparing quality from 1998 to 2003; during that five-year period, there was already a major increase in quality of care for the major conditions we looked at, which were heart disease, asthma, and diabetes. So it looks like the government strategy was already having a significant impact. And even if it wasn’t improving quality of care across the wide range of diseases that is now being incentivized, what it had done was to get physicians’ practices organized so that they could collect data—for example, electronic records were coming in quite rapidly.

Galvin: Given the fact that the NHS might have underestimated the degree of quality already being attained and might have enabled more "gaming" than it intended, has there been broader public opinion that the government has added a lot of money to the health care budget just to pay doctors for what they were doing anyway?

Roland: There has been very substantial criticism along these lines, not least because the NHS has got to the end of this year with a significant financial deficit. While the cause of the deficit was due to several factors, the QOF was certainly one of them.

The QOF is interesting as an investment in health care in that by and large, the results of improved performance will be invisible to the public. So the fact that people’s blood pressure is better controlled and their cholesterol is kept down is not going to get many votes. So you could say that it’s quite visionary of the government to be prepared to put a lot of money into improving health care in a form that really wasn’t going to be a vote winner, whereas other health care improvement strategies like reducing wait times were much, much more visible.

Galvin: How was it determined how much to reward? And, as a corollary question, could these same results have been achieved for less money?

Roland: Those are interesting questions. I think the answer to the latter question is probably yes. But there are a number of issues there. First of all, prior to the scheme, GPs were paid primarily by capitation. That essentially meant that if they hired more staff and more nurses to help manage patients, they took less profit from their practices. So I think you could argue that family practitioners were previously the only NHS employees who actually took home less money if they did a good job. That was certainly the view of the British Medical Association. They wanted to reverse that, and it certainly has been reversed.

I was involved in the negotiations around the contract, and I was closely involved in developing the indicators. But I don’t get the feeling that anybody around that table had any idea quite how much money would end up being on the table. That was part of a separate political negotiation, and I was personally very surprised.

Galvin: Some have commented that structural changes in the form of hiring more nurses to follow patients were key to the quality improvement. Is that your view as well?

Roland: Yes. As a result of the QOF, there have been significant increases in the numbers of nurses employed by practices, and also an increase in administrative staff.

Galvin: Changing your practice to improve requires resources. Do GPs feel that the increase they’ve gotten actually covers what it costs to improve?

Roland: Oh, yes, and with a healthy margin. The figures aren’t quite clear yet, but it’s clear that there has been a big increase in GPs’ personal earnings this year, and it will be greatest for those physicians who’ve already invested in their practices. But physicians who had little infrastructure in place who had to start with IT [information technology] systems, who had to start employing nurses and data clerks, will certainly have taken home less.

Galvin: Have there been any surveys of GPs to see if they’re happy with the QOF and if it might induce more medical students to become GPs?

Roland: My sense is that GPs are pretty satisfied. We’ve done serial surveys of GP job satisfaction and have found that external changes, particularly reorganizations, don’t go down well with practitioners and that they’re generally associated with a reduction in job satisfaction. We don’t yet have the results of surveys carried out since the pay-for-performance scheme, but recruitment to general practice now looks as if it’s quite good.

The Data Challenge

Top The Quality And Outcomes... The Data Challenge Unintended Consequences Going Forward Lessons Learned Advice To The United... NOTE

 
Galvin: One issue that the United States is focused on is data reliability and validity. What impact did the quality of data have on the improvement in clinical quality scores?

Roland: The data that we collected from 1998–2003 were taken directly from medical records by our own staff, so we are fairly confident that care was improving in the five years up to the introduction of the pay-for-performance scheme. Within the QOF program, the single issue that is most important in relation to "what does that figure of 91 percent mean?" relates to something called exception reporting. Exception reporting was introduced on the basis that if you have some form of evidence-based guideline, nobody would ever suggest that it should be applied to all patients. So, for example, if you’ve got a diabetic who’s dying of cancer, you’re not going to try and get his cholesterol down.

In a pay-for-performance scheme, you’ve really got two options. You either set the upper limit somewhere below 100 percent, so that you are not constraining doctors to give patients inappropriate treatment, or you take the approach that the U.K. has taken, which is to allow doctors discretion to take such patients out of the equation. So the average of 91 percent guideline compliance does not relate to all patients with, say, diabetes; it relates to all patients except those whom the doctors took out.

Galvin: What kind of impact did that have on the overall percentiles?

Roland: There has been a lot of discussion about that. One of the great difficulties has been that although all of the data for the performance indicators were extracted automatically from GPs’ computer systems, they omitted to collect data on exception reporting. So there’s a big problem with the first year’s data in that there are no data on the number of exception reports, so they have to be estimated using techniques that can only be applied to about half the indicators. Our own estimates are that around 6 percent of patients were exception-reported, but with a very wide range from none in some practices to a few practices that have excepted as many as 85 percent of their patients. Now that’s a very skewed distribution, and there are very few practices with such high levels of exception reporting. But this is the key issue for Primary Care Trusts, whose responsibility it is to police and inspect the system. It is their responsibility to address the problem if they have some practices that have essentially gamed the system by exception reporting more patients than would be clinically reasonable. So, taking the figure of 91 percent achievement across all of the indicators and overall exception reporting of around 6 percent, you might say that achievement in the whole population was around 85 percent, at least for the indicators where exception reporting could be estimated.

Galvin: Sticking with data reliability for a moment, given that the government funded IT systems for GPs, could some of the improvement have been the result of simply better recording?

Roland: I think that was a big factor, in that I think doctors are now recording in a way they didn’t before.

Galvin: A major area of concern for U.S. physicians is data accuracy, particularly around case-mix adjustment. They do not want their incomes put at risk if having a greater proportion of sicker patients leads to lower scores. This has led to concerns among policymakers about sicker patients’ potentially having less access to physician practices. The QOF has no case-mix adjustments, and yet some of the practices are small enough for performance measures to be skewed by relative differences in illness burden. Was that an issue for GPs?

Roland: No. I mean, I think that GPs have argued that with some population groups, it would be more difficult to achieve the targets, but then they had considerable latitude to exception-report patients in whom targets were difficult to meet, such as patients who didn’t show up for health checks. The second issue is that by and large there’s very little cream skimming in health care in the U.K. Even in the days of fundholding, when GPs held the budget for hospital treatment and there would have been some incentive to cream-skim, it was probably only occurring on a very small scale. There hasn’t really been any suggestion that this incentive scheme has led doctors to throw patients off their lists. This is perhaps partly because there is a very strong ethos in the NHS of providing care for the whole population.

Galvin: Private-sector U.S. payers have moved aggressively to develop P4P programs, but there has not been a coordinated evaluation strategy. As a result, data about P4P’s effectiveness are lacking. What kind of formal evaluation was set up in the design of the QOF to evaluate success?

Roland: Evaluation of the QOF is part of our strategy at Manchester as a large government-funded health services research center, and there are certainly other groups involved in research on the financial incentives as well.

Unintended Consequences

Top The Quality And Outcomes... The Data Challenge Unintended Consequences Going Forward Lessons Learned Advice To The United... NOTE

 
Galvin: Unintended consequences accompany all major policy shifts. What would you say are the most important ones arising from the QOF?

Roland: I think that it’s too early to tell which are the important ones. There is definitely the potential for fragmentation of care. In my own practice, for example, most of the coronary heart disease care is now done in a clinic by nurses, overseen by physicians. So I could potentially become deskilled in that area because I’m not managing that problem as frequently as I was.

GPs have always regarded the fact that they look after the whole person as being a very important part of their jobs. Whereas they have always characterized specialists as being "interested in a disease and not the person" and they are physicians "interested in the person, rather than the disease," there is concern that the focus in primary care is now shifting toward diseases rather than patients.

Galvin: One of the big concerns is "managing to the measure," leading to the situation in which the conditions not incentivized will not get the same level of attention as those that result in rewards. Do we know if this has occurred?

Roland: No, we don’t. There certainly are studies going on, but they’ve not been reported on yet, and some of it may be very subtle. Say, for example, if you come to see me with hypertension, it may just be that I’m more interested in taking your blood pressure as the first thing I do, rather than asking if you’ve had any side effects of the medication. So you might just not get around to telling me that you’re having problems with potency with the medication. It will be very hard to detect that sort of thing.

Galvin: One of the potential ways to measure that would be to ask patients if they felt that there was undue emphasis on certain conditions and less view of them as a whole. Yet in the QOF, although there were patient surveys, no rewards were tied to the survey results.

Roland: Absolutely not.

Galvin: Why not, and what’s the plan on that going forward?

Roland: Why not is that the British Medical Association was pretty nervous about introducing the concept of patient surveys to their doctors. They thought that the notion that physicians were essentially running a service industry where the customers’ views mattered would be quite hard for the doctors to adjust to. And so they negotiated a reward system where, essentially, doctors got a substantial reward, like eight or nine thousand dollars, for simply doing a survey and doing nothing with it. And there was an additional reward for making a plan as to what one might do with the survey results and a further reward for involving patients in those discussions, so there was no suggestion of paying doctors based on points.

Now, the next question is, Will the U.K. take the next step of paying against questionnaire scores? This certainly can’t be done with the current methodology because actually there are no constraints whatsoever in terms of how the current questionnaires are administered. However, the next iteration, in April of this year, has just been announced. And they’re going to do precisely that. To reward doctors for access, they’re going to have an independently conducted survey, and they will then pay against the patients’ scores.

Galvin: A particularly feared consequence voiced by physicians is that tying financial rewards to patient care will inexorably lead to negative effects on professionalism. Has there been evidence of this?

Roland: Some people have observed doctors’ keenness to get the last point. It’s almost as if they’ve been trained to jump through hoops ever since they were teenagers to pass exams, go to medical school, pass more exams, and then you show them another set of hoops to jump through and off they go. So it’s been very interesting to see how physicians have responded. They certainly have responded to the incentives offered. One of the important questions is what providing external financial incentives does to internal motivation. There is quite a lot of evidence from the psychological, sociological, and economic literature that increasing external incentives reduces internal motivation.

If you’re thinking about potential adverse consequences of pay-for-performance, the most damaging in the long term could be if you ended up with a system where, essentially, doctors only did anything because they were paid for it and had lost their professional ethos. And that, in a sense, is one of the reasons why I think it’s so important that incentives be aligned with professional values—to avoid distorting those values. But I still think that it’s a risk.

Going Forward

Top The Quality And Outcomes... The Data Challenge Unintended Consequences Going Forward Lessons Learned Advice To The United... NOTE

 
Galvin: What are some of the major changes that have been made for the next version of the QOF?

Roland: The system will include a wider range of conditions: There are more indicators for mental health, and there are indicators for depression, dementia, learning disability, atrial fibrillation, chronic renal disease, and obesity.

Galvin: In the new measures, has there been consideration of care coordination?

Roland: Not really, due, I suspect, to the difficulty in measuring it. Coordination and continuity are two things that could suffer, again coming back to the fragmentation and the disease groups that we talked about before. But there’s nothing in the original or the reworked indicators that would incentivize continuity or coordination.

Galvin: One of the problems we see in the States is the lack of coordination between the discharge instructions given at the end of a hospital stay and what happens subsequently in the primary care physician’s office. It’s going to be difficult to get to the level of improvement we want to see in our health care systems without measuring this and improving it.

Roland: That certainly is a problem in the NHS, too, and there are approaches to dealing with it, but not through this pay-for-performance scheme. The mechanism that government sees as trying to improve coordination between sectors is a return to giving family practitioners the budget for specialist care—so-called practice base commissioning. And you would expect that if primary care physicians have budgets for specialty care, coordination would be high on their list of priorities.

Galvin: What about the pricing or the bonuses in the subsequent years of the quality outcome? Will there be another 30 percent increase in the GPs’ income?

Roland: No, there’s no new money in this present round. That’s not surprising, because the government clearly thinks that it had to pay in the first round. Some indicators have been taken out, but levels have generally been raised—the floor levels have been raised and some of the top levels have been raised—so there’s been, as you would expect, a revision. There will be no further revision next year, but a major one is planned for 2008.

Lessons Learned

Top The Quality And Outcomes... The Data Challenge Unintended Consequences Going Forward Lessons Learned Advice To The United... NOTE

 
Galvin: Although we touched on the financial impact of the QOF earlier, accurate budgeting is of great interest in the States. Could you go into more detail about the root causes of the underbudgeting?

Roland: When you’re starting a pay-for-performance scheme, you’ve got to make assumptions about current levels of performance and likely achievement. Government essentially guessed that the GPs would average 750 points in the first year, and that’s what it put in the budget. The data have now come out about the extent to which care improved from 1998 to 2003, so, in retrospect, government’s assumption about the baseline was probably wrong. Care was already improving, and government’s assumption about how hard GPs would work to meet the new targets was also probably wrong.

Galvin: Do you think that the amount of reward to attain certain measures was out of synch with the effort it took to attain them?

Roland: Coming back to your earlier question, the government, in retrospect, paid out more than it needed to, to achieve the levels of quality, but nobody knows how much more.

Galvin: Although I’m sure the British public is not engaged at the level of detail that we’re discussing, has their experience with the NHS been affected as a result of the QOF?

Roland: There are initiatives other than the QOF, and in general, I would say that access to specialist services has increased, and waiting times have come down—in many cases quite dramatically. But public expectations have also risen, which must be desperately frustrating for a government if you’re producing a service that is improving and the public expectations are rising at a faster rate, so that the public thinks that they’re getting less when in fact they’re getting more.

Galvin: This sounds like a description of the American public. And I wonder if the U.K. might not be beginning to face up to a very "American" problem. Now that the percentage of gross domestic product accounted for by health care has increased, there is a higher base upon which medical inflation is going to compound. You might be facing a situation similar to what we contend with in the States, which is how to improve quality while controlling costs. Our framework for achieving this is to combine measures of effectiveness with measures of efficiency, focusing on what we are calling "value." Has there been any consideration to measurements of efficiency or resources used per point achieved?

Roland: I think that it’s certainly something that is exercising government, especially the Treasury. Government would like to be able to quantify the benefits in some readily measurable form. Much of it is not easily quantifiable in that form, and, therefore, there are no very good measures of efficiency. It is particularly difficult, because at the beginning, I said that the pay-for-performance scheme was one part of a wide range of quality improvement initiatives in the U.K. In a sense, you shouldn’t he able to disentangle them.

So to take the example of chest pain, one of the initiatives brought in during the late 1990s was "rapid access chest pain clinics," to which physicians would be able to refer patients with recent-onset chest pain to a clinic where they’d get all of their tests done within a very short period of time. That is clearly a good thing and likely to be effective. But in terms of impact on heart disease mortality, you can’t disentangle it from better routine care of patients with angina.

Galvin: Is there any sense, looking back on the part of the NHS, that they wish they had either waited to see the results of a baseline year or piloted first to see what the results would be?

Roland: Politicians never like piloting schemes because they need quick results. I could argue as an academic that to do a randomized controlled trial would be ideal, and I’d give the politician the answer in three or four years. By that time, he probably wouldn’t be minister of health, and the government might even have changed. So governments are generally reluctant to pilot things if they can avoid it.

Galvin: As one of the advisers or architects of this Quality and Outcomes Framework, and given the benefits of hindsight, what would you have done differently?

Roland: One of the difficulties of any such scheme—and it’s hard to get around—is that some conditions are quite easy to get indicators for, and others are much more difficult. Coronary heart disease and diabetes are some of the easy ones. Mental health care would be a good example of the opposite. What do patients want of mental health care? Well, right near the top of that is a doctor who’ll spend time with them and listen to them when they’re distressed. That is probably, from the work we’ve done, what patients want more than anything else. And that’s really hard to assess, or to reward, or even to know whether it would be appropriate to reward in a pay-for-performance scheme. So it’s not an entire answer to, "What would you do differently?" but it is a regret that it is the nature of medical care that you incentivize some conditions much more easily than others.

Advice To The United States

Top The Quality And Outcomes... The Data Challenge Unintended Consequences Going Forward Lessons Learned Advice To The United... NOTE

 
Galvin: Let’s see if I can solicit your advice to the United States as we move down the road of pay-for-performance. One of our unique challenges is that we don’t have a mandate to increase overall spending. In fact, our view on P4P is that we’re in a zero-sum game, meaning that there’s no new money. How would this circumstance have affected the U.K. strategy?

Roland: We would have certainly gone for a less ambitious scheme, because it would have been politically difficult to bring in that big scheme where individual physicians stood to lose a lot. So we would have brought it in more slowly, so that the risk to physicians and the payer was reduced, and we would have concentrated on the indicators for which there was the clearest evidence of health gain, because those would be the easiest to justify. And I think that we would have certainly wanted a good idea of the baseline before we started, because otherwise, one doesn’t know that one is going to get a zero sum. We didn’t.

Galvin: One of the ideas to achieve budget-neutrality is that doctors with poor performance get paid less. What do you think of that idea?

Roland: Well, that’s inevitable in zero sum, isn’t it? Some get more, some get less, and the budget is overall neutral.

Galvin: It could play out that doctors get less of an increase versus an actual decrease. But either way, how do you think that this might affect doctors’ sense of professionalism?

Roland: Well, again, it’s much easier to give somebody a smaller increase than it is to reduce his or her pay. I think that most people in most jobs would find it pretty demotivating to get less pay. You can only introduce a zero-sum pay-for-performance scheme fast if you are prepared to have a substantial number of doctors losing significant amounts of money.

Galvin: One issue we’re debating in the States concerns how much reward to give to attainment versus improvement. Any thoughts on that?

Roland: The issue of whether you reward improvement or achievement depends on the situation you’re in. The principal argument for rewarding improvement is that you’ve got physicians working in a situation that is particularly difficult, where it would be hard to attain the levels that you’re aiming for, such as in an area of high socioeconomic deprivation or one with a high proportion of ethnic minorities. However, these might also be the populations that have worse health. So there would be an argument for keeping the high levels that you’re aspiring to, but also rewarding the people for improving along the way.

Galvin: On the other hand, we don’t want to enable the "poverty of ambition" that so often characterizes clinical goals. It always amazes me that in medicine, unlike other sectors, 91 percent compliance, which is around three sigma, is considered a very good achievement. Rewarding physicians for simply measuring and recording blood pressure at a three-sigma level seems to me to be not asking enough from the profession.

Roland: I don’t think that providing health care is quite the same as manufacturing computers. So taking blood pressure, for example, depends on whether the patient comes in. One of the reasons for being able to exception-report patients with hypertension in the U.K.’s pay-for-performance scheme is that you’ve sent the patient three written appointment notices and they haven’t turned up for any of them.

Galvin: But even with exception reporting, looking at the individual numbers and scores, they weren’t at 100 percent.

Roland: There is certainly an argument that if you allow such reporting, the top level should be 100 percent.

Galvin: Let me put you on the spot and ask your opinion on the relative weighting of the impact of the following four factors on the performance scores seen in the U.K.: The first would be that the baseline was higher than the NHS thought, the second would be better reporting due to IT systems, the third would be exception reporting, and the fourth would be substantive quality improvement, due in large part to structural changes, such as more nurse managers and clinics specializing in specific conditions. Can you give me your sense of their relative impact on first-year results?

Roland: I think that all of those things are important, and I couldn’t quantify which was greater than any other.¹

Galvin: Despite the differences in the structures of our respective health care systems, there’s a lot to learn from each other. Thank you for sharing your thoughts on this complicated but critical issue.

Editor's Notes

 
Martin Roland is director of the National Primary Care and Research and Development Centre, at the University of Manchester in England. Bob Galvin (robert.galvin{at}corporate.ge.com) is director, Global Health Care, at the General Electric Company in Fairfield, Connecticut.

NOTE

Top The Quality And Outcomes... The Data Challenge Unintended Consequences Going Forward Lessons Learned Advice To The United... NOTE

 

1. See T. Doran et al., "Pay-for-Performance Programs in Family Practices in the United Kingdom," New England Journal of Medicine 355, no. 4 (2006): 375–384.[Abstract/Free Full Text]

                      What's this?

This article has been cited by other articles:

				K. C. Stange and R. L. Ferrer The Paradox of Primary Care Ann. Fam. Med, July 1, 2009; 7(4): 293 - 299. [Full Text] [PDF]

				T. Marmor, J. Oberlander, and J. White The Obama Administration's Options for Health Care Cost Control: Hope Versus Reality Ann Intern Med, April 7, 2009; 150(7): 485 - 489. [Abstract] [Full Text] [PDF]

				K. C. Stange The Problem of Fragmentation and the Need for Integrative Solutions Ann. Fam. Med, March 1, 2009; 7(2): 100 - 103. [Full Text] [PDF]

				P. W. Lee, A. J. Dietrich, T. E. Oxman, J. W. Williams Jr, and S. L. Barry Sustainable Impact of a Primary Care Depression Intervention J Am Board Fam Med, September 1, 2007; 20(5): 427 - 433. [Abstract] [Full Text] [PDF]

				S. Campbell, D. Reeves, E. Kontopantelis, E. Middleton, B. Sibbald, and M. Roland Quality of Primary Care in England with the Introduction of Pay for Performance N. Engl. J. Med., July 12, 2007; 357(2): 181 - 190. [Full Text] [PDF]

				J. B. Christianson Evaluating Pay-For-Performance In Medicaid Through Real-World Observation Health Aff., July 1, 2007; 26(4): w528 - w531. [Abstract] [Full Text] [PDF]

Comments: