Milbank Memorial Fund


Better Information, Better Outcomes
The Use of Health Technology Assessment and
Clinical Effectiveness Data in Health Care Purchasing
Decisions in the United Kingdom and the United States

July 2000


(To request a bound copy of this report, click here.
To see a complete list of Milbank reports, click here.
When ordering, be sure to specify which report you want,
your name, mailing address, and phone number.)



Table of Contents

UK/US Purchasers' Group
Foreword
Executive Summary
Introduction
Background and Context
U.S. Health Care Purchasing
U.K. Health Care Purchasing
Common Themes, Observations, and Insights from Purchaser Interviews
Conclusions and Recommendations





UK/US Purchasers' Group

Alan Burns
Chief Executive
Cambridge (formerly North West Anglia) Health Authority

Pamela Charlwood
Chief Executive
Avon Health Authority

Helen Darling
Formerly: Manager, International Compensation and Benefits
Xerox Corporation
Currently: Practice Leader, Group Benefits and Health Care
Watson Wyatt Worldwide

Daniel M. Fox
President
Milbank Memorial Fund

Lee Greenfield
Lead DFL Member, Health and Human Services Finance Committee
Minnesota House of Representatives

Lynda Hamlyn
Chief Executive
South and West Hertfordshire (formerly Northamptonshire) Health Authority

Chris Howgrave-Graham
Chief Executive
Coventry Health Authority

John H. James
Chief Executive
Kensington & Chelsea and Westminster Health Authority

Ron Kerr
Director of Operations
NHS Executive, Department of Health

Velvet G. Miller
Formerly: Deputy Commissioner
New Jersey Department of Human Services
Currently: President and CEO
My Parent's Concierge

Sandy Praeger
Chair, Public Health and Welfare Committee
Kansas Senate

Margaret T. Stanley
Formerly: Assistant Executive Officer, Health Benefit Services
California Public Employees Retirement System
Currently: Vice President for Public Policy
Regence BlueShield

Foreword

This report is based on interviews with health care purchasers in the United Kingdom and the United States about how they value and use research assessing health technology and clinical effectiveness. The principal finding of the report is that purchasers value this research "but few use it when making health care purchasing decisions." Moreover, those "who do use this information tend to do so sporadically, rather than applying it in a proactive, systematic manner."

The Milbank Memorial Fund commissioned this report on behalf of an informal group of persons who make policy for purchasing health care in the United Kingdom and the United States. The members of this group are identified on the facing page.

The Fund is an endowed foundation based in New York City. It collaborates with policymakers in the public and private sectors to analyze, develop, implement, and communicate about health policy.

The group of purchasers met in London in 1997 and in New York the following year. The purpose of these meetings, as a British purchaser wrote, was to

consider the developments that have taken place…in relation to purchasing as the process has become more sophisticated and specifically to address the issue of the information that purchasers need to purchase effective and cost-effective health care…Discussing the developments and limitations of the situation in [each] country in the presence of individuals from a different system, but with many areas of shared concern, is…likely to be mutually beneficial.

The organizers of these meetings hypothesized that people who did similar work each day would find common interests despite the enormous differences in the organization and financing of health services in the United Kingdom and the United States. Moreover, focusing on common interests might reduce the tedious hours of elementary descriptions of national health care policies and systems that absorb considerable time at many international meetings.

The hypothesis proved correct. The UK purchasers—each of whom worked within the National Health Service—and the US purchasers—who served in the executive and legislative branches of government, private industry and a public corporation—immediately found common ground. They described their systems and policies in the context of discussing problems in purchasing health services for populations.

The members of the group have completed three projects. A report published in June 2000 assesses the implications for policy of research on genetics. A second project convened experts in assessing health care technology from the two countries to explore practical possibilities for collaboration in the dissemination of information. This report describes the current use of the research findings that are disseminated.

Many people deserve credit for this report. Pam Charlwood, John James, and Margaret Stanley formulated the questions about the use of research by purchasers to which policymakers wanted answers. John James and Barbara Stocking participated in describing the overall purpose of the group and recruiting its members from the United Kingdom. The members of the group are identified by title above. Barbara Stocking, Regional Director, National Health Service Executive, Southeast Regional Office, could not participate in the ongoing work of the group.

Charlwood, James, and Stanley guided the persons in both countries who conducted the interviews and wrote the report. Angela Coulter, then Executive Director of Policy and Development at the King's Fund, now Chief Executive of the Picker Institute Europe, and Janie Dallender, then Primary Care Health Researcher in the Directorate of Research and Development of the Kensington & Chelsea and Westminster Health Authority, now Senior Researcher in the Mental Health Research Section of the Sainsbury Centre for Mental Health, participated in planning the project. Dallender conducted interviews in the United Kingdom. Jessie Gruman and Cynthia Gibson of the Center for the Advancement of Health helped plan the project. Gibson conducted interviews in the United States. Harry Nelson, a staff writer for the Milbank Memorial Fund, conducted interviews and wrote the earliest draft of the report. Gibson was the principal writer for subsequent drafts, which had significant contributions from Coulter, Dallender, and Gruman.

We owe particular thanks to the 55 purchasers in the two countries who supplied the information on which this report is based in telephone interviews. The interviewees were offered anonymity in order to facilitate their participation in the project.

Daniel M. Fox
President

Samuel L. Milbank
Chairman

Executive Summary

In 1998, 55 health care purchasers in the United States and the United Kingdom were interviewed to discover the extent to which purchasers—who increasingly make the decisions about which health technologies become part of routine health care—have access to and make use of health technology assessment (HTA) and clinical effectiveness data. The major finding of this research was that purchasers in both countries value this information but few use it when making health care purchasing decisions. Those who do use it tend to do so sporadically, rather than applying it in a proactive, systematic manner.

Purchasers cited four general factors that have contributed to their relatively limited use of HTA and HTA information: (1) their overriding concern with the cost rather than the quality of services; (2) difficulties in accessing clinical and cost-effectiveness data; (3) insufficient training in using, interpreting, and critically appraising HTA information; and (4) a lack of skills and/or training in translating research evidence into practice. U.S. purchasers generally agreed that they primarily use this information when there are claims for new interventions, particularly those that are costly, or when there is controversy about a particular issue.

U.S. and U.K. purchasers agreed that it is difficult to obtain access to HTA/clinical effectiveness data. Many U.S. purchasers, especially non-clinicians, said that it is a "major struggle" to find this information—and that, even when it is available, it is "nearly impossible" to understand. U.K. purchasers were more likely to complain about lacking the time or the skill necessary to access the evidence. They were also more likely to rely on their public health colleagues to retrieve, analyze, and summarize the information.

Purchasers from both countries mentioned the "questionable" quality of much of the available data. Moreover, few studies tell providers how study results should or might be incorporated into practice, nor do they typically offer cost-benefit analyses. Purchasers also expressed concern that current data and research are often irrelevant to their populations, especially those with low incomes or special needs.

The purchasers used a similar set of criteria to assess the validity of such information. To be taken seriously, information should be produced by a "credible" and/or "reputable" organization, agency, or institute; based on studies that use an experimental or quasi-experimental research design and that are in the "public domain" (i.e., not conducted for marketing purposes or on behalf of a particular company or manufacturer); relevant to the issues in which purchasers are interested; peer-approved or regarded by clinicians as state-of-the-art; and published (or publishable) in top medical journals.

Purchasers identified some information sources as more helpful than others, and they relied more heavily on these. In the United States, several purchasers have developed formal or informal clinician networks to help them analyze HTA data and make decisions about health coverage. In the United Kingdom, the National Health Service Research and Development Programme has provided funding support to the Cochrane Collaboration, which carries out systematic reviews of randomized controlled trials and other studies of the efficacy of different treatments. Research and information produced by the Agency for Health Care Policy and Research (AHCPR) (U.S.) and NHS Research and Development Programme (U.K.) were the more highly respected and uniformly consulted resources. Only a few U.S. purchasers mentioned regular use of clinical effectiveness data published by the larger managed care plans or ECRI.

A review of the data culled by this study suggests several factors that are critical to ensuring that purchasers have access to, use, and value HTA/clinical effectiveness data. These factors include:

Introduction

Decisions about which health technologies become part of routine health care are frequently assumed to be the domain of individual health care providers. Increasingly, however, that responsibility belongs to those who pay for health care services: the purchasers. In the United States, purchasers are primarily private employers who purchase health care for their employees and dependents and federal, state, and local governments that purchase health care for employees as well as for beneficiaries of public programs such as Medicare and Medicaid. In the United Kingdom, where health care is more centralized, purchasers are part of the National Health Service, which covers the entire population of England, Scotland, Wales, and Northern Ireland. The term purchaser came into use in the United Kingdom along with changes introduced in 1991 by the Conservative government then in power. The current Labour government prefers the term commissioner, but, for simplicity's sake, this report uses purchaser.

Given the influential role purchasers currently play in determining which health care services will be covered, to what extent, and for whom, it is important to examine how purchasing decisions are made. Of particular interest is whether purchasers have access to and use empirical evaluations of the clinical value and cost-effectiveness of a wide variety of new technologies, procedures, and interventions. Available for decades, health technology assessment (HTA) and clinical effectiveness data have the potential to help purchasers better select, monitor, and assess the quality and value of the health care services they purchase for millions of beneficiaries. But do purchasers have access to this information, and, more important, do they use it in their decision-making processes?

To answer these questions, researchers commissioned by the UK/US Purchasers' Group in 1999 interviewed health care purchasers from public, private, and nonprofit organizations in both countries. Interviews with 55 purchasers conducted over a period of three months yielded the finding that, although purchasers value this information, few use it when making health care purchasing decisions. Those who do use HTA and clinical effectiveness data tend to do so sporadically—for example, when evaluating coverage for high-cost treatments, procedures that have not previously been covered, and/or services that generate controversy in the mainstream media—rather than applying it in a proactive, systematic manner.

Purchasers cited four general factors that have contributed to their relatively limited use of HTA and HTA information: (1) their overriding concern with the cost rather than the quality of services; (2) difficulties in accessing clinical and cost-effectiveness data; (3) insufficient training in using, interpreting, and critically appraising HTA information; and (4) a lack of skills and/or training in translating research evidence into practice.

These and other results detailed in this report raise serious questions about purchasers' ability and incentive to base purchasing decisions on the most objective scientific and clinical research available. What follows is a more detailed description of findings gleaned from these interviews, as well as a set of recommendations for addressing the knowledge and usage gap that currently exists among health care purchasers in both countries. With greater attention to the issues highlighted by this research—including what purchasers say they want and need in order to increase their use of HTA and other clinical effectiveness information—purchasers, health care professionals, and health care industry leaders can engage in more thoughtful and informed discussions, focusing not only on the cost but also the quality and breadth of the services that health care systems provide and cover.


Interview Methodology

A total of 55 purchasers were interviewed: in the United Kingdom, 14 nonmedical Health Authority purchasers, 7 public health physicians, and 9 general practitioners, and, in the United States, 13 public officials from 11 states, 4 private-sector purchasers, 5 representatives of private purchasing coalitions, and 3 consultants.

Interviews took place over a three-month period in early 1999 and were conducted by telephone. The semi-structured format that was used allowed for modification depending on the interviewee's role, the size of the company or organization, and the population for which health care benefits are purchased.

Interviews began with an explanation of the purpose of the project—that is, to determine how HTA and clinical effectiveness information figures in interviewees' decisions about health care contracting and purchasing. Interviewers also informed respondents that "HTA and clinical effectiveness information" was being defined as "information derived from the scientific study of the links between specific medical practices to health outcomes and that is often summarized in comprehensive technical reports and in practice guidelines." Respondents were then asked a series of questions that explored the following themes:

  • The value purchasers place on HTA and clinical effectiveness information
  • Purchasers' use of HTA and clinical effectiveness information in making contracting decisions
  • Purchasers' access to this information
  • Purchasers' assessments of the strengths and weaknesses of available information
  • The criteria purchasers use to feel confident in the credibility and authority of the information they use
  • The extent to which respondents' beneficiaries care about this information (U.S. only)
  • The sources of information that purchasers currently find to be most credible, as well as which sources they find least useful
  • Purchasers' preferences regarding how such information is presented, as well as the sources they prefer to use

Background and Context

U.S. Health Care Purchasing

Health care spending now accounts for almost 14 percent of spending on all goods and services in the United States and is expected to continue growing. The private sector accounts for approximately 42 percent of total health care spending, the public sector about 58 percent. Funding for public programs—Medicare, for seniors, and Medicaid, for low-income individuals—is extracted from transfers in the form of workers' payroll taxes and other tax revenues.

Employment-based health insurance is the most common form of health insurance coverage in the United States and has been for nearly six decades. Currently, nearly 155 million workers and their families—65 percent of nonelderly and 35 percent of elderly Americans—rely on their employers to provide or help pay for private health insurance.

This system of employer-sponsored coverage emerged to restrain wage inflation during World War II and afterward continued when the federal courts ruled that unions could collectively bargain with employers for benefits, including health care coverage. These benefits are considered public-sector "tax expenditures" because they are excluded from workers' wages for purposes of taxation and defined as an untaxed cost of business for employers.

For the next 30 years, a relatively healthy economy allowed this system to flourish among a wide range of employers—large, small, private, and public. In 1974, Congress passed the Employment Retirement Income Security Act (ERISA), which stipulated that employers (including government employers) that self-insure (i.e., pay health costs as incurred rather than purchasing insurance policies for employees) are not subject to states' insurance regulation laws, including mandates for coverage of particular services and practitioners. Until the late 1990s, the federal government chose to impose comparatively few requirements on self-insured health plans. As a result, approximately half of American workers and their dependents have employer-purchased health coverage that is not subject to regulation. This has had a dramatic impact on the way health care is purchased.

In the late 1970s and 1980s, an economic downturn and rapidly increasing health care costs forced employers to begin implementing cost-control strategies, including channeling employees away from fee-for-service indemnity insurance and into managed care systems that were designed to reduce costs and streamline delivery. Managed care's proliferation was not limited to the private sector, however, as most states also began relying on managed care as the health care delivery model for Medicaid recipients and as Medicare recipients began to enroll in managed care plans. Still, managed care enrollment has been more dramatic among the privately insured than among the publicly insured. In 1997, for example, only 21.2 percent of privately insured individuals were still enrolled in fee-for-service plans, compared with 80.5 percent of publicly insured individuals.

Managed care's increasingly dominant position in the health care industry has fueled the perception that health care is a commodity to be purchased on the open market. The individuals whose job it is to purchase health care services for companies, government agencies, and other organizations are now the people making most of the "consumer" decisions and have therefore gained considerable power in evaluating the products offered by different health plans and selecting which services will be available to a given set of beneficiaries.

Recently, however, some employers have begun to retreat from purchasing by providing a fixed annual health benefit that employees can use to enroll in managed care plans of their choice. Under this emerging system, employees can choose plans that cost more, less, or the same amount as the benefit provided by their employer.

Also, some purchasers are joining employer health coalitions to increase their clout through collective purchasing, which allows them to bypass health plan intermediaries and contract directly with health care provider systems. A few such coalitions are beginning to implement innovative performance-management practices aimed at creating systems that deliver a full continuum of care and that are clinically and fiscally accountable for outcomes and the health status of their enrolled populations. These coalitions, whose leverage is growing, are able to demand and obtain detailed HTA information and clinical effectiveness data and, most important, to base their purchasing decisions on this information combined with analyses of their populations' needs.

These efforts are still relatively rare, however, which means that many purchasers remain frustrated by the lack of available information on which practices offer the best value and the best quality. As one U.S. public purchaser noted, "The numbers are easy to work out as far as cost, but determining the quality of what you get is more difficult. Everyone wants to make sure that treatments are efficacious and the information is available on outcomes. Especially now, with new technologies that are so costly, this information can't be ignored."

U.K. Health Care Purchasing

The United Kingdom's health care system, the National Health Service (NHS), is a universal service, financed through public taxes, that is available to all and mostly free at the point of service. Although generally regarded as successful, the NHS has in recent years been criticized as underfunded and inefficient.

Because it is tax-funded, the U.K. system is necessarily subject to political direction. The interviews with U.K. purchasers in this study reflect their experience under the Conservative government (until 1997) primarily, and under the subsequent Labour government to a much smaller extent. An emphasis on evidence-based decision-making was constant throughout this entire period, although under the current administration the focus has shifted to a more centralized and national approach. This has displaced what was seen as an inequitable variation in behavior among decision makers (purchasers) from one locality to another.

A series of NHS reforms in 1991 introduced a purchaser-provider split to improve the system's cost-effectiveness. In this limited form of regulated competition, referred to as the "internal market," hospitals, which had previously been under the direct control of local Health Authorities, became self-governing NHS Trusts. Although the Trusts continued to be under the aegis of the NHS, they gained much greater control over their own affairs and, at least in theory, were able to sell their services to any purchaser. The Health Authorities remained responsible for purchasing services for their local populations.

The most radical change instituted by the 1991 NHS reforms, however, was the introduction of general practitioner (GP) "fundholding." As independent contractors responsible for providing primary care to a list of NHS-registered patients, GPs could, if they chose, be given a budget that would allow them to purchase selected services (mainly outpatient consultations, elective surgery, and community services) for their practice populations. A majority of GPs elected to become fundholders.

In 1997, a system that clustered GPs into Primary Care Groups (PCGs) replaced fundholding. PCGs, which range in size from about 10 to about 30 practices and each of which serves, on average, about 100,000 people, are responsible for coordinating primary care services for patients and purchasing secondary care. Currently, PCGs can apply to become Primary Care Trusts (PCTs), the first of which were officially established in 2000. Operating relatively independently from the Health Authority but within a local framework called the Health Improvement Plan, PCTs will have control over a fully integrated, capitated budget covering patients' general practice, community, and hospital services. At the time of writing, it is not clear what proportion of PCGs will want, or will be allowed, to have this responsibility.

The Labour government's reforms also included the creation of the National Institute of Clinical Excellence (NICE), designed to inform the development of clinical standards across the NHS. Because the institute was just being created, none of the interviewees in this study reflected on the role that NICE might play, but it is likely to have a significant influence over purchaser behavior in the United Kingdom in the years ahead.

Despite these changes, the NHS continues to operate as a tax-financed system under which Health Authority and PCG budgets are determined by the central government. Money is supposed to follow patients, but because most purchasing is based on block contracts or service agreements, patients tend to follow the money. While the administration of service agreements is carried out by Health Authority staff (who are advised by PCGs, hospital specialists, and Directors of Public Health), performance monitoring across the entire NHS is the responsibility of the NHS executive through its Regional Offices.

The United Kingdom also has a small private-sector health system. About 13 percent of the population has private health insurance coverage, and private hospitals continue to provide elective surgery. There are also some private psychiatric hospitals and a relatively large number of private nursing homes. In certain limited cases, NHS purchasers can and do purchase services from private hospitals, and many NHS hospitals have a small number of private beds for use by patients with private health insurance.

Common Themes, Observations, and Insights from Purchaser Interviews

Despite differences between U.K. and U.S. purchasers—primarily because U.S. respondents were more likely to serve as program or benefits managers rather than as clinicians or health administrators—there were several common themes that emerged from the interviews. Among these were the following:

Purchasers view health technology assessment and clinical effectiveness information as valuable and important.

The majority of purchasers agreed that this information is "terribly important" and "valuable" (although many, for a variety of reasons, said they did not use it). "It's the only way to make sound purchasing decisions," said one U.S. public purchaser, "because a dirty little secret of medicine is the huge variation in treatment approaches." Many respondents commented that having access to this information is critical to ensuring better quality and will play a more important role in future decision-making efforts because of the proliferation of new technologies and pharmaceuticals. Others were more dismissive, such as one British GP, who said, "Evidence-based nonsense! Evidence-based information is simply a collection of old-fashioned trials and cheap drugs."

Several purchasers were confused as to the meaning of health technology assessment/clinical effectiveness data.

Several purchasers were unable to distinguish between HTA/clinical effectiveness information and performance indicators such as the Health Plan Employer Data and Information Set (HEDIS) produced by the National Committee for Quality Assurance (NCQA) or NHS activity indicators such as waiting lists or response times. One purchasing consultant, who has written extensively about these issues, said that he was not surprised that purchasers confused the two, since "all they care about is whether health plans are accredited or adhere to existing HEDIS standards." Nearly every U.S. purchaser interviewed, in fact, mentioned that he or she reviews HEDIS data, report cards, and patient-satisfaction surveys to "figure out what plans are actually doing." Only a handful, however, conceded that this information does not necessarily indicate whether patients are getting state-of-the-art care. As one private purchaser noted, "It's just a measure of whether [plans] are doing what they're supposed to do, rather than of outcomes or appropriate care." A Medicaid purchaser admitted that, although he relied on HEDIS data and patient-satisfaction surveys for information, "neither is very successful in understanding outcomes or moving us toward better outcomes, more research, or practice guidelines."

Although purchasers value this information, their use of it is reactive, sporadic, and limited to certain circumstances.

Purchasers from both countries admitted that their use of HTA/clinical effectiveness information is sporadic, and limited to specific circumstances or cases. Few purchasers have implemented proactive, systematic procedures to evaluate these resources carefully, and those that do tend to enlist the assistance of medical directors, public health consultants (U.K.), external clinician networks, and, in a few cases, the health plans or NHS providers themselves.

U.S. purchasers generally agreed that this information is primarily used when claims for new interventions "come across the desks of reviewers." According to one U.S. private purchasing consultant, "The processing people don't know what to do with it, so they pass it on to the policy people and those folks check the literature." A U.K. commissioning manager agreed that use of this information tends to be reactive: "We consult it usually when there's a problem—either costs or demand on the system. Nothing ever happens unless there's a crisis. It's not the case that it's done systematically." A U.K. colleague added that he only uses this information when he's "under fire" and that this is "not very proactive." "I only use it when I'm being asked a question on changing our purchasing contracts or purchasing style. We don't sit down and review all our contracts, but only respond to something [specific]."

Financial concerns were an especially powerful incentive to seek out HTA information, especially when purchasers were confronted with having to make decisions about whether to pay for costly new technologies, equipment, and/or pharmaceuticals. "So much of the new technology costs are so high and add to the cost of health care, we can't ignore this information," said a U.S. purchasing consultant. HTA information is also useful in determining which treatments or technologies can be eliminated from coverage because they are not particularly effective, and thus, "companies don't want to pay for it." A U.K. commissioner was more blunt: "I don't use this information as often as I'd like, only when there is a difficult financial situation. Financial problems come first."

Controversy about a particular issue is another incentive for purchasers to consult clinical effectiveness data. One purchaser for a large American public-sector agency, for example, pointed to the use of autologous bone-marrow transplants for the treatment of breast cancer—an issue that crept into coverage decisions and, later, court cases. "When the courts started getting involved, that's when purchasers got involved and started caring about what the clinical data was saying about the best practice," she noted. Another respondent, though, said that even in these cases, "it's a blanket kind of concern—that is, should we cover it or not?" A U.K. general practitioner agreed that politics plays a role in purchasing decisions and that information about clinical worth and cost effectiveness can be helpful: "The facts can be a way in which to cut through the vested interests."

Purchasers cite several barriers to using this information.

Both groups of purchasers cited similar barriers to their use of this information. Among the impediments were:

Lack of access

Many purchasers, especially non-clinicians, asserted that they had relatively little access to HTA information. This problem was more acute for U.S. purchasers. Several U.S. respondents who indicated that they were interested in reviewing this information more systematically complained that they struggled to obtain access to it. "You really have to look for it," said one consultant, "and most purchasers simply don't have the time to do this kind of research." One U.S. purchaser, whose mid-sized company recently struggled over decisions about the kinds of procedures to cover for "three traumatic cancer cases," expressed deep regret that he "had nothing to compare the care these people got against anything that shows that they got good, quality services as far as procedures, recovery time, bone marrow transplants, etc. Employers just don't have this information when they go out and look at plans."

U.K. purchasers were more likely to complain about lacking the time or the skill necessary to access the evidence. "Browsing on Medline is terribly time-consuming and laborious," said a former GP fundholder. "We are not trained. It should be a full-time job for someone. We don't have the time, with 60 patients to see."

Some purchasers pointed to an unwillingness among institutions engaged in this research to provide information to purchasers. One purchaser for a small, private American company said that he had requested clinical data from some of the "major institutions that have it" but had always gotten "a guarded response from them about the studies they're doing." This is especially true, he added, if the question is asked by a purchaser with less name-recognition.

A state Medicaid director who created clinical working groups to analyze the data and literature on a number of conditions was stymied when she began collecting HTA information. "I had to call and call to get to the right people who could tell me what kind of things I should be looking at. . . . It [took] three years of diligently hunting down these people—and there are only about 50 of them in the United States." She added that she is now on this network's mailing list and so has access to this kind of information. "You not only have to know about this information and that it exists," she said, "you have to be persistent about getting hooked into it!" In short, said one respondent, "It's like the myth of Sisyphus trying to get information like this."

Complex and technical format

Numerous purchasers complained that the information that is available is complex or "riddled with statistics and technical language that makes it impossible to comprehend." Moreover, studies tend to be academically oriented and rarely offer interpretations of findings, which makes it difficult for a purchaser to discern what, exactly, a study has concluded or if its results are even relevant to his or her beneficiary population.

U.K. purchasers were more likely to rely on their public health colleagues to retrieve, analyze, and summarize the information. As one U.K. commissioner said, "I have complete reliance on the Director of Public Health and colleagues . . . and hold them in high esteem. Not being a clinician, it's not for me to know."

Questionable data quality

A number of purchasers, especially those with clinical backgrounds, complained about the quality of the data that do exist. "There is no shortage of data," said one U.S. purchaser, but "good data is hard to find." Although evidence-based medicine is assumed to be the ideal of the medical field, it is "often in short supply," say purchasers from both countries. "We probably care about [evidence-based medicine] more than any group in the country, and we're continually shocked by how little data there is about technologies and how they affect outcomes," said a member of a large statewide purchasing coalition. Exacerbating this problem, said one respondent, is a growing number of journals that are relatively uncritical about what they publish. "A lot of these studies draw too-hasty conclusions."

A lack of standardization among studies contributes to purchasers' inability to assess the bottom line regarding which technologies and/or interventions are most clinically valuable and/or most cost-effective. "The data's useless in many cases because there's no meaningful correlates and it's not contextual," said one U.S. physician. A colleague agreed: "The volume of information out there is overwhelming, with many of them reaching different conclusions about the same thing." In short, "there are just too many different standards, measures, and methodologies."

For a number of years, U.K. purchasers have had the benefit of summaries produced for their use by the NHS Centre for Reviews and Dissemination at the University of York. These Effective Health Care bulletins summarize the research evidence (systematic reviews) about the effectiveness of specific health technologies. Health Authority commissioners said that they found them useful but couldn't always persuade clinicians to implement the recommendations: "Effective Health Care bulletins are useful for an overview but can be contentious in certain areas and don't always enable you to take the discussion forward. A recent article on cardiac services was responded to by surgeons here saying, 'What a load of rubbish!' Clearly, there are some discrepancies between those at the coal face and those writing effectiveness bulletins."

Absence of real-world applications

Many purchasers said that the information that is available is given little attention because it often fails to tell providers how the data should be incorporated into practice. One state Medicaid managed care purchaser said, "You have to translate this stuff into the real world"—to "make it actionable," in the words of another respondent. Yet a third respondent—a state health commissioner—agreed: "There is simply not enough information about how physicians should be implementing or incorporating new procedures or data into practice. Doctors can recite all the right guidelines, but when you look at what they actually do in practice, they often don't follow them." As an example of this, one purchasing consultant pointed to "all the research about beta blockers which practitioners haven't integrated into treatment."

Moreover, few studies offer cost-benefit analyses of new technologies, which are important to purchasers, said one public purchaser. "Since cost is what most companies care about, better cost-benefit analyses should be available, i.e., what a company or individual could expect to receive as value for cost of the treatment/intervention." "The people reading this stuff," said another, "want to know that it will improve value and drive down costs." Purchasers also want these analyses to say more about which treatments or technologies are ineffective or a "waste of money."

Narrow focus

Both groups of purchasers expressed concern that current data and research are often irrelevant to their populations, especially low-income people or people with special health care needs. As a Medicaid purchaser said, "We need information about conditions and illnesses for our population and that will address the fact that our coverage is broader than what most plans provide. Most of what's out there takes a narrower view of coverage or is published under an assumption that plans or purchasers want to limit coverage." Another Medicaid purchaser agreed: "The daily proliferation of informational materials out there are of little use to the Medicaid program."

Several purchasers noted a dearth of information relating to the needs of specific patient groups. A U.K. commissioner, for example, said that she had difficulty finding appropriate clinical effectiveness information about mental health issues. Other services that purchasers said were overlooked in the literature are preventive or promotional health interventions, new pharmaceuticals (especially antidepressants), communication devices for the deaf and blind, social services, community services, and palliative care.

Timeliness

Several purchasers expressed frustration about the time lag between completed research and the publication/dissemination of results. "Purchasers consider this information very little," said a private U.S. consultant, "because when they need to consider it, it isn't there. The claims often come in before the data does." Another purchasing consultant concurred: "The incredibly slow diffusion of research findings is very frustrating." One Medicare purchaser said that information is "extremely dated by the time we get it—especially information about new drugs."

Purchasers often have little incentive to use this information more systematically.

U.S. purchasers expressed frustration over the lack of consumer demand for better quality, state-of-the-art services. Such demand, it was thought, would create an incentive for companies and agencies to require more systematic review and application of HTA and clinical effectiveness information. "Purchasing for employees and other individuals is very difficult because . . . consumers don't care about health plans' performance as long as they have a choice of providers and services, access to them, and they don't cost a lot," a public consultant said. He added that, because most U.S. purchasers are human resources people whose job is to "attract and retain employees who don't care about quality and are more concerned with getting the plan that gives them as many providers as possible, they have . . . little incentive other than to make their employees happy."

Purchasers also complained that they only tend to hear from beneficiaries when they have a problem. Nearly every U.S. purchaser said that beneficiaries care "little" or "not at all" about this information unless they are faced with a life-threatening disease and want or need "to know the most up-to-date treatments available and who's providing them." In such cases, "they usually turn to the Internet to get this information," many respondents noted.

If purchasing is going to focus on quality, numerous U.S. purchasers said, "consumers are going to have to scream for this information, and they're not screaming for it yet." Several respondents attributed this lack of demand to a dearth of reputable, credible, and comprehensive materials or tools that consumers could use to educate themselves about what constitutes quality health care services or providers. A few purchasers blamed the health care industry and its "resistance to being measured" for this situation. A private purchaser, who is also a member of a state coalition, said, "As a consumer, if I'm going to buy a car, I can [check] Consumer Reports, the Internet, etc., and find out every detail about it so I can purchase wisely. But if I need a certain procedure done on my heart or kidney or something, I can't find any information. It's all mostly anecdotal, asking people for a referral, or what they did, etc. Nowhere can I get a book that says, 'this hospital's the best at this and has this outcome.' That's a big problem. . . . I can find out more about a washing machine than health care."

A few purchasers, however, said that although consumers could demand this information, they choose simply to "trust their providers to tell them about medical care or treatment." As proof, respondents pointed to their own attempts to produce report cards and other materials about plans and health services, which were largely overlooked by consumers, who "didn't really use them or seem to care."

Purchasers in rural areas also have little incentive to use this information because it is secondary to "getting enough providers in the pool." According to one Medicaid purchaser for a small group, "We have a shortage of providers, . . . so requiring that [they] be on the cutting edge would probably mean we'd have even less access. . . . We don't have the luxury of making sure they're providing treatment according to the latest clinical trials."

Information that purchasers do use must meet certain criteria.

Purchasers agreed on a set of criteria they use to assess the validity of the information they do obtain and review. Generally, to be taken seriously by purchasers, information must be

Purchasers identified some information sources as more helpful than others, and they relied more heavily on these.

Clinician networks

Purchasers in both countries tended to rely heavily on clinicians, including medical directors or clinicians affiliated with area health plans (U.S.) and public health consultants (U.K.), to help them sort through HTA/clinical effectiveness information. In the United States, several purchasers have developed formal or informal clinician networks to help them analyze HTA data and make decisions about health coverage. One large public purchaser, for example, is convening a group of cardiac specialists to help develop a database of the best available information to help doctors improve their practices—a process that is "unique because it's a provider- rather than plan-based push toward evidence-based medicine." A participating clinician noted, "It doesn't make sense to keep arguing about what we should cover when we don't know what the state of the art is. Providers know best what that is, so [we're designing] processes for facilitating their coming together and developing new ways to compile this information and get it out there. If we can get this community to agree on the single best approaches to these kinds of issues, we can dramatically improve quality."

In the United Kingdom, the NHS Research and Development Programme has provided funding support to the Cochrane Collaboration, an international network of people engaged in carrying out systematic reviews of randomized controlled trials and other studies of the efficacy of different treatments. Completed reviews are included in an electronic database that is accessible on CD-ROM or via the Internet. Surprisingly, few purchasers in either country reported having consulted this database when making their purchasing decisions.

Purchasing coalitions

In both countries, there have been attempts to package information for use by purchasers in new and better ways. In the United Kingdom, several NHS regions have established committees to produce reviews of the evidence and to make recommendations about clinical value and cost-effectiveness. In the United States, state-based purchasing coalitions have been proactively compiling and disseminating information to providers and purchasers, who in turn are beginning to view these organizations as the "best source of data [they] have about new technologies and treatments."

In Maine, for example, a coalition of business purchasers—"out of sheer frustration over the lack of information about quality health care"—compiled available data about best-practice treatments as well as statewide inpatient and outpatient outcomes and then sent this information to a statewide panel of physicians (including specialists) for review. The panel not only identified variability in certain areas (e.g., bypass surgery, C-section, and back surgery rates) but also helped to assess why these variances were occurring.

Some agencies, for example, had been using stress tests to diagnose heart disease, while others used imaging—and these different diagnostic procedures led to different rates of surgery referrals. This discovery led to the creation of a new state protocol for recommending surgery (similar to one released by the American Medical Association six months later) as well as new set of practice guidelines. "Now," said one purchaser who helped organize the coalition, "treatments and interventions for this condition [can be] driven by best practices that stem from the data rather than from what plans think is best or want to offer." The coalition also took steps to ensure that these new protocols and practices would be implemented across the state by inviting "all possible stakeholders to the table to discuss what the data showed and then get consensus about them." The state Medicaid office was so impressed by the effort that it joined as a member of what has become a successful private-public collaboration.

Research institutions

Purchasers from both countries said they turned to local universities for assistance in tracking down information about interventions, treatments, and procedures. Purchasers also cited research and information produced by the Agency for Health Care Policy and Research (AHCPR) (U.S.) and NHS Research and Development Programme (U.K.) as among the more highly respected and uniformly consulted resources. Many U.K. purchasers also relied on research generated by public health departments as well as information provided by voluntary bodies such as Bandolier, a monthly publication (distributed primarily to GPs) that summarizes HTA information in a lively and readable form.

Journals

U.S. purchasers put the most stock in the New England Journal of Medicine, the Journal of the American Medical Associaiton (JAMA), and, to a lesser extent, Health Affairs. Public purchasers were more likely to consult policy-related publications produced by nonprofit groups such as the Center for Health Care Strategies. Some purchasers said they liked to review industry trade publications that publish guidelines, such as HMO, Managed Care, and others. Only a few U.S. purchasers mentioned regular use of clinical effectiveness data published by the larger managed care plans (e.g., Kaiser Permanente, Blue Cross/Blue Shield) or ECRI.

U.K. purchasers said they rely on The Lancet and the British Medical Journal—"well-known, well-reviewed, blue chip" journals that are "objective and credible." Bandolier was mentioned by several U.K. purchasers as a resource that is especially useful because of its independence; its "reader-friendliness, especially for non-clinicians"; and its general "readability." Some respondents preferred Effective Health Care bulletins because they provide summaries and "short cuts" about information that has been "thoroughly assessed and appraised" and "that considers all the evidence."

Electronic/Internet resources

U.K. purchasers were more likely than their U.S. counterparts to use electronic resources, such as Medline, although U.S. purchasers were increasing their use of the Internet. Although British purchasers cited the Cochrane database as a useful tool for obtaining clinical effectiveness data, several admitted that their use of this database was indirect—that is, that they relied on public health doctors and consultants to analyze and interpret this data for them.

Purchasers from both countries agreed, almost unanimously, that the least helpful information—information they tend to ignore—was "anything produced by manufacturers [or] pharmaceutical companies" and industry-produced "marketing-type materials." Some U.K. purchasers mentioned their aversion to materials produced by special interest or pressure groups.

Purchasers, particularly in the United States, were mixed in their opinions about the usefulness of health-related information in the mainstream media. While some said they make it a point to monitor a range of major daily newspapers for health-related news, others said they were skeptical of the "propaganda" published in these outlets. Said one U.S. clinician, "Most of what appears in the mainstream press [such as the New York Times] is unhelpful because it raises people's expectations. . . . Plus, it's a lot of sound bites, which creates perception without depth."

Both groups of purchasers agreed on the ways in which they would prefer this information to be presented.

Purchasers from both countries agreed that they need more materials that

Purchasers also suggested other ways to disseminate this information:

Conclusions and Recommendations

It is clear from the interviews in this study that both U.S. and U.K. purchasers' use of health technology assessment information in their purchasing decisions is partial, selective, and sporadic. This appears to be largely due to an inability to access or understand this information, much of which is extremely complex and technical and/or available only in formats that are not easily digestible. Purchasers also cite pressures to focus on health care coverage costs, rather than quality, as an additional and important disincentive to making more systematic use of this information.

There were slight differences, however, between the two groups of purchasers. Unlike U.K. purchasers, many of whom were familiar with HTA information, the majority of U.S. purchasers were confused about what it is or tended to view it as synonymous with performance indicators such as the Health Plan Employer Data and Information Set (HEDIS) published by the National Committee for Quality Assurance (NCQA). Moreover, when asked to identify the sources of information on which they tended to rely most for clinical effectiveness information about particular treatments or conditions, only a handful of U.S. purchasers cited two of the most respected sources of HTA information: Blue Cross/Blue Shield and Kaiser Permanente. In contrast, most U.K. purchasers said they were aware of the Cochrane database (which systematizes and reviews randomized controlled trials and other studies about the efficacy of various treatments), but because many were unable to interpret and analyze this information, they used consultants and public health doctors to do this for them.

Exacerbating this information "disconnect" was the inability of many purchasers in both countries to adequately translate study findings in ways that could aid in their decision-making processes—an inability due primarily to a lack of training in interpreting research data. Even those purchasers who were able to translate the data, however, tended to have reservations about its usefulness, expressing skepticism about evidence-based medicine. Specifically, they said, data that are available are often contradictory and/or simply not generalizable because there are no universally agreed-upon standards, measures, and/or methodologies by which to assess the results. Moreover, much of what is published in peer-reviewed and respected journals is outdated once physicians or others interested in this information receive it.

As health care interventions and technology become increasingly complex, it is essential that purchasers (public, private, medical, and corporate), purchasing consultants, and others have access to standardized, synthesized, and "bottom line" data that will give them a clear and concise indication of which treatments are the most efficacious and cost effective. This will require developing a comprehensive, centralized, and standardized information system that provides timely information that purchasers from both countries can easily consult when making their health care purchasing decisions. The first step in this process is to identify and agree on a chief agent or organization that would be able to streamline and coordinate the many—and often overlapping—streams of information that now exist.

In the United Kingdom, many health authorities sought to provide standardized guidance to staff involved in commissioning. An example is the Avon Health Authority's "Code of Practice for the Public Health Directorate" (see box). At a national level, the National Institute of Clinical Excellence (NICE) will produce authoritative guidance on a limited number of new technologies. There will remain a need at a local level for a capacity to assess new technologies, although these are designed to be relevant to smaller populations and to cover a broader range of technologies.

Similarly, an American member of the UK/US Purchasers' Group recommends that "we should allocate a large amount to [create] a centralized clearinghouse for high-quality technology assessment and related research and reports on medical effectiveness. It could be hyperlinked to NIH, [the] National Library of Medicine, and other public and private organizations that contribute to technology assessment."

Another important factor limiting use of HTA and clinical effectiveness data, and one that also must be addressed, is the culture in which purchasing decisions are made. Currently, purchasers are rewarded more for "keeping costs down" than for choosing high-quality care with proven results. This is especially true in the United States because of the prevailing belief that quality care, including high-tech interventions, is guaranteed to those with adequate coverage. Traditionally, health care coverage in the United States has placed few constraints on treatment; as a result, there is little incentive for purchasers to search for high-quality treatments or interventions that clinical data show to be most effective. But it is also true that such searches are difficult to conduct, given funding bodies' lack of incentive to produce solid information about cost-effectiveness or to disseminate it in a user-friendly form. Those responsible for commissioning research, therefore, must redouble their efforts to discover what information is most useful to purchasers—and the form in which is should be presented—and then to synthesize and disseminate it widely.

Purchasers must, however, be able to understand research findings. As things stand, many lack a research or medical background and thus have insufficient training in using, interpreting, and critically appraising HTA information, or they simply lack the skills needed to translate research evidence into practice. Moreover, such skills-training is rarely available to purchasers or encouraged by their employers. Providing purchasers with opportunities to engage in this kind of training—and with incentives to make quality rather than cost their top priority in health care coverage decision-making—would be strong steps toward facilitating a culture in which the use of clinical effectiveness and HTA information becomes the rule rather than the exception.

Finally, there must be greater recognition and discussion of the larger political and intellectual climate in which purchasing occurs. In the United States, for example, there has been a small but influential movement among purchasers to rely on small, relatively closed circles of professionals for determining which treatments should be covered, especially in specialty care. While this process may be relatively efficient, over the long term it may preclude open debate or consultation with other professionals who may have different knowledge or opinions. In the United Kingdom, the structure of the health care system continues to depend largely on politics; the purchaser's role, therefore, changes with the political climate. Clarifying the impact that these and other shifts will have on health care purchasing is extremely important, as is identifying ways to address them to ensure that the greatest number of beneficiaries have access to high-quality health care.


Code of Practice for the Public Health Directorate—Avon Health Authority

Questions about the effectiveness of health care interventions arise in numerous ways during the course of the Health Authority's work. As a Directorate, we are frequently involved in helping to answer these types of questions. This code of practice sets out the approach we aim to follow in undertaking this work.

  1. When presented with a question about effectiveness, we will first try to clarify and define the question through discussion and by assembling further information where appropriate.
  2. We will agree on an appropriate depth of investigation and time-scale for preparing evidence-based advice about effectiveness. Some issues will warrant only a brief search, and others need a rigorous approach, which can be time consuming.
  3. For important questions, amenable to evidence assessment, we will search, as appropriate, some or all of the following electronic literature sources:
    Basic search: Cochrane database, clinical evidence, the TRIP (Turning Research into Practice) database, specialized drug information sources, National Institute for Clinical Excellence reports, the new electronic Library for Health, systematic reviews and effectiveness bulletins from the York Centre for Reviews and Dissemination, Bandolier, and the Wessex Development and Evaluation Committee reports.
    In-depth search: Health Evidence Bulletins (Wales), the American College of Physicians' Best Evidence, the National Research Register, primary search of Medline database using Pub Med, and specialized data sources (for complementary therapies and others).
  4. We will produce a summary of the findings that includes explanation of: what was searched, an assessment of the quality and strength of the findings, what the evidence means regarding likely benefits to patients, and how the situation may change in the future (upcoming research reports, e.g.).
  5. We will keep a record of search results where relevant electronic and paper records are filed.
  6. We will maintain up-to-date guidance that includes web addresses for all the above and that will be available electronically via the professional website (ACHeW).



(To request a bound copy of this report, click here.
To see a complete list of Milbank reports, click here.
Be sure to specify which report you want,
your name, mailing address, and phone number.)

Milbank Memorial Fund
645 Madison Avenue
New York, NY 10022

(c) 2000 Milbank Memorial Fund. This file may be redistributed electronically as long as it remains wholly intact, including this notice and copyright. This file must not be redistributed in hard-copy form. The Fund will freely distribute this document in its original published form on request.

Printed in the United States of America.

ISBN 1-887748-37-7

Milbank Memorial Fund HomepageMilbank Memorial QuarterlyReportsBooksEditorial and Program StaffBoards