Ray Moynihan
Co-Published with AcademyHealth
October 2004
(To request a bound copy of this report, click here.
To see a complete list of Milbank reports, click here.
When ordering, be sure to specify which report you want,
your name, mailing address, and phone number.
To print this report click here for the pdf version.)
Foreword
Many policymakers and clinicians take professional pleasure in grounding their decisions in the best available evidence, especially when they have confidence in that evidence and access to it when they need it. Similarly, many researchers are eager for their work to be useful, especially when they are only required to communicate it in journal articles and on a schedule that they choose.
In this report, Ray Moynihan, an experienced journalist, describes collaboration between researchers and users of their skills and findings in nine countries. Moynihan introduces the report by briefly recapitulating controversial issues in the application of knowledge derived from research to policy and practice. He concludes by describing common themes in the nine stories.
Moynihan began his reporting of the nine stories at an international conference in the fall of 2003 that was sponsored by AcademyHealth and the Agency for Healthcare Research and Quality. The Milbank Memorial Fund commissioned him to roam the conference in search of stories about how research informs policy and practice. He completed his reporting after the conference.
Many people helped to make these stories by providing, interpreting, and contextualizing evidence and then making decisions based in part on it. We thank the persons named in each of the nine stories, each of whom had an opportunity to comment on what Moynihan wrote, as well as the reviewers who commented on the report as a whole.
Daniel M. Fox
President
Milbank Memorial FundDavid Helms
President and CEO
AcademyHealth
Acknowledgments
The following persons were interviewed for this report and/or reviewed it in draft. They are listed in the position they held at the time of their participation.
Jordi Alonso, Unit Institut Municipal d'Investipaun Media, University of Barcelona, Spain; Donald M. Berwick, Chief Executive Officer, Institute for Healthcare Improvement, Boston; Gabriel Carasquilla, Doctor, University of Valle, Colombia; Timothy S. Carey, Professor of Medicine, University of North Carolina, Chapel Hill; Mushtaque Chowdhury, Visiting Professor, Columbia University, New York; Carolyn Clancy, Director, Agency for Healthcare Research and Quality, Rockville, Md.; Karen Davis, President, The Commonwealth Fund, New York; Andrés de Francisco, Deputy Executive Director, Global Forum for Health Research, World Health Organization, Geneva; Richard Deyo, Professor, University of Washington, Seattle; Mary Ferguson-Paré, Chief Nursing Executive, University Health Network, Canada; Kathleen M. Foley, Chief, Pain Service, Department of Neurology, Memorial Sloan-Kettering Cancer Center, New York; Miguel Gonzalez-Block, Programme Manager, Alliance for Health Policy and Systems Research, World Health Organization, Geneva; David Helms, President and CEO, AcademyHealth, Washington, D.C.; Liliana Jadue Hund, Instituto de Epidemiología y Políticas Públicas, Facultad de Ciencias de la Salud, Universidad del Desarrollo, Chile; Egon Jonsson, Leader, Health Evidence Network, World Health Organization, Copenhagen; John Lavis, Associate Professor, McMaster University, Hamilton, Ontario, Canada; Steven Lewis, President, Access Consulting Ltd., Saskatoon, Canada; Peter Littlejohns, Clinical Director, National Institute for Clinical Excellence, London; Jonathan Lomas, Executive Director, Canadian Health Services Research Foundation, Ottawa; Princeton N. Lyman, Senior Fellow, Director, Africa Policy Studies, Council on Foreign Relations, Washington, D.C.; Rashad Massoud, Director, Quality and Performance Institute, University Research Company, Washington, D.C.; Andrew D. Oxman, Director, Department of Health Services Research, Norwegian Directorate for Health and Social Welfare, Oslo; Tikki Pang, Director, Research Policy and Cooperation, World Health Organization, Geneva; Jonathan Perlin, Deputy Under Secretary for Health, Department of Veterans Affairs, Washington, D.C.; Patricia Pittman, Director for International Initiatives, AcademyHealth, Washington, D.C.; Hernan Sandoval, Executive Secretary, National Committee for Health Reform, Chile; Kenneth I. Shine, Executive Vice Chancellor for Health Affairs, University of Texas; Olive Shisana, Executive Director, Human Sciences Research Council, South Africa; Ronald J. Stamp, Institute of Medicine, Health and Social Care, University of Portsmouth, United Kingdom; William Tierney, Chief of General Internal Medicine and Geriatrics, Indiana University; Suwit Wilbulproprasert, Deputy Permanent Secretary, Ministry of Public Health, Thailand.
The author would like to thank those who assisted with note-taking and other arrangements at the Washington, D.C., conference: Anne Gauthier, Ethan Joselow, Tamar Klaiman, Patricia Pittman, Deborah Queenan, Deborah Rogal, Kari Root, and Wendy Valentine.
Executive Summary
As the move toward an evidence-based approach to health care has gathered momentum over the past two decades, increased attention has been paid to strategies that try to ensure that knowledge gained from the best evidence is actually used in practice. This move to enhance the utility of research involves both making research evidence more usable and improving the capacity of management, policy, and clinical decision makers to use it.
Using evidence to improve health services has been the focus of a series of five international conferences held over the past decade, the most recent one in Washington, D.C., in September 2003. Drawing on presentations at that conference, this report presents nine short case studies from around the world, where very different sorts of research evidence are being used in very different policy and practice settings. The purpose of these case studies is not to identify "proven" translation strategies but to offer insights into the ways different people are trying to better use scientific research to improve human health.
Several common themes arise from the insights offered. Communication between those who produce and those who use research is critical, though close relationships between researchers and decision makers raise questions about the blurring of roles and potential impacts on scientific autonomy. Integration of research, research utilization, and routine practice is a key goal within health care organizations and systems, though rarely achieved. Evaluation of efforts to better use evidence is crucial yet often poorly done, reducing confidence that human health is actually being improved.
Introduction
One way to fight malaria, a disease that afflicts half a billion people, is to "promote the role of science in decision making," according to a recent report of the Global Health Council. "By fully implementing what we know today, more lives can be saved and the health of millions can be improved."1 The organization's president reiterated the message: "Health decision makers must be at the forefront of putting research into action if the promise of a world in which malaria does not dictate the lives of millions is to be fulfilled."2
The prescription—to better use existing knowledge—is by now as familiar as it is urgent. In the Lancet in 2003, an international working group on child mortality declared: "Application of what we know can reduce child mortality by two-thirds. . . . [T]he capacity of countries to obtain and use information to support child health programmes will be a determining factor in reducing child mortality."3
Using scientific evidence to improve health services has been the theme of a series of five international conferences held over the past decade. The most recent one, held in Washington, D.C., in September 2003 and jointly hosted by AcademyHealth and the Agency for Healthcare Research and Quality, focused on how nations organize their health services research and how that research can be used to improve clinical practice, management strategies, policymaking, and health outcomes.4 The presentations made at that conference and discussions with participants form the basis of the short case studies covered in this report.
As the move toward an evidence-based approach to health care has gathered momentum over the past two decades, a broadening of its focus has occurred. Along with generating and disseminating good-quality evidence about what works and what does not, increased attention is being paid to strategies that try to ensure that the knowledge gained from the best evidence is actually used in practice. This move to enhance the utility of health services research means both making research evidence more usable and improving the capacity of management, policy, and clinical decision makers to use it.5
The current debates about using evidence in practice range across many different fields of study and vast academic terrain. "It's a community, a trailer park of different disciplines," mused Institute for Healthcare Improvement chief executive officer Donald Berwick during the Washington conference, "but progress occurs at the intersections."6 The nonprofit Institute for Healthcare Improvement is one of the groups that is pioneering strategies to integrate the best evidence into routine practice within health care organizations. One of the strategies involves setting up "collaboratives," whereby different organizations and researchers come together to pool expertise in order to try and quickly achieve major improvements in the use of best evidence and ultimately the quality of care. A recent example involved 22 organizations working together to better use the evidence about the appropriate care of lower back pain, with 25 percent able to show some form of improvement within 12 months.7
Attempting to render an overview or bring coherence to these fragmented theoretical discussions within the "trailer park" is beyond the scope of this report. Instead, it concentrates on nine case studies from around the world where evidence of some kind is being used in practice and progress of some kind is thought to be occurring. Before moving to the case studies, however, a few threads of discussion arising from both the conference and the relevant scientific literature are worth noting by way of background.
According to a recent systematic review of the literature about how policymakers use evidence, one of the most commonly discussed theoretical issues in that literature is what is meant by the term use.8 The most common categorization suggests three different meanings—that is, a rough consensus exists in the literature that there are three different ways in which evidence is used. First, there is direct or instrumental use of evidence, where relevant research results are seen to be directly affecting policy decisions. Second, there is selective or legitimating use of evidence, where evidence is used to legitimate or sustain a predetermined position already taken by policymakers. Third, there is enlightening use of evidence that can help enrich or deepen understanding of a problem. As we will see, evidence is used in all three ways throughout many of the case studies.
Another important question concerns what is meant by the term evidence. Commonly, we refer to evidence as the results of scientific studies, but as the case studies will show, many other forms of information and knowledge, as well as values, judgments, experiences, and intuition, routinely play a role in health care decision making, often trumping the "pure" research evidence in terms of importance and utility. Moreover, as researchers have applied more scrutiny to the strength and quality of "pure" evidence in recent years, more and more of its weaknesses have been revealed. It is now widely accepted that much of the "scientific" evidence in health care is of extremely poor quality, biased in favor of professional or commercial sponsors, producing misleading and unreliable results that overstate the benefits and play down the harms of many health care interventions. The way forward appears to involve decision makers relying, to whatever extent is possible, on the best of the available evidence—based on systematic reviews of studies with designs that attempt to minimize bias.9 Likewise, a decision-making process that is as open as possible about the conflicts of interest of its participants is seen as a desirable model, one being actively pursued in England and Wales by the National Institute for Clinical Excellence (which is described below).
One of the basic tenets of the debate about how better to move research into practice is that personal relationships and trust will help facilitate that move, through direct communication between researchers and decision makers and via knowledge-broker intermediaries.10 While this strategy is widely accepted and highly valued, it has sparked some important questions that were explored during discussions and presentations at the Washington conference. Are the webs of influence in health care that affect decision making in fact much broader than the researcher-broker-policymaker circuit involving a host of other key actors and public opinion as well?11 Is it not practically impossible for decision makers to have personal relationships and direct communications with every health services researcher? Does a close relationship between researcher and policymaker come with any cost to the researcher's autonomy? Although there are no clear answers, these questions appear to be provoking valuable discussions, which echo through a number of the following case studies.
Attempting to define and delineate the roles of researcher and decision maker provokes more fascinating tensions in discussions about translating research into practice. At the Washington conference plenary session on national policies for organizing research, Thai policymaker Suwit Wilbulproprasert and Kenneth Shine, an American researcher who is also past president of the Institute of Medicine of the National Academies of Science, advanced quite different perspectives on the nature of those roles. Using the context of his nation's recent move toward universal health insurance—covered later in this report as a case study—Wilbulproprasert argued strongly that people who combined the roles of researcher and decision maker/advocate were more effective in facilitating the translation of evidence into practice. By way of contrast, Shine suggested that such a blurring of roles could undermine the credibility of the scientist/researcher.12
A final tension worth noting is the one between the local and the global, the center and the periphery. Whereas a sense exists that evidence is often seen as "global" in nature, putting that evidence into practice is regarded as a local affair. For example, the literature about using evidence to improve health care quality stresses the importance of individual health care organizations taking the centrally produced evidence or recommendations for best practice and "reinventing" them at the local level.13 Similarly, one of the most common refrains in these debates—and from the players involved in the following case studies—is that genuine commitment from local leadership is absolutely critical to the success of any efforts to put evidence into practice. Others, including University of Melbourne researcher Lenore Manderson, have pointed out that "global" evidence often arises from the experiences of the richest countries, which can limit both the relevance and success of translational efforts in resource-poor or culturally different settings.14
In an attempt to reflect global diversity, these nine case studies vary widely, featuring different sorts of research evidence being used by different kinds of decision makers in a variety of ways in a range of contexts. The one clear and important limitation of this report is that all the case studies are portrayed in a positive light, featuring enthusiastic testimony from key protagonists. Moreover, almost all the stories and evidence advanced here to illustrate the importance of translation and implementation efforts rely on opinions, weak study designs, or unpublished conference presentations, though in addition peer-reviewed literature is relied on where available. Hence, the enthusiasm of the protagonists must be weighed against weak evaluation methods and uncertainty that health outcomes are actually being improved. However, the purpose of these case studies is not to demonstrate "proven" translation strategies but to offer brief insights into the ways different people around the world are trying to better use scientific research to improve human health.
Bangladesh: Narrowing the Gap
Summary: Between 1980 and 1990 a nongovernmental organization in Bangladesh educated more than 12 million families about a simple, inexpensive treatment for diarrheal diseases, narrowing the gap between knowledge and use. Concurrent evaluation of the project helped refine and improve the process.
Mixing water with sugar and salt to fight deadly diarrhea was "potentially the most important medical advance of this century," according to a Lancet editorial in 1978.15 This simple, inexpensive solution—often called oral rehydration therapy, or ORT—has been credited with saving a million lives a year in the developing world.16 Current estimates suggest that more than a million children still die annually from diarrheal diseases, a leading cause of childhood illness and death.17 While improved water supplies, nutrition, and development opportunities are among a host of urgent global priorities, the importance of better using our knowledge about ORT has long been widely accepted.
In the late 1970s in Bangladesh, a nongovernmental organization called the Bangladesh Rural Advancement Committee embarked on a plan to educate every household about oral rehydration therapy. Between 1980 and 1990 the organization's trained field force went home by home and delivered face-to-face teaching to more than 12 million mothers.18 During a typical session lasting 20 to 30 minutes, often involving a demonstration with a child with diarrhea, women learned how to prepare the homemade solution using "a pinch of salt, a fistful of gur (brown sugar) and a half a litre of water." Speaking about what he saw as the success of the program, Mushtaque Chowdhury, former director of research at the Bangladesh Rural Advancement Committee, told the Washington conference plenary session on using evidence to improve health services management that oral rehydration therapy "is now a part of the Bangladeshi culture."19
Demonstrating the importance of an "organizational commitment to research," Chowdhury said his Research and Education Division was responsible for evaluating and modifying the mass campaign to implement the knowledge about oral rehydration therapy. Discovering early on that mortality data were impractical to collect (in part because of the lack of birth and mortality statistics and difficulty assigning cause of death), the organization focused on measuring the process—specifically, the retention of knowledge by women and the actual use of the therapy in the home.20
An important modification introduced early in the program involved simplifying the formula from "two scoops" of sugar to "one fistful," so as to synchronize the number with "one pinch" of salt and reduce the possibility for confusion. Despite the modification, the first evaluation efforts discovered surprisingly low rates of uptake, according to Chowdhury, currently a visiting professor at Columbia University in New York. Searching for explanations, researchers discovered a lack of trust in the therapy among women, in part related to low male involvement in the program. Armed with this and other key research findings, implementation strategies were changed and expanded to include other broader educational activities, and significantly higher rates of knowledge retention and usage were subsequently achieved.21
A study published in Health Policy and Planning found evidence that some form of oral rehydration solution was being used in around 50 percent of diarrhea cases in Bangladesh, and that more than 70 percent of women could prepare a "safe and effective" solution even a decade after their household had been visited by the field-force workers. "The gap between knowledge of ORT and its use has narrowed considerably," the study authors wrote. Based on a survey of 9,000 households that had been previously targeted, the authors also found some evidence that the knowledge about oral rehydration therapy was being successfully transmitted from mother to child.22
As with any technology, there is uncertainty and controversy surrounding both the benefits and potential harms of ORT. The Health Policy and Planning authors cite another study, which found that following the mass education campaign, only 25 percent of mothers were able to make a "correct" solution. However, in that study, assessors used subjective criteria to analyze the prepared solutions, rather than the chemical laboratory analysis used in the original evaluation. Whatever the success in terms of education, major questions remain about the actual health impact of the program because evaluation efforts were focused on processes—or "surrogate outcomes"—rather than on health outcomes.23
There is also an important debate about the role of food in rehydration therapy and the type of diarrhea for which homemade or prepackaged solutions are most effective, with some authors suggesting that these solutions have only a limited role in nonhospital cases of diarrhea.24 Notwithstanding that debate, the Bangladesh Rural Advancement Committee home-to-home program has been held up as a global model for communicating lifesaving health information.25 For Chowdhury, the experience also stands out as an example of the effective use of scientific knowledge for action. "Research can justifiably claim credit for its part in this."26
Insights: Research evaluating initial implementation strategies can lead to refinements and improvements in strategies.
Canada: Reconnecting Nurses and Patients
Summary: Evidence suggests that improving nursing workloads and staffing levels can enhance nurse satisfaction and patient outcomes. Health care organizations in Canada are aggressively attempting to translate this evidence into practice. Some public hospitals are evaluating the impact on patient outcomes of giving nurses more time to learn, teach, and bring research into practice.
Addressing the nursing crisis in the health systems of many nations, Institute for Healthcare Improvement's Don Berwick told the Washington conference that "joy" must come back to the health care workplace, because "high quality cannot come from a demoralized workforce."27 In Canada, federal government figures show that of 48 occupations, nurses have the highest rates of "illness and injury-related absenteeism" of any occupational group. Every week, almost one in ten nurses has some absence from work, a rate almost double that of the average full-time Canadian worker.28
Alongside the growing evidence of absenteeism, unwanted overtime, and high levels of costly nurse turnover is strong evidence that the quality of nursing services has a direct impact on the health of patients.29 At a conference session in Washington devoted entirely to nursing, researchers presented evidence that patients were significantly less likely to die after a heart attack if they were in a hospital with higher numbers of registered nurses.30 Similarly, patients undergoing stomach surgery were significantly less likely to experience major complications if they were in intensive care units with higher nurse staffing levels.31
In Canada a concerted effort is being made to take all this research evidence and translate it into practice to simultaneously improve both the nursing workplace and patient outcomes. One person directly engaged in that translation effort is Mary Ferguson-Paré, the chief nursing executive at the publicly funded University Health Network, with three hospitals and 3,000 nurses in Toronto, Ontario: "I am on a path trying to take the research evidence, put it in practice, and then test it." Apart from using and testing best-practice nursing guidelines, Ferguson-Paré is also running two innovative demonstration projects in 2004 with high-level support from within her own organization and encouragement from within the federal and provincial governments. "Senior management," she says, "is really committed to us taking the evidence and acting on it, and the organization has the capacity to move research findings into practice."32
The first of the projects involves three nursing units in teaching hospitals across Ontario making slight increases to their nursing capacity on every shift. The plan grew out of a recent study at University Health Network in which nurses were asked how much extra time they would need per shift to satisfy patient needs adequately. The result of that study suggested nursing units should increase nurse staffing levels by the equivalent of an extra 30 minutes per nurse per shift. Nurses in these units will stick to their traditional shift lengths, so overall the amount of nursing staff will increase slightly, as will the ratio of nurses to patients. The project will run for 12 months, and researchers will track rates of absenteeism, adverse events, nurse turnover, and patient satisfaction.33
The second of the projects involves a controlled study of a new "innovation unit." Starting in early 2004, the 30 or so nurses of the orthopedic-rheumatology unit of Toronto Western Hospital will have to spend only 80 percent of their employment time on clinical work, taking the other 20 percent as release time to be spent teaching, learning, and translating evidence into practice. The idea is to moderate the nurses' workloads and increase their opportunities for continuous learning. "We want to improve the capacity of registered nurses to connect with patients and provide care," says Ferguson-Paré, who is a co-investigator of the controlled study. It will measure a host of indicators, including staff and patient satisfaction and genuine health outcomes, including the number of patient falls and the rate of medication errors.34
An important part of the context for these projects is the wider work of the Canadian Health Services Research Foundation and the Canadian Institutes for Health Research, organizations promoting the translation of evidence into practice, within nursing and more broadly across health care. One of the key academic advisers on the translational activities is the University of Toronto's Linda O'Brien-Pallas, a recipient of one of the recently created decade-long Canadian research chairs, a program designed to build both research capacity and utility, co-funded with public resources that come through both organizations. Based at the University of Toronto, O'Brien-Pallas has estimated that nursing turnover costs more than US$20,000 per nurse, and that, in total, high turnover, absenteeism, and the use of agency staff may be costing Canada C$1.5 billion a year.35
One of the desired outcomes from the current demonstration projects is that, although they require a bigger initial investment (extra nursing staff costs money), over time the projects will save resources or at least break even. One of the biggest challenges in the translation work for Ferguson-Paré is finding that extra money up front, which often calls for what she terms a "leap of faith" on the part of senior managers and funding agencies. "It really requires outstanding leadership in health care organizations to be scanning the environment, listening to the new research, and turning that into the 'how we do' the 'what we do.'"36
Insights: Organizational leadership needs to be committed to translating evidence into practice; investing up front in costly translation exercises and their evaluation may bring long-term rewards.
Chile: Evidence for Equity
Summary: Evidence about health inequity in Chile is indirectly affecting the current reforms of the national health care system and enhancing the importance of primary health care networks. An innovative team of researchers and policymakers called the Chilean Health Equity Gauge has been playing a role.
"Scientific knowledge has been one of the basic pillars of our proposal for reform, especially the knowledge about equity," according to National Committee for Health Reform executive secretary Hernan Sandoval, a chief health adviser to Chilean president Ricardo Lagos. "In discussions about what to do, everyone agreed we had to reform, but there was much disagreement on what to do. The only subject that nobody could contest was inequity."37
In recent decades Chile has experienced big increases in economic growth, decreases in poverty, and improvements in employment, education, and health, yet all groups within society have not benefited equally from these advances. Infant mortality rates in Chile remain five times higher for mothers with a basic education, compared with mothers who have completed secondary school.38 Likewise, infant mortality is more than ten times higher in southern, rural areas compared with wealthy parts of Santiago.39
Such evidence about health inequity is having an impact on health policy in Chile, according to Sandoval. This is in no small part due to the work of a small team of researcher/advocates called the Chilean Health Equity Gauge, a group actively involved in the reform process.40 The team in Chile is part of an international network focused on monitoring national inequities in health and health care services and committed to closing the gap in health status between different socioeconomic groups.41
The work of the Equity Gauge includes an innovative fusion of research, advocacy, and direct participation with stakeholders, and it has been described as an "extraordinarily effective alliance,"42 particularly in terms of building relationships between researchers and key decision makers. As part of its work, the group has produced and disseminated several reports on the problem of health inequities in Chile, as well as potential solutions.43 "The combination of advocacy and knowledge is very powerful," says Sandoval, who has worked with the group since 1999.44 A key achievement of the Equity Gauge was helping to develop and write specific health objectives for the Chilean government for the 2000–2010 decade. It has also been officially commissioned to monitor and disseminate data on health inequities as the government attempts to close the health status gap between different socioeconomic groups in Chile.45
Asked whether individual pieces of research evidence had been utilized directly in specific policy decisions, Sandoval said the impact of the evidence about health inequity was more cumulative and indirect—what might be called an "enlightening" use of evidence. Describing an interdependence between research and policy, he pointed to the fact that one of the major slogans of the Lagos party in Chile is "Desarrollo con Equidad" ("Development with Equity").46
One practical strategy for reducing health inequity being used in Chile's current reforms is a dramatic boost to the role and funding of primary health care. In 2000, at the time of the election of the Lagos government, primary care in Chile received just 12 percent of the health care budget and was still widely viewed as a "second-class system," mainly there to service the poor—essentially the poor cousin of the hospital and specialist sector. Yet in just four years, primary care's share of the national health budget has jumped to 18 percent, and there are plans to increase that to perhaps as much as 30 percent with ambitious programs involving 60 new facilities across the nation.47
Reformers such as Sandoval who believe social determinants play a key part in health see the new role of primary care as a critical interface between society and the medical system, between "public action for health" and "specific health services," as he puts it. The reforms are about enhancing the role of primary care, integrating it better with other parts of the system, and trying to change public perceptions about health and health care services. "One of the big problems is people still think they are going to find health solutions with specialists or in hospitals. We have to show that improving primary care can improve health."48
A senior health aide to President Salvador Allende in the early 1970s, Sandoval is enthusiastic about the current prospects for health reform in Chile and the use of evidence in the process. In February 2004 Sandoval attended a seminar in London, sharing the Chilean experiences of reform with policymakers in the National Health Service. Adding a historical perspective to the current changes, he explains that, in his view, although many of Chile's physicians and their representative organizations had fought hard for an end to dictatorship and the return of democratic political reform, the push for fundamental reform of the health system took a backseat—until now.49
Insights: Fusing research, advocacy, and stakeholder participation can create effective alliances. Research evidence can indirectly affect decision making by "enlightening" the process.
England and Wales: Evidence Is Not Enough
Summary: The National Institute for Clinical Excellence is translating evidence into practice by producing guidance for managers, professionals, and consumers on effectiveness and cost-effectiveness of health care interventions. Yet other forms of information and knowledge also influence decisions about cost-effectiveness. Decision making about a new, high-profile cancer drug provides a good example of where the evidence is seen as being not enough.
The National Institute for Clinical Excellence (NICE) has emerged in recent years as a strong, international example of an organization dedicated to translating evidence into practice. Using evidence-based methods, NICE facilitates systematic reviews of research about health care interventions and produces usable guidance on those interventions for people across England and Wales. No matter whether a "guidance" finds for or against a new treatment, clinicians and other health professionals are expected to take it into account, though they are not strictly bound by it.50
The aim is to help health care managers, health professionals, and consumers make better-informed decisions about how well different treatments, therapies, tests, and devices might work, and which ones give the best value for the money. Yet as senior executives from the institute wrote in Lancet Oncology in 2003, "Although NICE's approach to decision-making is evidence based, pure research evidence alone is not sufficient to make judgements."51
The official "guidance" NICE recently produced on the controversial new leukemia drug imatinib (sold under its brand names, Glivec and Gleevec) provides a fascinating insight into the way evidence can interact with other forms of knowledge on the path to implementation.52 As the NICE executives candidly elaborated in commenting on the institute's imatinib guidance, "Although the guidance produced is firmly based in published evidence, it is also considerably influenced by information from numerous other stakeholders including professional organizations, patient/carer groups, and the pharmaceutical industry."53
Appraising the effectiveness and cost-effectiveness of cancer drugs can be particularly difficult, according to Peter Littlejohns, clinical director of NICE, because there is often a lack of good-quality data from long-term randomized, controlled clinical trials.54 With imatinib, members of the NICE appraisal committee were forced to make decisions about the drug's cost-effectiveness in the absence of good, long-term data and on the basis of "surrogate" outcomes, rather than on the basis of evidence demonstrating proven advantages in morbidity or mortality.55
After releasing a very negative preliminary finding for public consultation on this much-publicized drug, the committee was persuaded to recommend in favor of the drug's use. In a section of their guidance titled "Consideration of the Evidence," the committee wrote that apart from reviewing additional updated scientific data, which helped strengthen the link between surrogate outcomes and clinically meaningful benefits, they also considered "evidence on the nature of the condition and the value placed on the benefits of imatinib from people with CML [chronic myeloid leukemia], those who represent them, and clinical experts."56
"There is no algorithm for this decision making," observes Littlejohns. A former clinician and professor of public health turned policymaker, Littlejohns stresses that making judgments and recommendations about cost-effectiveness necessitates weighing different assumptions and opinions, and that the same evidence base can be read very differently, depending on the perspective you bring to the reading. Asked whether NICE processes make decisions vulnerable to political or commercial pressures from vested interests, Littlejohns acknowledges that the public consultation procedures allow for "a form of lobbying." He argues, however, that this is legitimate, as long as it is a comprehensive consultation and all relevant conflicts of interest are made public as part of the process.57
As it happens, part of the reason NICE was created was because different health authorities across England and Wales were looking at the same evidence and making different decisions about which services or interventions to fund. For example, some local authorities supported the use of a drug class called taxanes for breast cancers, while others did not. Now if NICE makes a positive recommendation in its national "guidance"—as it did with imatinib—a local authority has to make funds available for that specific intervention within three months, even if that cuts across other priorities deemed more important at the local level.
"There will always be a tension between the center and the periphery," says Littlejohns. "If you want a health service where there is equal access to services, there is a reduction in local autonomy that you have to live with."58
Insights: Research evidence alone is often not enough for making cost-effectiveness judgments about health care interventions; decision making can involve comprehensive consultation with all stakeholders, and all conflicts of interest should be made public as part of the process.
Russia: Evidence-Friendly System Saves Lives
Summary: The redesign of a system of care for sick newborns in a region of Russia produces improvements in mortality. Evidence is used to help redesign the system, which is evaluated in five hospitals before being implemented across 43 health care facilities in a region with a population of 2 million.
In Tver Oblast in the Russian Federation in late 1998, heath professionals and managers from local health authorities and international organizations—including the Tver Oblast Health Authority, the Central Public Health Research Institute in Moscow, the Russian Ministry of Health, the U.S. Agency for Healthcare Research and Quality, and the U.S. Agency for International Development's Global Quality Assurance Project implemented by University Research Co. LLC—decided to embark on improving the care for newborns suffering respiratory distress syndrome, a condition in which a baby's lungs do not function properly, making breathing extremely difficult.59 Baseline data taken before any changes were implemented suggested that almost half of the neonates arriving at local health care facilities with this condition were suffering hypothermia, and many of them were dying in their first week of life.60
Those health professionals involved with providing the care worked with others to redesign the management of this clinical condition across five hospitals in the city of Tver, with an emphasis on finding and using the best available evidence. Most significant, the system of care was changed: A new central referral center was created; a new system of transportation with ambulances was devised; and relevant staff members at all facilities were trained in neonatal resuscitation techniques. Simultaneously, "evidence-based clinical guidelines" were developed for all stages within the new system. Critically, those implementing the changes helped evaluate their impact and, as an integral part of the improvement methodology, fed the results back into the ongoing redesign of the system.
The new system was rolled out toward the end of 1999. Within three months, the percentage of these neonates arriving at hospitals with hypothermia had fallen from more than 50 percent to almost zero, according to University Research Company's Rashad Massoud, in a presentation about the case study at the Washington conference.61 The proportion of babies dying in the first week of life also began to drop, but less dramatically and more slowly; nevertheless, in relative terms it fell 64 percent over 12 months.
"What we're trying to do is implement evidence and test different modalities for implementing evidence, with quasi-experimental research in real time," says Massoud, the project leader from the United States side of the collaboration. The teams continued to collect data on the redesign through 2003, though it is as yet unpublished. "It's showing that the gains have held."62
What was first tried on a small scale in five hospitals has since become a much larger scale-up in implementing evidence and improving care. In 2000 the program expanded from five hospitals to 43 hospitals covering all of Tver Oblast. Tracking data between 2001 and 2003 for the whole region, the researchers found improvements in the health of babies with respiratory distress syndrome: Early neonatal mortality in Tver Oblast was cut in half, from 10.8 per 1,000 in 1998 to 5.3 per 1,000 in 2002.63
Asked about the lack of comparable data from a "control" region, Massoud said, "Research is not the first priority in this project; this is a quality improvement project with multiple interventions being tested." Explaining that local health professionals were essentially redesigning and improving their own systems of care, he stressed that there was no real separation between the research and the provision of care: "The researcher is the same person as the provider." Asked about important insights gained through the project, he said: "One of the lessons here is that we can use evidence in resource-constrained settings. We can do it, and it may in fact be more wasteful not to."64
Insights: Monitor and evaluate changes on a small scale first, and use evidence at every stage in the system. Implementing evidence can work in resource-constrained settings.
South Africa: AIDS Survey Hits Home Run
Summary: A survey on HIV/AIDS in South Africa contributes to national policy change toward the virus and its treatment. The study's principal investigator explains the reasons behind this example of evidence affecting policy and practice.
It's not every day a health survey makes world headlines and then goes on to help effect changes in national health policy. But then, unlike the South African HIV/AIDS survey released in December 2002, most surveys are not commissioned and launched by Nelson Mandela. Funded in part by the Nelson Mandela Foundation and conducted by the South African Human Sciences Research Council, the first community-based survey of the HIV epidemic in South Africa serves as an important example of how evidence can interact effectively with the policy process.65
At the time of the survey's release, headlines from Johannesburg to Washington highlighted its frightening finding that almost 6 percent of South African children were HIV-positive. But alongside the alarm, the survey also produced some good news: Almost half the population reported that they were changing their sexual practices in response to the epidemic. South African men and women were changing partners less frequently and using condoms more often, suggesting the success of publicly funded prevention messages.66
However, one of the results pushed hardest by the survey investigators was the finding that more than 95 percent of South Africans wanted to see antiretroviral drugs made available to treat people with AIDS and to prevent mother-to-child transmission of the virus. Six months later the South African Cabinet reversed policy and decided to roll out an antiretroviral program. Clearly not the only factor by a long shot, and coming after years of hard work by many different groups, the Mandela survey has been publicly credited with playing a role in this important policy turnaround.67
The study's principal investigator was Dr. Olive Shisana, executive director of the HIV/AIDS research program at the Human Sciences Research Council. For her there were two key reasons why the survey evidence may have proved influential on policy: the involvement of Mandela, and the way in which key political stakeholders were briefed before the results were made public in order to ensure an informed response. "We allowed people to interrogate the results, before we published, so they understood the implications of the research findings," noted Shisana.68
According to Shisana in her presentation to the Washington conference in September 2003, the team from the Human Sciences Research Council briefed the South African president's office and the ministers of health, education, and science and technology, as well as officials from parliamentary committees and national and provincial departments.69 On the day the study findings were released, December 5, 2002, the South African government released a one-page, prewritten statement welcoming the survey and noting that the public expected more resources to be spent combating the epidemic.70 Because they were prepared beforehand, "the government responded directly to the findings," Shisana says, "and the messages did not get lost in controversy."71
Shisana is the former director general of health in South Africa and is experienced in bridging the gap between research and policy; thus, it is not so surprising that the survey had an impact. A week after the report's release the nation's deputy president was referring enthusiastically to the findings about changes in sexual attitudes and condom use. Three months later in Parliament the minister for education pointed to the survey finding that more women were infected with HIV than men (17.7 percent versus 12.8 percent) and described renewed efforts to target educational prevention messages at girls.
Most important, according to a report in the South African daily the Mail and Guardian, the survey findings (along with the lobbying efforts of Shisana, Mandela, and others targeting government and party officials) played a role in the "Cabinet U-turn on antiretroviral drug treatment for HIV/AIDS." According to the newspaper, "In a dramatic reversal after years of dithering . . . [the] government decided to ask the Department of Health 'as a matter of urgency' to develop a detailed operational plan on an antiretroviral treatment programme. Its decision was welcomed by the United Nations last week."72 Shisana adds that pressure from groups such as the Treatment Action Campaign, the Congress of South African Trade Union Organisations, churches, and many individuals also helped force the policy turnaround.73
In November 2003 the South African Cabinet finally approved an operational plan for comprehensive treatment and care, essentially rolling out a national antiretroviral program with estimates that 50,000 people could be treated in the first year.74 At present, newspaper reports suggest that major delays have occurred in that planned rollout.75
Insights: Influential and respected people can enhance impact by disseminating research findings to key decision makers before publication to prevent defensive reactions and encourage use of evidence, and by combining bad news about the extent of problems with good news about potential solutions, which may increase decision makers' willingness to listen.
Thailand: Evidence for Action on Universal Coverage
Summary: Evidence from health services researchers has played a role in Thailand's recent move to universal coverage. A senior government decision maker talks candidly about the interdependence between research and policymaking.
While many health services researchers and policymakers in the United States continue to talk about the benefits of universal coverage, their contemporaries in Thailand have gone ahead and helped make it happen. When a new national government was elected in 2001 with a commitment to universal coverage, less than 70 percent of the Thai population had health insurance. Three years later, 95 percent have coverage, and the remaining 3 million people as yet uncovered have the right to it.76
The new system offers a comprehensive benefit package covering acute, chronic, and preventive services. It is financed mainly through taxation though supplemented with a co-payment of 30 baht (about 75 cents) per medical visit from which the poor, children, and elderly are exempt. Health services research in Thailand and those who produce it have been key to both the development and implementation of the new system, known popularly as the "30 Baht Scheme."77
According to Dr. Siriwan Pitayarangsarit, who recently completed her doctoral thesis on the system, the concept of universal coverage was first communicated to the (ultimately successful) Thai Rak Thai political party by researchers during the election campaign of 2000. This happened first through personal contacts and then via a brief paper from a group of researchers within the Ministry of Public Health. Essentially a synthesis of international and national studies that discussed the feasibility of universal coverage, the paper was later disseminated in the form of a concise 24-page booklet to help build support for the reforms among political parties and nongovernmental organizations.78
"There is no better time for health services researchers and social movements than the preelection period," says Suwit Wilbulproprasert, the senior Thai civil servant with direct responsibility for financial management of the new system and its evidence-based implementation. For Wilbulproprasert, while the brief 2000 paper on universal health care was significant, the more important context was the web of personal relationships between members of the health policy "elite" and members of the aspirant political party—relationships that date back three decades to days of shared student activism.79
Following the election of the new government in 2001, research to help implement universal coverage was done both inside and outside the Health Ministry, with members of the academy recruited to support the internal policy processes. Outside research comes primarily from the Health Systems Research Institute, an independent public research management agency.
As a priority, research was required to work out exactly who was eligible for the 30 Baht Scheme (which complements two existing insurance programs in Thailand) and to determine the per capita cost to government of the new system. More broadly, researchers began evaluating different aspects of the new 30 Baht Scheme and feeding back results to facilitate further adjustments and evaluation. "We do not separate the research arm and the policy arm—we work closely together," says Wilbulproprasert, describing a blurring of the role of researcher and policymaker: "Many of our policymakers are not totally nonacademic, but most of the research they do is more practical, more ready to use. On the other hand, many academic researchers also become policy advocates at the same time—not all of them—but the ones that do, produce research that is more effective. I am not requesting [that] every researcher becomes a policy advocate, but if you want to change things you have to be a policy advocate as well, and there will be a higher chance the knowledge you generate will be used."80
Speaking during a plenary session of the Washington conference about the role of research and researchers in Thailand's move to universal coverage, Wilbulproprasert described the linkage between social movements, knowledge generation, and political activity as the "triangle that moves the mountain." According to Wilbulproprasert, the comments reference both a widely known Chinese story about how to move a mountain that stands in the way of social prosperity, and the work of a Thai senior citizen, Professor Prawase Wasi. Urging researchers and policymakers alike to work with wider social movements and the media to promote health reforms, Wilbulproprasert told the conference: "Research should not float in the air. We need to link the sky and the earth."81
Taking a different view, the chair of that same session, Dr. Kenneth Shine, then with the RAND Corporation, told conference participants, "To the extent you are a social advocate, you're not a scientist, you lose your credibility."82 Asked about Shine's comments, Wilbulproprasert said: "I agree with him. If you are too close to a political party and let down the scientific evidence, you may lose credibility. But you can definitely maintain your social credibility if you base your political advocacy on scientific evidence."83 Shine responds that good research can be used for both education and informing political advocacy, but the roles should not be too blurred: "It is very important that when researchers report their work, they do so in a form that is scientifically valid and do not include a strong advocacy position."84
Like any major reforms, the move toward universal care in Thailand has not been smooth, with, for example, important academic disagreements over the size of the co-payment and the per capita cost of the 30 Baht Scheme to government. Moreover, the expansion of coverage has been tied to a wider expansion and reform of the entire health system. In practice, more of the additional resources are flowing to poorer provinces and less to wealthy provinces in an attempt to correct traditional inequities. This, in turn, is causing much controversy. But according to Wilbulproprasert, the redistribution of the new resources is also being driven by a large amount of evidence about the previous maldistribution of facilities and human resources.85
Insights: Preelection periods provide good opportunities for health services researchers; people who mix policymaking and research can produce more effective research that is more ready to use.
United States: Best Evidence to the Bedside
Summary: During the 1990s the Veterans Health Administration reengineered its system to embed best evidence into routine practice. Evaluation of the reengineering suggests improvements in the quality of health care. Independent observers praise the strategy as an example of embedding best evidence in systems of care.
When it comes to talk of using evidence in practice, the folks at the Veterans Health Administration like to describe themselves as a "measurement laboratory for the nation."86 With 75,000 clinicians working under one roof servicing almost 7 million Americans, it is certainly a very large lab.
In the mid-1990s the Veterans Health Administration embarked on a major "reengineering" of its health system to try and improve the use of information technology, the measuring and reporting of performance, the quality of care, and health outcomes. According to a study published in the New England Journal of Medicine (NEJM) in 2003, that reengineering "appears to have resulted in dramatic improvements in the quality of care provided to veterans."87
Comparing data between 1994 and 2000, the study reported statistically significant improvements in the quality of preventive, acute, and chronic care across nine indicators used in all years. By 2000 the proportion of patients receiving appropriate care was 90 percent or greater for nine of 17 selected quality-of-care indicators. The proportion of women screened for cervical cancer jumped from 62 percent to 93 percent; the percentage of patients offered a beta-blocker after acute myocardial infarction rose from 70 percent to 95 percent.88
"We wanted to translate the best evidence to the bedside," says Jonathan Perlin, then deputy under secretary for health (now acting under secretary for health) at the Department of Veterans Affairs (VA) and one of the NEJM study authors. "We've had remarkable progress."89
One key part of the reform process is the Veterans Health Administration Quality Enhancement Research Initiative, a national system for the translation of research into practice, with a mission "to enhance the quality and outcomes of VA health care by systematically implementing clinical research findings and evidence-based recommendations into routine clinical practice." As part of this initiative, multidisciplinary, condition-specific teams gather evidence on best practice, translate that evidence into routine care processes, and then measure the impact of those translation interventions.90
Internal data (publicly available on the Web) suggest that the translation efforts may have produced real improvements in outcomes: The substance abuse team has helped increase the number of opioid-dependent veterans with access to methadone treatment; the heart disease team has helped reduce the number of veterans suffering coronary events; and the mental health team has reduced the number of patients on nonguideline doses of antipsychotics.91
Director of that initiative John Demakis says the organization is keen to share what it is learning about the "science of implementation." At the Washington conference he outlined several important lessons, including winning support from local opinion leaders and hospital leadership, building performance measures into executive contracts, customizing interventions to local contexts, using good evidence to determine best practice, evaluating as you go, responding to barriers quickly, changing the perception of research to being integrated with practice rather than outside practice, and participating in demonstration projects, which can evolve into routine processes.92
For Jonathan Lomas, executive director of the Canadian Health Services Research Foundation, the Veterans Health Administration reforms serve as a powerful example. Here is a large frontline delivery organization, Lomas wrote in a recent BMJ editorial, "using health services research to respond to the direct needs of their managers and clinicians for better information on which to base their decisions." Rather than handing over research to experts in universities, says Lomas, the research agenda is set by the organization's own needs; the research is done collaboratively by managers, clinicians, and researchers; and the results find their way directly into practice through practice guidelines, computer reminders, test ordering systems, disease management teams, and so on. "For too long, implementation of health services research has been viewed as a technical exercise in better dissemination; now is the time to highlight the importance of inter-personal links and the need to embed exchange between applied research and practice within health service delivery organizations."93
One of the most important limitations of this evidence of quality improvement is that the results come from observational studies, so there is only limited confidence that the improvements are caused by the interventions.94 Ironically, even with that important limitation, the evidence of quality improvement offered here might be the strongest of any within these nine case studies of using evidence in practice.
Another consideration is that, notwithstanding important exceptions, the nature of the quality indicators selected within the Veterans Health Administration generally tends to reward doing more rather than less, at a time when some evidence is suggesting much systemwide excess.95 And there remains uncertainty about the meaningful benefits of even well-established procedures, including mammography screening—one of the quality measures used by Veterans Affairs.96
Asked about these issues, Perlin argued that his organization was "tackling misuse and underuse first, because unlike other parts of the health sector, in the VA there are no incentives that reward overutilization." He also explained that rates for prostate-specific antigen (PSA) screening were not measured because of scientific uncertainty surrounding its benefits, whereas rates of counseling about PSA tests were measured. Similarly, he described controversial discussions within the organization about the targets for controlling lipid levels. In this case, professional cardiology groups were suggesting more aggressive reductions than another camp within the organization that he described as the "evidence jocks."97
Reflecting on the move to better use evidence within his organization, Perlin said that because the viability of Veterans Affairs was in question in the early 1990s, discomfort with change was less uncomfortable than the status quo. As part of that change, he says, quality was defined in patient-centric terms (access, satisfaction, and functional status), research was used to outline evidence-based approaches, and electronic records were used to provide timely evidence and to generate data for measurement against prespecified benchmarks. "VA measures not to record the past," he says, "but to define future performance."98
Insights: Win support for change from leadership; customize interventions to local contexts; evaluate as you go; respond to barriers quickly; and change the perception of research to being integrated with practice rather than outside practice.
The World Health Organization's Health Evidence Network
Summary: The World Health Organization is providing a new service—the recently created Health Evidence Network—that asks policymakers what policy questions they have, then summarizes the best evidence to give them brief and timely answers.
Imagine a scenario where policymakers could regularly ask questions about how to solve their nation's most pressing health problems—and get prompt answers from a credible global group summarizing the best available scientific evidence. Now imagine that the service was available to policymakers in 50 nations and that answers sometimes came back in a matter of weeks, with the world's best scientific evidence summarized in a single paragraph.
Based in Denmark, the World Health Organization's new Health Evidence Network is aiming to offer just such a service.99 "What we are doing is very different from what most people producing evidence are doing," says an enthusiastic Egon Jonsson, leader of the network since its creation in early 2003. "Most researchers are satisfied with publishing," he says. "Whether everything goes down a big black hole or not, most researchers don't care because what counts is publishing. We go to the policymakers first and ask them what questions they have—then we go to the production plants and see what evidence can be found."100
Already the outfit has commissioned and produced more than a dozen "synthesis reports" answering such broad policy-relevant questions as "What are the best strategies for ensuring quality in hospitals?" The report about this question has been downloaded from the network's Web site more than 9,000 times.
Typically, a question arrives from a health department or ministry, an independent expert is commissioned to review all the relevant evidence systematically, and a ten-page report and one-page summary are produced as quickly as possible. "When we met with a hundred policymakers during the pilot phase, they said it's fine, it's wonderful that you have a one-page summary, but can't you make it shorter?" Jonsson laughs into the phone from his Copenhagen office. "We are seriously considering having only a paragraph."101
A professor at the Karolinska Institute in Stockholm and a former head of the Swedish government agency responsible for assessing health care interventions, Jonsson concedes that with such brevity, some complexity and subtlety can be lost, but he says that researchers dealing with the policy process routinely have to boil down their work. "I know myself from experience in Sweden, the minister may only have eight minutes to hear about a huge 800-page assessment that I have been working with for three years."102
Based in Europe, the service is currently available to more than 50 nations, though the questions received are often global in relevance. For example, the Belgian Ministry of Health recently asked, "Is there any evidence to support various suicide-prevention strategies?" Keen to get answers, the ministry asked to see the draft report, which examined evidence from the evaluation of 30 different suicide-prevention strategies. While no single strategy appeared to be effective, certain prevention programs for at-risk populations appeared promising.
The initiative in some ways complements the processes of the international Cochrane Collaboration and other similar groups, where researchers independently drive the agenda for producing systematic reviews of the evidence about health care interventions.103 As with other forms of health technology assessments, the Health Evidence Network synthesis reports will, when appropriate, draw on existing systematic reviews.
For Jonsson, there is no danger in policymakers having more influence over the research agenda, because, he argues, they currently have so little: "If people make more evidence-based decisions, it is better for patients, it's better for financing, it's better for access, and it's better for quality of care."104 Yet the network is still in its early days, and questions remain about the applicability of global evidence to national questions, about the portability of the synthesis reports across different national settings, and about how these reports might be maintained or updated.
Insights: Asking policymakers what they want may increase the relevance and impact of research; short summaries of one page and even one paragraph are desirable.
Common Themes: Communication, Integration, Evaluation
Several common themes emerge from an informal synthesis of the insights offered by this diverse group of case studies of using evidence in practice. They are summarized here as being communication, integration, and evaluation.
Communication
Several different modes of communication emerge as being important to translating evidence into practice. First, for those advocating better use of evidence, communicating with organizational management or system leadership is deemed highly valuable, in order to secure support, ownership, and "buy-in" for change. This was a key lesson learned within the U.S. Veterans Health Administration transformation and is stated as an important precondition for change within the Canadian nursing case study. In the South African example, communicating controversial research findings to key decision makers before publication was credited with enhancing research impact and utilization.
Second, in a broader sense, better communication between those who produce and those who use research evidence for decision making is widely regarded as vital. In Thailand's move toward universal health insurance coverage and in Chile's move to reduce health inequities, close working relationships between researchers and decision makers were seen as increasing the effectiveness of research and improving policy processes. In both situations, however, that closeness raises questions about a blurring of roles that some regard as detracting from dispassionate scientific autonomy. In the case of the World Health Organization's new Health Evidence Network, it is as yet too early to assess the success of innovative research processes initiated by specific communications from senior decision makers.
Third, some observers see great value in better communication between not only those involved in producing and using research, but also communication between researchers, decision makers, and the wider set of stakeholders affected by the use of that research. For the managers of the National Institute for Clinical Excellence in England and Wales, consultation with a wide range of stakeholders, including physicians, patients, and product manufacturers, during the technology assessment process is seen as highly desirable. Others stress that such consultation needs to be rigorously structured, transparent, and comprehensive if it is to inform rather than confuse or undermine evidence-based decision making.105
Integration
Integration of research, research utilization, and practice is a key goal within health care organizations and systems, though it is rarely achieved. Within this group of case studies, the Bangladesh Rural Advancement Committee, the U.S. Veterans Health Administration, and the health authorities in the Tver Oblast region of Russia have all attempted such integration, though the limitations of evaluation procedures leave uncertainty about their success. Nevertheless, those promoting evidence-based decision making see embedding "exchange between applied research and practice within health service delivery organizations" as an important path to improving health.106
Evaluation
Evaluation of efforts to better use evidence is crucial, yet often poorly done, reducing confidence that human health is actually being improved in the process. In none of these case studies is there strong, high-quality evidence from evaluation studies proving that using research in practice has improved health outcomes. In Bangladesh there is limited "surrogate outcome" evidence that mothers retained knowledge of simple effective solutions for a common health problem. In Russia there is before-and-after data suggesting improvements in neonatal health from system reform, though as yet no published data comparing the target areas with "control" areas. Similarly, in the United States, the large body of evaluative data from the Veterans Health Administration comes from observational studies.
Ironically, while evaluation is universally regarded as an integral part of any attempt to better translate evidence into practice, the degree of rigor applied to the evaluation process does not yet match the enthusiasm of its proponents.
Notes
3Bellagio Study Group on Child Survival 2003.
4Many of the presentations are available at http://www.icsbhs.org under the "Agenda" section.
5Observations based in part on author's interview with Jonathan Lomas, executive director, Canadian Health Services Research Foundation, 2003.
6Author's interview with Don Berwick, chief executive officer, Institute for Healthcare Improvement, 2003.
7See http://www.ihi.org/collaboratives/ (accessed May 10, 2004). See also Deyo et al. 2000; Headrick 2000.
11Gerlier-Forest 2003.
12Author's notes of conference discussions.
14Manderson 2003.
15Water with Sugar and Salt (editorial) 1978.
17Parashar, Bresee, and Glass 2003.
19Chowdhury 2003.
20Interview with Mushtaque Chowdhury, visiting professor, Columbia University, 2003.
21Ibid.
23Ibid.
25Black 1985.
26Chowdhury 2003.
27Berwick 2003b.
28Office of Nursing Policy, Health Canada 2004.
29O'Brien-Pallas 2003.
30Allison 2003. See also Person et al. 2004.
31Dang 2003. See also Dang et al. 2002.
32Author's interview with Mary Ferguson-Paré, chief nursing executive, University Health Network, 2004 (hereafter Ferguson-Paré interview).
33Ibid.
34Ibid.
35O'Brien-Pallas 2003.
36Ferguson-Paré interview.
37Author's interview with Hernan Sandoval, executive secretary, National Committee for Health Reform, 2004 (hereafter Sandoval interview).
39Sandoval interview.
40http://www.gega.org.za/profiles/chile.php (accessed February 4, 2004).
42Pittman 2003.
43http://www.gega.org.za/profiles/chile.php (accessed February 4, 2004).
44Sandoval interview.
45Vega and Sandoval 2003.
46Sandoval interview.
47Ibid.
48Ibid.
49Ibid.
50http://www.nice.org.uk/cat.asp?c=57705 (accessed May 10, 2004).
51Littlejohns, Barnett, and Longson 2003.
52http://www.nice.org.uk/pdf/TA70_Imatinib_fullguidance.pdf (accessed May 10, 2004).
53Littlejohns, Barnett, and Longson 2003.
54Author's interview with Peter Littlejohns, clinical director, National Institute for Clinical Excellence, 2004 (hereafter Littlejohns interview).
55Littlejohns, Barnett, and Longson 2003.
56http://www.nice.org.uk/pdf/TA70_Imatinib_fullguidance.pdf (accessed May 10, 2004).
57Littlejohns interview.
58Ibid.
59The USA-Russia Joint Commission on Economic and Technological Cooperation 2001.
60Massoud 2003.
61Ibid.
62Interview with Rashad Massoud, director, Quality Performance Institute, 2003 (hereafter Massoud interview).
63Massoud 2003.
64Massoud interview.
65Human Sciences Research Council 2002.
66Ibid.
67AIDS: Ministers Revolt 2003.
68Author's interview with Olive Shisana, executive director, Human Sciences Research Council, 2003 (hereafter Shisana interview).
69Shisana 2003.
70http://www.doh.gov.za/docs/pr/2002/pr1205.html (accessed May 10, 2004).
71Shisana interview.
72AIDS: Ministers Revolt 2003.
73Shisana interview.
74See http://www.aegis.com/news/re/2003/RE031116.html (accessed May 10, 2004).
76Author's interview with Suwit Wilbulproprasert, deputy permanent secretary, Ministry of Public Health, 2004 (hereafter Wilbulproprasert interview).
77Pitayarangsarit 2003.
78Ibid.
79Suwit Wilbulproprasert's comments at Washington conference plenary session, in possession of author.
80Wilbulproprasert interview.
81Suwit Wilbulproprasert's comments at Washington conference plenary session, in possession of author.
82Kenneth Shine's comments at Washington conference plenary session, in possession of author.
83Wilbulproprasert interview.
84Author's interview with Kenneth Shine, executive vice chancellor for health affairs, University of Texas, 2004.
85Wilbulproprasert interview.
86Personal communication with Jonathan Perlin, deputy under secretary for health, Department of Veterans Affairs, 2004.
88Ibid.
89Author's interview with Jonathan Perlin, 2004 (hereafter Perlin interview).
90For more details on this program, see http://www1.va.gov/hsrd/publications/ (accessed May 27, 2004).
91See http://www1.va.gov/resdev/prt/QUERIQuarterly_VO5_NO1.pdf (accessed May 10, 2004).
92Demakis 2003.
93Lomas 2003.
95Fisher et al. 2003a, b.
97Perlin interview.
98Ibid.
99See http://www.who.dk/HEN/20030703_1 (accessed January 14, 2004).
100Author's interview with Egon Jonsson, leader, Health Evidence Network, 2004 (hereafter Jonsson interview).
101Ibid.
102Ibid.
103See http://www.cochrane.org (accessed May 10, 2004).
104Jonsson interview.
105E-mail communication from Andrew Oxman, director, Department of Health Services Research, Norwegian Directorate for Health and Social Welfare, 2004.
106Lomas 2003.
References
AIDS: Ministers Revolt. 2003. Mail and Guardian (Johannesburg, South Africa), August 15.
Allison, J. 2003. Presentation at Washington conference. Available at http://www.icsbhs.org/presentations/allison.pdf (accessed May 10, 2004).
Almroth, S., and M.C. Latham. 1995. Rational Home Management of Diarrhoea. Lancet 345:709–11.
Bellagio Study Group on Child Survival. 2003. Knowledge into Action for Child Survival. Lancet 362:323–7.
Berwick, D. 2003a. Disseminating Innovations in Health Care. Journal of the American Medical Association 289:1969–75.
Berwick, D. 2003b. Presentation at Washington conference. Available at http://www.icsbhs.org/presentations/berwick.pdf (accessed May 10, 2004).
Black, R.E. 1985. Communication for Improved Health Services. Development Communication Report 51:1–2.
Chowdhury, M. 2003. Presentation at Washington conference. Available at http://www.icsbhs.org/presentations/chowdhury.pdf (accessed May 10, 2004).
Chowdhury, M., F. Karim, S. Sarkar, R. Cash, and A. Bhuiya. 1997. The Status of ORT in Bangladesh: How Widely Is It Used? Health Policy and Planning 12(1):58–66. Available at http://heapol.oupjournals.org/cgi/reprint/12/1/58.pdf (accessed May 10, 2004).
Dang, D. 2003. Presentation at Washington conference. Available at http://www.icsbhs.org/presentations/dang.pdf (accessed May 10, 2004).
Dang, D., M.E. Johantgen, P.J. Pronovost, M.W. Jenckes, and E.B. Bass. 2002. Postoperative Complications: Does Intensive Care Unit Staff Nursing Make a Difference? Heart and Lung: The Journal of Critical Care 31:219–28.
Demakis, J. 2003. Presentation at Washington conference. Available at http://www.icsbhs.org/presentations/demakis.pdf (accessed May 10, 2004).
Deyo, R., M. Schall, D. Berwick, T. Nolan, and P. Carver. 2000. Innovations in Education and Clinical Practice: Continuous Quality Improvement for Patients with Back Pain. Journal of General Internal Medicine 15:647–55.
Fisher, E.S., D.E. Wennberg, T.A. Stukel, D.J. Gottlieb, F.L. Lucas, and E.L. Pinder. 2003a. The Implications of Regional Variations in Medicare Spending, Part 1, The Content, Quality, and Accessibility of Care. Annals of Internal Medicine 138:273–87.
Fisher, E.S., D.E. Wennberg, T.A. Stukel, D.J. Gottlieb, F.L. Lucas, and E.L. Pinder. 2003b. The Implications of Regional Variations in Medicare Spending, Part 2, Health Outcomes and Satisfaction with Care. Annals of Internal Medicine 138:288–98.
Gerlier-Forest, P. 2003. Presentation at Washington conference. Available at http://www.icsbhs.org/presentations/forest.pdf (accessed May 10, 2004).
Global Health Council. 2003. Reducing Malaria's Burden: Evidence of Effectiveness for Decision Makers, Technical Report, 6 (December). Available at http://www.globalhealth.org/view_top.php3?id=384 (accessed May 10, 2004).
Global Health Council. 2004. Reducing Malaria's Burden: Evidence of Effectiveness for Decision Makers, Technical Report (press release). Available at http://www.globalhealth.org/assets/press/pr-aids_012104.pdf (accessed May 10, 2004).
Headrick, L. 2000. Commentary: Two Kinds of Knowledge to Achieve Better Care. Journal of General Internal Medicine 15:675.
Human Sciences Research Council. 2002. The Impact of HIV/AIDS on the Health Sector: National Survey of Health Personnel, Ambulatory and Hospitalised Patients and Health Facilities. Available at http://www.hsrcpublishers.co.za/index.html?health_sector.html~content (accessed May 10, 2004).
Innvaer, S., V. Gunn, M. Trommald, and A. Oxman. 2002. Health Policy-Makers' Perceptions of Their Use of Evidence: A Systematic Review. Journal of Health Services Research and Policy 7:239–44.
Jha, A., J. Perlin, K. Kizer, and R. Dudley. 2003. Effect of the Transformation of the Veterans Affairs Health Care System on the Quality of Care. New England Journal of Medicine 348:2218–27.
LaFraniere, S. 2004. South Africa Is Criticized for Delay in AIDS Treatment. New York Times, February 20.
Littlejohns, P., D. Barnett, and C. Longson. 2003. The Cancer Technology Appraisal Programme of the U.K.'s National Institute for Clinical Excellence. Lancet Oncology 4:242–50.
Lomas, J. 2003. Health Services Research (editorial). BMJ 327:1301–2.
Manderson, L. 2003. Presentation at Washington conference. Available at http://www.icsbhs.org/presentations/manderson.pdf (accessed May 10, 2004).
Massoud, R. 2003. Presentation at Washington conference. Available at http://www.icsbhs.org/presentations/massoud1.pdf (accessed May 10, 2004).
Moynihan, R. 2004. Evaluating Health Services: A Reporter Covers the Science of Research Synthesis. New York: Milbank Memorial Fund.
O'Brien-Pallas, L. 2003. Presentation at Washington conference. Available at http://www.icsbhs.org/presentations/obrienpallas.pdf (accessed May 10, 2004).
Office of Nursing Policy, Health Canada. 2004. Trends in Illness and Injury-Related Absenteeism and Overtime among Publicly Employed Canadian Registered Nurses, 1987–2002.
Olsen, O., and P. Gřtzsche. 2001. Cochrane Review on Screening for Breast Cancer with Mammography. Lancet 358:1340–2.
Parashar, U., J. Bresee, and R. Glass. 2003. The Global Burden of Diarrhoeal Disease in Children. Bulletin of the World Health Organisation 81:236. Available at http://www.who.int/bulletin/volumes/81/4/en/Editorial2.pdf (accessed May 10, 2004).
Person, S.D., J.J. Allison, C.I. Kiefe, M.T. Weaver, O.D. Williams, R.M. Centor, and N.W. Weissman. 2004. Nurse Staffing and Mortality for Medicare Patients with Acute Myocardial Infarction. Medical Care 42:4–12.
Pitayarangsarit, S. 2003. The Introduction of the Universal Coverage of Health Care Policy in Thailand: Policy Responses. Ph.D. thesis, Health Policy Unit, Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, University of London.
Pittman, P. 2003. Health Equity Research: Beyond the Sound of One Hand Clapping. International Exchange for Health Services Research and Policy: Learning from Cross-Country Comparisons. Available at http://www.academyhealth.org/2003/presentations/pittman2.pdf (accessed June 28, 2004).
Shisana, O. 2003. Presentation at Washington conference. Available at http://www.icsbhs.org/presentations/shisana.pdf (accessed May 10, 2004).
The USA-Russia Joint Commission on Economic and Technological Cooperation. 2001. (Re)Designing the System of Care for Neonates Suffering from Respiratory Distress Syndrome. Tver Oblast. Available at http://www.qaproject.org/pubs/PDFs/Russian/neonat501.pdf (accessed May 27, 2004).
Vega, J., L. Jadue, I. Delgado, R. Burgos, F. Brown, F. Marin, and V. Zuńiga. 2002. Disentangling the Pathways to Health Inequities, Chilean Health Equity Gauge Report. Available at http://www.paho.org/English/HDP/Equity-Chile.pdf (accessed June 28, 2004).
Vega, J., and H. Sandoval. 2003. Building the Bridge between Knowledge in Health Equity and Practice: The Chilean National Objectives for the 2000–2010 Decade. Paper commissioned by AcademyHealth for the meeting "Health Equity Research: Beyond the Sound of One Hand Clapping," Rockefeller Conference Center, Bellagio, Italy (April).
Water with Sugar and Salt (editorial). 1978. Lancet 2:300–1.
Web Sites
http://www.aegis.com/news/re/2003/RE031116.html (accessed May 10, 2004).
http://www.cochrane.org (accessed May 10, 2004).
http://www.doh.gov.za/docs/pr/2002/pr1205.html (accessed May 10, 2004).
http://www.gega.org.za/profiles/chile.php (accessed February 4, 2004).
http://www.icsbhs.org (accessed May 10, 2004).
http://www.ihi.org/collaboratives/ (accessed May 10, 2004).
http://www.nice.org.uk/cat.asp?c=57705 (accessed May 10, 2004).
http://www.nice.org.uk/pdf/TA70_Imatinib_fullguidance.pdf (accessed May 10, 2004).
http://www.who.dk/HEN/20030703_1 (accessed January 14, 2004).
http://www1.va.gov/hsrd/publications/ (accessed May 27, 2004).
http://www1.va.gov/resdev/prt/QUERIQuarterly_VO5_NO1.pdf (accessed May 10, 2004).
DOI # 10.1599/0409Moynihan
The Author
Ray Moynihan is a Washington, D.C.–based reporter with an international reputation for quality medical journalism. A long-time radio and television reporter with the Australian Broadcasting Corporation, he has also published original studies, opinion pieces, and investigative reports in the New England Journal of Medicine, Lancet, BMJ, and the Medical Journal of Australia. He has also written previously for the Milbank Memorial Fund and is the author of Too Much Medicine? (ABC Books, 1998). In 1999 he was a Harkness Fellow in Health Care Policy at Harvard University. His most recent award is for outstanding journalism, given by the Medical Journalists' Association in Britain in 2003. He is currently a visiting editor and contributor to BMJ. His television documentary and book, Selling Sickness, will be available in 2005.
Selected Publications of AcademyHealth
A complete list of AcademyHealth's reports can be viewed online at www.academyhealth.org.
State of the States 2004
January 2004
by Isabel Friedenzohn, Elizabeth Greenbaum, Madeleine Konig, Benjamin Wheatley, LeAnne DeFrancesco, Jeremy Alberga, and Christina Folz
http://www.statecoverage.net/pdf/stateofstates2004.pdfHealth-Based Risk Assessment: Risk-Adjusted Payments and Beyond
January 2004
by Kathryn E. Martin, Sharon B. Arnold, and Deborah L. Rogal
http://www.hcfo.net/pdf/riskadjustment.pdfMapping State Health Insurance Markets, 2001: Structure and Change
September 2003
by Deborah Chollet, Fabrice Smieliauskas, and Madeleine Konig
http://www.statecoverage.net/pdf/mapping2001.pdfUsing Rural Health Networks to Address Local Needs: Five Case Studies
July 2003
by Ira Moscovice and Walter Elias
http://www.academyhealth.org/ruralhealth/casestudies.pdfFinancing End-of-Life Care: Challenges for an Aging Population
February 2003
by Bonnie Austin and Lisa Fleisher
http://www.hcfo.net/pdf/eolcare.pdfState of the States 2003
January 2003
by Christina Folz, Isabel Friedenzohn, LeAnne DeFrancesco, Madeleine Konig, Lesly Hallman, and Jeremy Alberga
http://www.statecoverage.net/pdf/stateofstates2003.pdfFor further information on AcademyHealth or this report, please write to AcademyHealth, 1801 K Street, NW, #701-L, Washington, D.C. 20006, call at (202) 292-6700, fax (202) 292-6800, or visit the organization's Web site, www.academyhealth.org.
(To request a bound copy of this report, click here.
To see a complete list of Milbank reports, click here.
Be sure to specify which report you want,
your name, mailing address, and phone number.)Milbank Memorial Fund
645 Madison Avenue
New York, NY 10022
(c) 2004 Milbank Memorial Fund
The Milbank Memorial Fund is an endowed operating foundation that engages in nonpartisan analysis, study, research, and communication on significant issues in health policy. In the Fund's own publications, in reports or books it publishes with other organizations, and in articles it commissions for publication by other organizations, the Fund endeavors to maintain the highest standards for accuracy and fairness. Statements by individual authors, however, do not necessarily reflect opinions or factual determinations of the Fund.
This file may be redistributed electronically or in print as long as it remains wholly intact, including this notice and copyright. The Fund will freely distribute this document in its original published form on request.
Printed in the United States of America.
Online producer: Stephanie Moe-Quiggle
ISBN 1-887748-60-1
     
|