INTRODUCTION

The viability and quality of the health care system is a central concern for Canadians. Between 1991 and 2000, the proportion of Canadians rating the health care system as excellent or very good declined from about 60 percent to about 25 percent (Canadian Institute for Health Information 2001). Canadian seniors were less likely than their counterparts in the United States, New Zealand, and Australia to rate the medical care they received as excellent, and about one in five felt the system needs to be rebuilt (Schoen et al. 2000). For Canadian policymakers and service providers, management of health expenditures is an important priority. Between 1990 and 2000 health expenditures rose from 9.0 percent to 9.2 percent of gross domestic product and from $2,181 to $3,152 per capita (Organization for Economic Cooperation and Development 2002). The concurrent pressures of public anxiety about health care and persistent fiscal constraints place extraordinary demands on those who must make decisions about the future direction of the health care system.

The catchphrase evidence-based decision making (EBDM) currently permeates clinical practice, policy development, and management in health care in Canada and abroad (Geddes and Carney 2001; McKibbon, Eady, and Marks 1999; Muir Gray 1997). The primary appeal of EBDM, promoted by Sackett and colleagues (1997) and others, is the promise that objectivity, evenhandedness, and precision will replace arbitrariness, bias, and error in decision making. The goal is to systematically integrate the clinical, managerial, and political expertise that a decision maker gains through personal experience with the best available evidence derived from research. Because the base of knowledge in health care is relatively new, limited, and very fluid, moving to an evidence-based approach to decision making is essential.

Health care is becoming increasingly complex. The needs of the populations served are in constant flux. Competent decision makers must be able to

• Frame the questions that must be answered in evidence-based terms
• Access relevant, timely, valid, and reliable data
• Evaluate evidence from multiple perspectives
• Apply judgments about that evidence directly to their current experience

The wide availability and use of microcomputers in the 1980s and development of the Internet in the 1990s marked the onset of the so-called Information Age. The number of scientific journals devoted to health care—both print journals and electronic publications—has exploded. There is the expectation that evidence, if not knowledge, is now at everyone's fingertips, simply awaiting the entry of the right set of key words in a favorite search engine. With the evidence-based part of the formula taken care of, decision making should be easy—or so the thinking goes.

The reality faced by health care decision makers is usually far from this optimistic ideal. First, the absence of universally accepted data standards raises first the question of the comparability of one study to another. Evaluating the evidence is difficult, and it will remain so until appropriate standards are put in place and adhered to throughout the spectrum of professions, organizations, and institutions that make up the health care "system." Moreover, other issues will continue to affect decisions no matter how compelling the evidence. Personal perspectives, local preferences, cultural differences, political objectives, and other evidence-independent factors will not go away. Decisions may be evidence-based in part, but they will almost certainly also be values-based (Smith 1989; Hunter 2001).

A further problem is that policymakers, managers, and clinicians may have insufficient expertise to analyze data and interpret findings effectively. Those who would practice EBDM must be able to do more than simply retrieve evidence; they must also be able to evaluate it and weigh its merits against other evidence that may point in different directions. As Sackett and colleagues (1997) and Muir Gray (1997) note, this requires a union of analytic skills, substantive knowledge, and judgment that draws from professional experience without being limited by it.

It is not uncommon for health policymakers to find that there are simply no usable data providing evidence for competing policy or managerial options. To give but one of myriad potential examples, the absence of national, standardized data makes it virtually impossible to draw any inferences about the cost-effectiveness, quality, or outcomes of community and institutional services for the elderly and persons with disabilities in Canada (Advisory Council on Health Infostructure 1999).

The experience of Ontario's Complex Continuing Care (CCC)1 hospitals and units provides a useful case study of the transition from anecdote-based to evidence-based decision making. Numerous gaps remain in the evidence about CCC available to policymakers, managers, and consumers, but the Ontario Ministry of Health and Long Term Care's 1995 decision to mandate use of the Minimum Data Set 2.0 (MDS 2.0; see Morris et al. 1997) represents an important step forward in building a foundation of evidence to support future decision making (Hirdes and Carpenter 1997).

THE EVOLUTION OF EVIDENCE IN COMPLEX CONTINUING CARE

Ontario is Canada's most populous province, with 11.7 million people. About 13 percent of its population is aged 65 and over. It is one of the few provinces in Canada to provide hospital-based care for the frail elderly who are not in the acute phase of illness. Through the early 1990s, persons requiring long-term institutional care received it from a mix of facilities, including

• Chronic care hospitals (as they were then called), which existed either as stand-alone facilities or as components of acute care hospitals
• Long-term care (LTC) facilities, including nursing homes (largely for-profit) and homes for the aged (not-for-profit)2
• To a limited extent, retirement homes (also largely for-profit)

Complex continuing care (previously known as chronic care³) hospitals and units currently contain 7,455 beds in Ontario, whereas nursing homes and homes for the aged together account for 58,514 beds (with facilities containing an additional 19,000 beds now in various stages of construction). Historically, admission to these facilities was typically based on the expected hours of nursing care required by the patient. In the mid- to late 1980s, however, rising health care costs forced a reduction in the number of acute care hospital beds, and the number of CCC and LTC⁴ beds also shrank in relation to the perceived need. The acuity of illness and complexity of required care increased quickly in all LTC facilities. The differences between LTC and CCC became less distinct, and the justification of funding for the much more expensive CCC hospital beds came into question.

Chronic Care Role Study

The 1993 Chronic Care Role Study (Hay Group 1993) reported the results of an 18-month effort that combined a survey of Ontario's 229 public hospitals with an extensive series of site visits and public consultations. The study produced a set of 42 recommendations to improve CCC service delivery and to enhance the partnership between CCC and long-term care facilities. The report recommended that CCC hospitals and units target a medically complex patient population and serve as centers of excellence for research and education related to the needs of persons with complex and/or unstable chronic illnesses. The report also identified the paucity of health information on CCC, noting that systems for assessment of needs, classification for funding, and tracking outcomes were inadequate.

Not unexpectedly, the responses of the Ontario Hospital Association (OHA) and the Council of Chronic Hospitals of Ontario expressed both support for and criticism of the various recommendations of the role study (OHA 1993). The most strongly stated disagreements concerned questions such as, Who should be in chronic (complex continuing) care? By how much should the bed capacity for such care in Ontario be reduced? What would be an appropriate funding level for CCC compared with LTC? Regrettably, this debate was essentially bereft of data on which the opponents could base their arguments. While it was easy to obtain consensus on points such as the need to respect the dignity and autonomy of patients, major policy questions related to funding and care needs remained the subject of speculation and argument rather than analysis. According to the OHA response, the role study "consultants' recommendations suffer[ed] overall from a lack of adequate information to justify their conclusions," and their recommendations on funding levels were "graphic evidence that the lack of an information system . . . result[ed] in inappropriate decision making" (OHA 1993, 11). Without agreement on the adequacy or veracity of the evidence, agreement on the role study's conclusions and the "appropriateness" of the recommendations proved hard to reach.

Implementation of the Minimum Data Set and Resource Utilization Groups (RUG-III)

The Ontario Joint Policy and Planning Committee (JPPC) is a partnership of the Ontario Ministry of Health and Long Term Care and the OHA. Although it has played a role in a variety of policy areas, it has been particularly active in refining the basis for funding of hospital-based care. Its committees and working groups typically comprise various stakeholders, including ministry and OHA representatives, hospital executives, clinicians, and researchers.

In 1994 the JPPC established the Chronic Care and Rehabilitation Working Group (CCRWG), which it charged with evaluating available patient classification systems for CCC and with recommending which system should be implemented in Ontario. This work was directly relevant to the recommendation by the Chronic Care Role Study that the "Ministry of Health in collaboration with Ontario hospitals should develop a methodology to determine the necessary funding for the new responsibilities of chronic hospitals and units" (OHA 1993, 23). The aim was to establish a more equitable, needs-based approach to funding CCC, which would replace the existing model of funding individual hospitals based on historical trends.

Although the CCRWG was charged with examining all possible systems, the options it considered rapidly narrowed to existing classification systems that presented at least some evidence of prior and ongoing scientific evaluation. Extensive consideration was given to three systems: the Alberta Resident Classification System (ARCS; Alberta Health 1988), Resource Utilization Groups (RUG-III; Fries et al. 1994), and the Function Related Groups (FRGs) being developed for the Functional Independence Measure (FIM; Stineman et al. 1994). Among these, ARCS was the initial front-runner. It had been developed in Canada and was already being used to fund long-term care facilities in Ontario. The ministry had put considerable effort into establishing an infrastructure to support ARCS-based funding, and it seemed to make sense for CCC's funding system to be the same as, or at least compatible with, the system used for LTC.

The CCRWG carried out a study in which 812 patients from complex continuing care and rehabilitation hospitals and units were assessed by each facility's nurses using a research instrument that combined the Minimum Data Set, the ARCS assessment used in Ontario, and the FIM. The objective was to determine how the various classification systems would group the same sample of patients. Although FIM data were gathered, the research on FRGs had not been completed to the point that it was possible to evaluate the applicability of that system to the population being studied. The study led to the conclusion that RUG-III, based on MDS data, was the more appropriate classification for CCC hospitals and units because (1) the MDS and RUG-III had undergone more rigorous scientific testing than ARCS; (2) ARCS was insensitive to clinical complexity; (3) MDS had already been implemented nationally in the United States, and there was evidence that its use was expanding internationally; (4) New York State was successfully using RUG-III as a basis for funding nursing homes; and (5) in addition to its use (with RUG-III) as a basis for patient classification, the MDS lent itself to clinical applications that would benefit patients directly (Hirdes et al. 1996; Hirdes 1997). On the recommendation of the JPPC, the Ontario Ministry of Health and Long Term Care made it mandatory for all CCC hospitals and units in the province to assess patients on intake and annually with the full MDS 2.0 and every three months with shorter quarterly MDS assessments. This recommendation also gained support from the Chronic Care Implementation Task Force (1995), which took the position that the JPPC and the ministry should "continue with the development of RUG-III and the Minimum Data Set for use in chronic care hospitals and units" and should "examine the feasibility of coordinating classification systems across sectors of health care."

On July 1, 1996, Ontario became the first province in Canada to implement MDS 2.0 on a province-wide basis. For the first time, CCC could forsee a comprehensive, province-wide health information system that would provide data on clinical characteristics, resource needs, quality, and outcomes of care. Implementation, however, was not smooth. A variety of factors made the introduction of MDS 2.0 in Ontario a substantial challenge:

1. The CCRWG was disbanded prematurely in the belief that further work on its part would not be required.
2. Only one large-scale pilot study of the MDS 2.0 had been completed in Ontario (that study was done in Toronto only).
3. There was no pre-existing working relationship with interRAI, the international research group that developed and owns the rights to the MDS.
4. Expertise in the clinical, quality, managerial, and policy applications of the MDS and other CCC data was limited to a handful of individuals.
5. The mandate required electronic submission of data, but interRAI had not licensed any software vendors to sell MDS software in Canada.
6. The role of the new Canadian Institute for Health Information (CIHI) with respect to MDS 2.0 was undefined, beyond the stipulation that it would train users and act as a repository for the CCC data set.

At this stage, an entirely new and powerful decision maker entered the scene. In the spring of 1996, the relatively newly elected government of the Province of Ontario introduced legislation5 that led to creation of the Health Services Restructuring Commission (HSRC), whose fundamental purpose would be to initiate major change in the organization and provision of health care services in Ontario, and to Ontario hospitals in particular.

Canada's other nine provinces had been establishing regional health authorities and gradually making other changes in health care organization and provision. In Ontario, however, four previous governments of different political persuasions had left the status quo relatively undisturbed despite the repeated recommendations for substantial change made over the years by many committees, commissions, task forces, and other groups, as well as the increasingly severe financial constraints placed on health care and other programs, which flowed from the federal and provincial governments' determination to escape and recover from deficit financing of publicly funded programs. (Health care was, and remains, by far the biggest of such programs.)

The HSRC's Role

The HSRC was to break the logjam, primarily by making major and dramatic changes in the organization of Ontario's public hospitals. Restructuring these highly visible, emotionally laden icons of Medicare would demonstrate to the providers of health care and the general public that change for the better in health care was not just desirable, but possible—and that it was going to happen.

The commission's overarching mandate was to foster progress toward a genuine system of health care services. The legislation passed by the Legislature of Ontario6 specified a "sunset" for the HSRC in four years (the commission was to function from April 1966 until March 2000) and gave the commission two powers:

• The power to restructure the province's approximately 220 public hospitals, including the power to direct them to close, merge, shift programs, or make any other change the commission considered to be in the public interest. The commission's directions were challenged unsuccessfully in the courts several times over the next four years. They were binding both on the affected hospitals and on the government.
• The power to make recommendations to the government, through the Minister of Health, on how best to restructure other elements of the so-called health services system, including nursing homes, home care, LTC, rehabilitation care, mental health, CCC, primary care, and so on.

Throughout most of its mandate the HSRC was made up of 11 members (including the chair), all volunteers who were each paid one dollar per year. These four women and seven men were a highly diverse group. They hailed from throughout the province—from a small community in the thinly populated north to Toronto, Canada's largest city. Some were English speakers; others were francophone. They brought to the commission experience gained in a variety of employment backgrounds—both private and public sector—and in a broad spectrum of vocations, which ranged from an owner/operator of a small business to a corporate magnate. Members included a physician, a nurse, an educator/researcher, a lawyer, and a senior public service manager. Some were policymakers; others were implementers of policy. Some were health care providers; others consumers. Their experience in health care encompassed international public health, health professional education and practice, acute and long-term care, institutional management, governance, and philanthropy. No member was charged with representing a particular sector or constituency. Each was expected to bring his or her own perspective to bear on each decision, so that the outcomes would represent as faithfully and objectively as possible the best interests of the public throughout the province.

The HSRC was, above all, an apolitical body. It carefully considered every element in the complex calculus of health care decision making except one. That excluded element was the decision's political impact, either on the government of the day or over the long term. Despite dire predictions by some politicians and others, the results of the provincial election that was held partway through the HSRC's mandate appeared completely unaffected by the commission's decisions. This was true even in areas where those decisions were thought by some to have been contrary to the governing party's partisan interests. The government was reelected by a substantial majority.

Making Decisions with Sparse Data

For the first two years of its mandate the HSRC focused on hospital restructuring, beginning with Thunder Bay, a multi-hospital community in northwestern Ontario. It soon became obvious that there were serious deficiencies in the available data—data that were needed as evidence on which to base decisions of many kinds.

Among such decisions were ones related to the capacity of existing health services and requirements for long-term care, including the required number of beds, whether provided in CCC hospitals and units, nursing homes, homes for the aged, or in patients' own homes (with services provided by home care).

The most comprehensive and reliable data available from the ministry and CIHI related to patients in acute care hospital beds. These data revealed that a substantial percentage of beds in Thunder Bay's acute care hospitals—and in those of many other communities—were occupied by patients categorized as Alternative Level of Care (ALC), that is, patients who required a less intensive level of care than that characteristically provided by acute care hospitals. Many such patients had been in acute care beds for extended periods of time. This was so despite the fact that when the commission was established in 1996, there were approximately 9,000 closed (unstaffed) beds (the equivalent of 30 medium-size institutions) scattered throughout Ontario's approximately 220 public hospitals. The cumulative budgetary constraints noted above had led to the closing of these beds and corresponding staffing reductions, yet no hospital had closed.

In exploring options to make the organization of Ontario's hospitals more rational, it was essential to consider two issues: (1) how and where to consolidate acute care services to create more efficient and effective physical plants; to decrease administrative, managerial, and maintenance costs; and to provide an environment supporting higher quality of care; and (2) where patients categorized as ALC should best be cared for. It soon became apparent that the Alternative Level of Care category was a catchall designation; it merely identified patients who did not need the services of an acute care hospital. Without finer gradation, it was impossible to make definitive judgments on the proportion of patients in need of CCC, those needing the services of a LTC facility, and those who could be cared for safely and well in their own homes with support by home care. In the words of the commission, "In general, there are no evidence-based guidelines nor an evaluative framework available to determine the appropriateness (or adequacy) of chronic care bed supply or other long term care services" (HSRC 1998).

This deficiency was as applicable to CCC hospitals and units as to LTC facilities and home care. Data from the province-wide implementation of the MDS 2.0 in CCC facilities were not readily available in 1997, when the HSRC's Interim Planning Guidelines and Implementation Strategies document was released. As a result of the challenges encountered during the first year of implementation, a full year's worth of MDS 2.0 data was not available until late in 1997. By default, the primary CCC data source available to the commission was a 1995 study by the Metropolitan Toronto District Health Council in which about 1,200 patients in 13 Toronto chronic care hospitals and units were assessed using the MDS. Other smaller MDS data sets were available from Thunder Bay and Niagara, but the Toronto data were the most comprehensive.

The following year, the commission released a report (HSRC 1998) that distinguished between LTC beds and CCC beds based, at least in part, on RUG-III categories, with patients in the Clinically Complex, Extensive Services, and Special Care levels generally designated for CCC and those in the Behaviour Problems, Impaired Cognition, and Physical Functions Reduced categories designated for LTC. The commission did not direct that patients falling into the latter categories were to be relocated. Rather, these categorizations were to be used as guidelines in planning new admissions to CCC hospitals and units.

In a report to the Ministry of Health and Long Term Care, Hirdes and colleagues (2001) noted that 68.1 percent of LTC residents, as compared with 21.4 percent of CCC patients, fell into the lowest three RUG-III hierarchy levels, which was consistent with HSRC directives about the relative roles of these types of facilities. When new admissions to CCC were compared with existing CCC patients (Teare et al. 2000), the shift toward more short-stay, clinically complex patients in CCC became even more pronounced. Hence, although data were not readily available at the time the commission made its decisions about the proportion of patients requiring CCC, it later became possible to monitor the extent to which its directive that these patients be admitted to the hospital and all others be admitted to institutional or home-based care was translated into practice.

Recommendations for Assessment and Classification Systems

Ontario's health care system has become increasingly complex over time. Some of the growth in complexity is the result of fragmentation: in addition to nursing homes, homes for the aged, and chronic hospitals and units, organizations providing services to the frail elderly and people with disabilities include psychiatric hospitals and units, rehabilitation hospitals and units, acute care hospitals, retirement homes, assisted living complexes, and Community Care Access Centres (CCACs)7. One central difficulty posed by this multiplicity of providers is that these organizations have historically used different, incompatible assessment and classification systems. This has had the effect of restricting policymakers' ability to evaluate data—when data are available at all—from a system-wide perspective. These different providers' inability to communicate has an effect on patients, as well, reducing continuity of care as individuals move from one type of provider to another. And from the health care provider's perspective, the lack of common classification systems, at least between LTC and CCC, has perpetuated perceptions of inequities in funding. The level of funding remains more a function of the type of facility in which a specific patient resides than of what his or her actual needs (and the associated costs of service provision) are. For this reason, the commission recommended that

The Ministry of Health [should] work with CCACs across the province to develop an accountability framework that will support effective implementation of home care reinvestment in the province. The accountability framework should be supported by the early development of an information system that will introduce more consistent standards for data collection and analysis, and allow comparison of data across home care programs and with hospitals. (HSRC 1998, 18)

The Ministry of Health, in collaboration with chronic hospitals/units, physicians, nursing homes, homes for the aged, supportive housing services and CCACs, should expedite the development of a single point of access and adoption of a unified classification system for determining eligibility and placement into LTC facilities (including CCC beds). Because of the importance of the tool in ensuring appropriate placement in and access to LTC services, the HSRC recommends adoption of the Minimum Data Set (MDS) tool. (HSRC 1998, 45)

At the end of its mandate, the commission released a summary report (HSRC 2000) that continued to stress the need for common classification systems for LTC and CCC based on the MDS.

Although in some ways progress toward the goal of a comprehensive health information system (Sinclair and Hooper 1998) has been slower than hoped, Ontario's CCC has made some important positive steps that have been extended to other kinds of care settings. First, the JPPC Psychiatric Working Group and interRAI have spearheaded an effort to create an assessment system for inpatient psychiatry that is compatible with MDS 2.0 while at the same time measuring the particular needs of psychiatric patients (see Hirdes, Marhaba, et al. 2001 for details). Second, the CCACs and the Ministry of Health and Long Term Care have worked together to conduct a review of available assessment systems for home care. The ministry has recently identified MDS–Home Care (Morris et al. 1997, 1999) as the provincial standard for the assessment of patients receiving community-based care. Third, a large-scale project funded by Health Canada's Health Transition Fund allowed for a six-city trial that piloted MDS instruments for long-term care, home care, mental health, and acute care with the goal of refining the use of interRAI's series of instruments to serve as an integrated health information system (see Hirdes, Fries, et al. 1999). One product of this study was the aforementioned report (Hirdes, Fries, et al. 2001), which, for the first time, allowed for a direct comparison, based on MDS 2.0, of needs, access to services, resource intensity, and clinical characteristics of persons in LTC facilities and CCC hospitals and units in Ontario.

THE DAWN OF THE INFORMATION AGE

Five years after the Ministry of Health and Long Term Care made MDS 2.0 mandatory for CCC hospitals and units, the time has clearly passed when policymakers, managers, and service providers had to make decisions in the absence of evidence. CCC has made notable steps forward in two areas: case mix-based funding and quality reporting systems.

A Case Mix–Based Funding System

In April 1998, the Ontario Ministry of Health and Long Term care approved the JPPC Hospital Funding Committee's recommendation that RUG-III be used as the basis for funding CCC. The JPPC established the Complex Continuing Care Funding Working Group to develop operational recommendations for the funding methodology Ontario should use to translate RUG-III scores and other information into a payment system for CCC (JPPC 1999). Technical issues the working group addressed included conversion of RUG-III Case Mix Index weights from U.S. values to weights, based on Ontario wage rates, derivation of RUG-III-weighted patient days to account for the resource intensity of patients over the duration of their stay in a given year, establishment of data quality checks to ensure complete capture of the volume of patient activity, and specification of various alternatives for the funding formula (Teare 1999).

The first application of RUG-III-based funding for CCC occurred in the spring and summer of 2001, when RUG-III was used as part of a formula to allocate funding based on relative efficiency in the use of CCC beds. This formula is now integrated into an ongoing framework for CCC funding. In addition, RUG-III results were used to specify new allocations of beds for CCC in November 2000, and the ministry now requires CCC hospitals and units to report on the acuity and resource intensity of their beds using MDS data.

Despite considerable progress in the implementation of a RUG-III-based funding methodology, some operational complexities remain. In particular, the use of RUG-III for CCC units within acute hospitals and Case-Mix Groups™ (CMGs™)8 for acute care beds means that those facilities must contend with two distinct funding approaches: a per diem system for CCC beds and an episodic system for acute care beds. This is not to say that this issue would not have been of concern had another system been put in place for CCC. ARCS, which the JPPC deemed inadequate for CCC, is also a per diem system; its application would have resulted in the same problem in acute care hospitals. Similarly, CMGs cannot be applied to CCC; episodic models are ineffective when applied to populations of long-stay patients because of the substantial variability in length of stay and in resource utilization over the course of the episode (JPPC 1999).

Quality Initiatives Based on MDS 2.0

One of the primary applications of MDS data is in evaluating the processes and outcomes of care with a series of quality indicators developed by Zimmerman and colleagues (1995). The Chronic Care Role Study called on the Ministry of Health to "develop appropriate outcome measurements for quality of life, quality of care, and methods of client empowerment." Five years after MDS 2.0 became mandatory, CCC now has an active provincial quality network, annual reports on the status of CCC prepared by CIHI, and the capacity to use those data as part of a province-wide public report card initiative.

Ontario MDS Quality Network

As mentioned, the initial implementation of MDS 2.0 was a relatively challenging process complicated by a lack of experience at many levels, short time frames, and ambiguity regarding the roles of various stakeholders. The first meeting of the Ontario MDS Quality Network was held in Toronto as a result of the combined leadership of three hospitals: Sunnybrook Health Sciences Centre, Providence Centre, and SCO Elizabeth Bruyère. Until this point, the potential of MDS data to affect quality of care was not well known in Ontario, but after a relatively short period of time it became clear that their utility in quality improvement would become one of the main factors to increase the sustainability of MDS 2.0 in CCC.

One of the most visible quality issues had to do with the use of physical restraints. This issue gained the attention of members of the Ontario MDS Quality Network in part because the Providence Centre's 1997 Accreditation Survey had cited restraint use as an issue of concern. When data from this and other Toronto facilities were compared with international data available through interRAI, the rates of restraint use in Ontario were found to be substantially higher than in other jurisdictions (Hirdes, Mitchell, et al. 1999). These results stimulated considerable debate in the provincial quality network over what constitutes a restraint and what standards should apply to CCC patients in Ontario. Interest in the issue grew within the health care system and in the media when reports commissioned by CIHI demonstrated substantial regional variability in the use of restraints. In the summer of 2001, Frances Lankin, a former Minister of Health from a previous government, put forth a private member's bill on restraint use that passed in the provincial legislature. This bill's intention to improve practice patterns related to the use of restraints cannot be attributed solely to the effects of implementing MDS 2.0 in Ontario CCC, but the debate on restraint use was clearly enhanced by the availability of credible evidence. The effect of the bill will be easy to evaluate given that directly relevant data will be available to all interested parties.

CIHI Provincial Status Reports

Beginning in 1998, CIHI commissioned a series of annual reports on the quality of care in Ontario CCC hospitals and units. Nenadovic and colleagues (1999) provided the most extensive report up to that point on applications of MDS 2.0 for case mix, quality, outcome measurement, and care planning. The report included regional comparisons of Zimmerman and colleagues' (1995) quality indicators and demonstrated clear practice pattern differences among the five main OHA regions. It also brought the restraint issue to the public's attention for the first time by demonstrating that about one-third of CCC patients were being restrained. The following year (2000), the CIHI report included temporal analyses of trends in quality indicators and a special report on restraint use (Teare et al. 2000). This was also the first year that CIHI provided hospitals with personalized reports detailing their own performance on these quality measures compared with other CCC facilities. CIHI intends to continue releasing these types of in-depth reports coupled with hospital-specific results on an annual basis.

OHA Report Card

In 1999, the OHA publicly released its first balanced scorecard for acute care hospitals. The scorecard rates individual hospitals against four main sets of indicators: clinical and utilization, financial, patient satisfaction, and system integration and change. In 2001, the OHA released its prototype balanced scorecard for CCC, making heavy use of MDS 2.0 data for the financial and clinical indicator quadrants. The initial CCC report card dealt only with regional differences, but in subsequent years it will be expanded to include hospital-specific results. It should be noted, however, that the quality indicators in the report card relate only to long-stay patients because the current MDS 2.0 implementation does not include discharge assessments for persons discharged before their regularly scheduled 90-day reassessment. The report card's section on system integration and change includes an examination of the extent to which MDS applications for care planning, outcome measurement, quality improvement, and resource allocation have been incorporated into a facility's normal practice.

LESSONS LEARNED

Although it is difficult to summarize more than a decade's worth of changes in health policy related to CCC, it is possible to point to some major lessons learned from Ontario's experience. These lessons include the following:

1. Evidence is an essential ingredient for change. The debate following the Chronic Care Role Study showed that the absence of evidence hindered agreement on the place of CCC in the continuum of care. Although the HSRC used what evidence was available as the basis for its basic decisions about CCC in relation to LTC, the commission might have been much more innovative had it had the type of data now available for CCC.
2. It takes effort to get good data. It is not enough that all providers of health care are gathering data. Standardized, valid, reliable data are the necessary, fundamental building blocks to evidence-based decision making in health care. Acquiring such data requires systematic implementation of a common assessment methodology with appropriately trained staff, effective quality monitoring systems for data, and high levels of analytic expertise. For health care to truly come into the Information Age, health information management must have the capacity to embrace all the "players." Developing this capacity will not happen overnight, nor will it occur without substantial investment of resources and a collaborative effort on the part of governments and providers throughout the system.
3. Beware the black hole. The single most important threat to the successful implementation of MDS 2.0 in Ontario was the absence of feedback in the form of reports to providers of the data. Even after many years of data collection, most Ontario hospitals with CCC beds had received only limited hospital-specific results based on their MDS data. Staff and administrators' efforts to gather data went unrewarded by information until the establishment of the Ontario MDS Quality Network and the release of CIHI's first mini status report in 1998.
4. Ongoing training and education are essential at all levels in health care. Effective implementation of a health information system is a complex undertaking requiring the full participation of many stakeholders. It cannot be assumed that individuals with professional training as clinicians, managers, or policymakers know how to incorporate evidence-based decision making into their daily work. Particular emphasis should be given to incentives so that those contributing data will make full use of the information in applications relevant to them. Without such incentives there is considerable risk that information systems and the evidence they generate may suffer from reduced accuracy or incompleteness. Training that emphasizes benefits to the individual and organization and that enhances autonomy in decision making will be more effective than mechanistic training approaches in ensuring uptake.
5. Sometimes change must be mandated. Ontario made MDS 2.0 mandatory for chronic hospitals and units after a two-year consultation and study process. At about the same time, Saskatchewan opted for the "slow seduction" approach by implementing MDS 2.0 in one Health Region (Prince Albert) and waiting for others to follow suit. After several years, implementation remained partial, and Saskatchewan ultimately mandated the submission of RUG-III data in April 2001. Despite the period of considerable difficulty that Ontario endured as it initially struggled to implement MDS 2.0, the province now has five years' worth of useful data to evaluate policy changes and plan for future directions in CCC.
6. Sometimes it is necessary to fracture the status quo. Although Ontario's Health Services Restructuring Commission put fewer reforms in place than its members would have preferred, it remains true that this body, legislatively empowered to direct change, created an environment in which change could occur by fracturing the status quo. Most of the commission's directions and recommendations would not, perhaps could not, have been made had the usual, time-consuming processes of consensus-building and balancing diverse and divergent interest groups not been short-circuited by the HSRC's mandate and powers.
7. Weak data are the Achilles' heel of a policy debate. Some CCC hospitals slated to close by the HSRC remain open today because they were able to use political pressure to question the HSRC's per capita guidelines for CCC bed allocations.
8. Involving a broad range of stakeholders in decision making is an important condition for successful change. The JPPC's CCRWG included active participation by hospital administrators, clinicians, policymakers, and researchers. This gave the committee the ability to address key questions on the information needs of CCC from a variety of perspectives. It also provided a nucleus of opinion leaders who could represent and speak to different influential constituencies (e.g., geriatricians) within CCC.

PROSPECTS FOR THE FUTURE

In an August 1993 letter to the minister of health, the president of the Ontario Hospital Association wrote, "We cannot move the chronic and long-term care system forward without a solid base of information on which all interested and involved parties can make informed planning and management decisions." Without doubt, CCC in Ontario has made substantial steps forward since that time. Nonetheless, there remain many key challenges that must be addressed if the new system is to be sustained and enhanced.

No fundamental change in LTC health information systems has occured since the HSRC first made its observations about the lack of evidence for bed allocations in 1997. LTC facilities continue to use a system (ARCS) that is incompatible with the MDS 2.0 system implemented in CCC. An important constraint for the industry and the government has been the cost of implementing an adequate computerized infrastructure to permit the effective use of instruments like the MDS 2.0 in LTC facilities. Meanwhile, LTC faces increasing numbers of residents with higher levels of clinical complexity, and the insensitivity of ARCS to this complexity will make it difficult to allocate resources in a way that is truly responsive to the needs of individuals in nursing homes and homes for the aged. Until these two sectors share a common assessment and classification system with CCC, it will be impossible to make direct comparisons of the costs, quality, and relative outcomes of these different forms of long-term care. In 2000, an advisory committee to the Ministry of Health and Long Term Care conducted a review of assessment and classification systems for LTC. The committee concluded that LTC had no practical alternatives to MDS 2.0, and it was supportive of a pilot project to test the feasibility of using MDS 2.0 in LTC (Ministry of Health and Long Term Care 2000). Funds for such a project were not available at the time, however, so the effort had to be temporarily delayed.

The quality of data appears to have been reasonably good for the first five years of MDS 2.0 implementation in CCC. The primary concern noted by the JPPC in preparing for RUG-III-based funding models has been in regard to the completeness of MDS data sent to CIHI rather than the accuracy of the clinical information (e.g., patient days in the MDS records did not account for all patient days in the facility census). As CCC begins to be implement RUG-III-based funding, the gaming of assessment systems for financial purposes may emerge as a threat to data quality. Based on U.S. experience, one may expect unscrupulous software vendors or consultants to develop and market products aimed at maximizing mean case-mix index scores rather than maximizing the accuracy of observations. Controlling this will require a multifaceted approach, including training administrators in the pitfalls (and ethics) of gaming, establishing audit systems to monitor data quality, and using quality benchmarks to track quality of care. The JPPC has established a CCC Technical Working Group to review the potential for gaming and to establish mechanisms to control the problem.

Clinicians, managers, and policymakers have underutilized the wealth of data now available to CCC. A number of factors may account for this. Ontario clinicians have had relatively little training in the use of MDS data for care planning or for measuring outcomes. This has been less of a concern in provinces (e.g., Manitoba, Nova Scotia, British Columbia) where current pilot implementations have placed much greater emphasis on clinical applications than on funding systems. At the ministry level, two divisions (Health Care Programs and Integrated Policy and Planning) are working to improve internal processes to organize and analyse MDS 2.0 data.

A lack of expertise and the relatively high level of technical complexity associated with quality and funding applications of MDS 2.0 data may be a barrier for some managers and policymakers. This is not necessarily unique to the MDS assessment system, since evidence-based decision making is inherently complex. As Sackett and colleagues (1997) note, an evidence-based health system will demand of its participants a commitment to lifelong learning together with the capacity to integrate new knowledge with their professional experience. The challenge is at least twofold: educating professionals in how to incorporate evidence-based decision making into their work lives and training them in the substantive knowledge required to make full use of all MDS applications. This can only be achieved by making clear how all of the effort of implementing such a system can benefit health professionals and the populations they serve.

GLOSSARY

ALC
Alternate Level of Care

ARCS
Alberta Resident Classification System

CCAC
Community Care Access Centre

CCC
Complex Continuing Care

CCRWG
Chronic Care and Rehabilitation Working Group

CIHI
Canadian Institute for Health Information

CMGs
Case Mix Groups

EBDM
Evidence-based Decision Making

FIM
Functional Independence Measure

FRGs
Function Related Groups

HSRC
Health Services Restructuring Commission

JPPC
Joint Policy and Planning Committee

LTC
Long-Term Care

MDS
Minimum Data Set

OHA
Ontario Hospital Association

RUG-III
Resource Utilization Groups

NOTES

¹Complex Continuing Care hospitals have patient populations that are similar to those of skilled nursing facilities and chronic care hospitals in the United States. These patients tend to be more severely functionally impaired than residents in long-term care facilities in Ontario, and their problems are more clinically complex.

²In the past, LTC facilities were funded separately because of their different relationships with two provincial ministries (initially the Ministry of Health for nursing homes and Ministry of Community and Social Services for homes for the aged), even though they were accommodating residents with similar care requirements. Reform of the LTC system brought all facilities together under one administrative, funding, and accountability structure under the Ministry of Health and Long Term Care.

³The term Complex Continuing Care hospitals was introduced to replace chronic care hospitals by the Health Services Restructuring Commission in its July 1997 report, Rebuilding Ontario's Health System: Interim Planning Guidelines and Implementation Strategies.

⁴Throughout this case study, the term long-term care facility will be used generically to refer to both nursing homes and homes for the aged in Ontario.

⁵The Savings and Restructuring Act, 1966 (S.O.1996, Chapter 1, Schedule F ["Bill 26"]), which repealed section 8 of the Ministry of Health Act and substituted a new section 8.

⁶The regulations mandated by the Ministry of Health Act (O.Reg.88/96) and the Public Hospitals Act (O.Reg.87/96) came into force on April 1, 1996. The powers given to the commission in these two statutes, together with the provisions in section 8 of the Ministry of Health Act and sections 6 and 9(10) of the Public Hospitals Act constituted the powers and authority of the commission.

⁷CCACs are single-point-entry agencies that act as contracting agents with home care providers for the populations they serve. They make all decisions about the level and number of hours of home care services the client will receive, as well as all decisions about admission to LTC and, if requested by the facility, to some CCC hospitals and units.

⁸CMGs™ are the Canadian equivalents of the Diagnosis Related Groups used in the United States.

REFERENCES

Advisory Council on Health Infostructure. 1999. Canada Health Infoway—Paths to Better Health. Ottawa: Minister of Public Works and Government Services.

Alberta Health. 1988. Alberta Resident Classification System for Long Term Care Facilities: Instructions for Completing the Resident Classification Form. Edmonton.

Canadian Institute for Health Information. 2001. Health Care in Canada. Ottawa.

Chronic Care Implementation Task Force. 1995. Report of the Chronic Care Implementation Task Force. Hamilton, Ont.: Educational Centre for Aging and Health.

Fries, B.E., D.P. Schneider, W.J. Foley, M. Gavazzi, R. Burke, and E. Cornelius. 1994. Refining a Case-Mix Measure for Nursing Homes: Resource Utilization Groups (RUG-III). Medical Care 32(7):668–85.

Geddes, J., and S. Carney. 2001. Recent Advances in Evidence-Based Psychiatry. Canadian Journal of Psychiatry 46(5):403–6.

Hay Health Care Consulting Group. 1993. Chronic Care Role Study, vol.1, Executive Summary of the Project Report, Project Report of the Chronic Care Role Study Steering Committee. Ottawa: Ministry of Health and Long-term Care.

Health Services Restructuring Commission (HSRC). 1998. Change and Transition—Planning Guidelines and Implementation Strategies for Home Care, Long Term Care, Mental Health, Rehabilitation, and Sub-Acute Care. Toronto.

Health Services Restructuring Commission (HSRC). 2000. Looking Back, Looking Forward—The Ontario Health Services Restructuring Commission (1996–2000) Legacy Report. Toronto.

Hirdes, J.P. 1997. Development of a Cross-Walk from the Minimum Data Set 2.0 to the Alberta Classification System. Health Care Management Forum 10(1):27–34.

Hirdes, J.P., C.A. Botz, J. Kozak, and V. Lepp. 1996. Identifying an Appropriate Case-Mix Measure for Chronic Care: Evidence from an Ontario Pilot Study. Healthcare Management Forum 9:40–6.

Hirdes, J.P., and G.I. Carpenter. 1997. Health Outcomes among the Frail Elderly in Communities and Institutions: Use of The Minimum Data Set (MDS) to Create Effective Linkages Between Research and Policy. Canadian Journal on Aging/Canadian Public Policy (special joint issue; suppl.):53–69.

Hirdes, J.P., B.E. Fries, J.N. Morris, K. Steel, V. Mor, D. Frijters, P. Jonsson, S. LaBine, C. Schalm, M.J. Stones, G. Teare, T. Smith, M. Marhaba, and E. Pérez. 1999. Integrated Health Information Systems Based on the RAI/MDS Series of Assessment Instruments. Healthcare Management Forum 12(4):30–40.

Hirdes, J.P., L. Mitchell, G. Ljunggren, and M. Schroll. 1999. International and Regional Variations in Restraint Use: Implications for Selecting Benchmarks. Canadian Journal on Quality in Health Care 15(2):19–23.

Hirdes, J.P., B.E. Fries, D. Frijters, and G.E. Teare. 2001. Canadian, US and International Comparisons of Resource Intensity and Responses to the Needs of Residents in Long Term Care Facilities. Toronto: Technical report to the Ontario Long Term Care Association, Ontario Ministry of Health and Ontario Association of Not-for-Profit Homes and Services for Seniors.

Hirdes, J.P., M. Marhaba, T.F. Smith, L. Clyburn, L. Mitchell, R.A. Lemick, N. Curtin Telegdi, E. Pérez, P. Prendergast, T. Rabinowitz, and K. Yamauchi. 2001. Development of the Resident Assessment Instrument—Mental Health (RAI-MH). Hospital Quarterly 4(2):44–51.

Hunter, David J. 2001. Rationing Healthcare: The Appeal of Juggling through Elegantly. HealthcarePapers 20(2):31–7.

Joint Policy and Planning Committee (JPPC). 1999. Cost per Case-Mix Weighted Activity for Complex Continuing Care in Ontario—An Interim Summary Report from the JPPC Complex Continuing Care Funding Working Group (JPPC Reference Document RD #8-11). Toronto.

McKibbon, A., A. Eady, and S. Marks. 1999. PDQ Evidence-based Principles and Practice. Hamilton, Ont.: B.C. Decker.

Ministry of Health and Long Term Care. 2000. Report of the Long Term Care Facilities Classification Review. Toronto.

Morris, J.N., B.E. Fries, K. Steel, N. Ikegami, R. Bernabei, G.I. Carpenter, R. Gilgen, J.P. Hirdes, and E. Topinkova. 1997. Comprehensive Clinical Assessment in Community Setting: Applicability of the MDS-HC. Journal of the American Geriatrics Society 45:1017–24.

Morris, J.N., B.E. Fries, R. Bernabei, K. Steel, N. Ikegami, G.I. Carpenter, R. Gilgen, J.N. DuPasquier, D. Frijters, J.C. Henrard, and J.P. Hirdes. 1999. RAI-Home Care (RAI-HC©) Assessment Manual for Version 2.0. Boston: interRAI.

Muir Gray, J.A. Evidence-based Healthcare—How to Make Health Policy and Management Decisions. New York: Churchill Livingstone, 1997.

Nenadovic, M., E. Gilbart, K. Hallman, G. Teare, and J.P. Hirdes. 1999. Provincial Mini-Status Report—The Quality of Caring—Ontario April 1997–1998. Toronto: Canadian Institute of Health Information.

Ontario Hospital Association (OHA). 1993. Response to the Chronic Care Role Study and Long-term Care Redirection. Toronto.

Organisation for Economic Co-operation and Development. 2002. OECD Health Data 2001—A Comparative Analysis of 30 Countries. Paris.

Sackett, D.L., W.S. Richardson, W. Rosenberg, and R.B. Haynes. 1997. Evidence-based Medicine—How to Practice and Teach EBM. New York: Churchill Livingstone.

Schoen, C., E. Strumpf, K. Davis, R. Osborn, K. Donelan, and R.J. Blendon. 2000. The Elderly's Experiences with Health Care in Five Nations: Findings from the Commonwealth Fund 1999 International Health Policy Survey. New York: Commonwealth Fund.

Sinclair, D.G., and L. Hooper. 1998. Knowledge is Care: Whither Health Information Systems? Hospital Quarterly 1(4):73–5.

Smith, G. 1989. Missionary Zeal and the Scholarly Stance: Policy and Commitment in Research on Old Age. Ageing and Society 9:105–21.

Stineman M.G., J.J. Escarce, J.E. Goin, B.B. Hamilton, C.V. Granger, and S.V. Williams. 1994. A Case Mix Classification System for Medical Rehabilitation. Medical Care 32(4):366–79.

Teare, G.F. 1999. Cost per Case-Mix Weighted Activity for Complex Continuing Care in Ontario—Technical Paper from the JPPC Complex Continuing Care Funding Working Group (JPPC Reference Document RD #8-12). Toronto: Joint Policy and Planning Committee.

Teare, G.F., J.P. Hirdes, M. Ziraldo, W. Proctor, and M. Nenadovic. 2000. Provincial Status Report—The Quality of Caring—Ontario April 1998–1999. Toronto: Canadian Institute of Health Information.

Zimmerman, D., S.L. Karon, G. Arling, B.R. Clark, T. Collins, R. Ross, et al. 1995. Development and Testing of Nursing Home Quality Indicators. Health Care Financing Review 16(4):107–27.