Transcript annotation in FANTOM3: Mouse gene catalog based on physical cDNAs
2006. April 2006

N Maeda
T T Kasukawa
R Oyama
J Gough
PäM Frith

Broadband Communication Service Business Unit, Network Service Solution Business Group
Genome Science Laboratory, Discovery Research Institute
RIKEN Wako Institute
Wako

Australian Research Council Center in Bioinformatics
Institute for Molecular Bioscience
NTT Software Corporation
YokohamaJapan

Center for Computational Science
the Computational Biology Unit
University of Queensland
St LuciaBergenQueensland

Programme for Genomics and Bioinformatics
Department of Cell and Molecular Biology, Karolinska Institutet
Department of Biomedical Sciences
University of Bergen
Bergen, Norway, StockholmSweden

Institute for Molecular Bioscience
Australian Research Council Special Research Centre for Functional and Applied Genomics
Creighton University School of Medicine
OmahaNebraska

Genetics and Cell Biology Section
University of Queensland
BrisbaneQueenslandAustralia

Department of Bioinformatic Engineering
Graduate School of Information Science and Technology
National Cancer Center Research Institute
TokyoJapan. J

Dana-Farber Cancer Institute and Harvard School of Public Health
Osaka University
Osaka, BostonMassachusettsJapan. C. A., United States

Transcript annotation in FANTOM3: Mouse gene catalog based on physical cDNAs

PLoS Genet
242006. April 2006Technical Report are in the Genome Exploration Research Group (Genome Network Project Core Group), RIKEN Genomic Sciences Center, RIKEN Yokohama Institute, Yokohama, Japan. T. Kasukawa is in the Fletcher are at the Genomics Institute of the Novartis Research Foundation, San Diego, California, United States of America. K. W. Beisel is in the United States of America. C. J. Bult, M. Furuno, D. Hill, and Y. Zhu are in the Mouse Genome Informatics Consortium, The Jackson Laboratory, Bar Harbor, Maine, United States of America. A. R. R. Forrest, T. Ravasi, and D. A. Hume are at the. M. Katoh is in the Quackenbush is at the Institute for Genomic Research, Rockville, Maryland, United States of America. B. Z. Ring is at Applied Genomics, Sunnyvale, California, United States of America. K. Sugiura is at The Jackson Laboratory, Bar Harbor, Maine, United States of America. Y. Takenaka is in the Wells is in the School of Biomolecular and Biomedical Science, Eskitis Institute for Cell and Molecular Therapies, Griffith University, Nathan, Queensland, Australia. Y. Hayashizaki is at Yokohama City University, Yokohama, Japan, and in the Graduate School of Comprehensive Human Science, University of Tsukuba, Tsukuba, Japan. * To whom correspondence should be addressed. Current address: Functional Genomics Subunit, Center for Developmental Biology, RIKEN, Kobe, Japan ¤b Current address:Abbreviations: CDS, coding sequenceGO, Gene OntologyMGI, Mouse Genome InformaticsncRNA, noncoding RNA
T he international FANTOM consortium aims to produce a comprehensive picture of the mammalian transcriptome, based upon an extensive cDNA collection and functional annotation of full-length enriched cDNAs. The previous dataset, FANTOM2, comprised 60,770 full-length enriched cDNAs. Functional annotation revealed that this cDNA dataset contained only about half of the estimated number of mouse protein-coding genes, indicating that a number of cDNAs still remained to be collected and identified. To pursue the complete gene catalog that covers all predicted mouse genes, cloning and sequencing of full-length enriched cDNAs has been continued since FANTOM2. In FANTOM3, 42,031 newly isolated cDNAs were subjected to functional annotation, and the annotation of 4,347 FANTOM2 cDNAs was updated. To accomplish accurate functional annotation, we improved our automated annotation pipeline by introducing new coding sequence prediction programs and developed a Web-based annotation interface for simplifying the annotation procedures to reduce manual annotation errors. Automated coding sequence and function prediction was followed with manual curation and review by expert curators. A total of 102,801 full-length enriched mouse cDNAs were annotated. Out of 102,801 transcripts, 56,722 were functionally annotated as protein coding (including partial or truncated transcripts), providing to our knowledge the greatest current coverage of the mouse proteome by full-length cDNAs. The total number of distinct non-protein-coding transcripts increased to 34,030. The FANTOM3 annotation system, consisting of automated computational prediction, manual curation, and final expert curation, facilitated the comprehensive characterization of the mouse transcriptome, and could be applied to the transcriptomes of other species.

Introduction

The RIKEN Mouse Gene Encyclopedia project was launched with the aim of cloning and sequencing full-length mouse cDNAs. An international annotation consortium (FANTOM) was organized to annotate the collected mouse cDNAs. In FANTOM1, the consortium annotated 21,076 cDNAs with the development of a Web-based annotation interface [1]. In FANTOM2, this interface was extended to be an all online annotation system from remote sites via the Internet, through the Mouse Annotation Teleconference for RIKEN cDNA Sequences (MATRICS). The increased efficiency and throughput was essential in the functional annotation of 60,770 mouse cDNAs [2]. FANTOM1 and FANTOM2 considerably extended our knowledge of the mouse transcriptome, but compared with the number of predicted protein-coding genes from mouse genome sequencing, the cDNA resource covered only half of all predicted genes. Therefore, cDNA collection from a number of novel cellular and tissue sources was continued. In this process, many novel cDNAs derived from distinct genomic loci were fully sequenced. In FANTOM3, these newly sequenced cDNAs were mapped to the mouse genome and subjected to functional annotation. Given the substantial increase in cDNA sequence information in mouse and other mammalian species since FANTOM2, the new annotation process provided the opportunity to update and improve the previous functional annotation of RIKEN cDNAs from FANTOM1 and FANTOM2.

Here we report the development of the new annotation interface and decision pipeline, and the modification of our annotation strategy to accelerate manual annotation. And we also provide functional annotation of 102,801 mouse full-length enriched cDNAs, to our knowledge the largest such dataset.

The result of this functional annotation was shared among FANTOM3 consortium members for further analyses such as protein coding analysis and noncoding RNA (ncRNA) analysis [3].

Results/Discussion

Issues Associated with Optimal Online Annotation

The Web-based online annotation system from FANTOM2 was likewise implemented for FANTOM3. This system allowed all curators to annotate transcripts from remote sites around the world through the Internet and resulted in significant acceleration of the manual annotation process. Nevertheless, time remained an issue. Even 10 min spent on manual annotation of each transcript would mean that the total task would consume 15,000 h, and our aim was to complete the task within a matter of weeks. In FANTOM2, curators could enter comments when they encountered problematic cDNAs or ones that were difficult to annotate. However, it was a heavy burden for expert curators, who reviewed and corrected annotations, to read all written comments and correct annotations one by one. For these and other reasons, we introduced a precomputational pipeline in which the annotator could accept the automated decision by ticking a series of boxes. Only where there was some ambiguity, or a better alternative name, was the annotator required to assess additional data and enter alternative decisions. In general, this process reduced the annotation time for unequivocal cases down to 10-20 s.

Modification of Annotation Rules and Pipeline for FANTOM3

We updated the original annotation rules that were determined during the FANTOM2 meeting [2,4,5] in order to improve the quality of curation, and introduced these into the automated annotation pipeline. The annotation items in the FANTOM3 new rule set are summarized in Table 1.

Firstly, coding sequence (CDS) annotation items were expanded in FANTOM3. In FANTOM2, curators annotated the following four items: CDS status (CDS region for coding, UTR region only, or no CDS), completeness of 59 and 39 ends of CDS, maturity of transcripts, and presence of in-frame insertion/deletion errors and stop codons. In FANTOM3, three additional items were introduced: exact positions of inframe insertion/deletion errors, and flags for selenoproteins and mitochondrial transcripts with unique codon usage. This information was used for computational translation to make a complete dataset of protein-coding transcripts, and it allowed us to avoid unwanted frameshifts and stop codons in the middle of a CDS region. Secondly, the set of CDS prediction algorithms, the outcomes of which are displayed at the top level to the annotator, was changed based upon our previous experience. ProCrest (unpublished) and NCBI CDS Predictor (unpublished), which were used in FANTOM2, were phased out because they cannot identify exact positions of in-frame insertion/deletion errors, although they are able to predict whether these errors exist or not within CDS regions [4].

Instead, three other algorithms were introduced: CRITICA [6], mTRANS (M. Furuno, unpublished data), and CombinerCDS [7]. In FANTOM3, curators could make their judgment on a CDS region by comparing all of the CDS predictions from DECODER [8], rsCDS [5], longest ORF, truncated longest ORF, CRITICA, mTRANS, CombinerCDS, and FANTOM2 curation (for FANTOM2 cDNAs).

Thirdly, we improved our annotation pipelines for DOI: 10.1371/journal.pgen.0020062.g001 Figure 1. Annotation Pipelines for Transcript Description and for GO Terms (A) Pipeline for transcript description. Query sequences falling into categories (black boxes) 1-3 were assigned the description of the matched target sequence DNA entry in MGI symbols, and synonyms were also transferred to our annotation database. Queries falling into categories 4-10 were assigned a transcript description corresponding to the matched protein name. For query sequences falling into category 5 or 6, the keyword ''homolog'' was appended to the matching protein name. Sequences assigned to category 7 or 8 were denoted with the prefix ''similar to'' attached to the target sequence name. The prefix ''weakly similar'' was used to identify sequences assigned to category 9 or 10. For all sequences in categories 5-10, the name of the organism corresponding to the matched protein was appended to the assigned transcript description. assigning transcript descriptions (renamed from ''gene names'' in FANTOM2), symbols (renamed from ''gene symbols''), and synonyms to transcripts ( Figure 1A) and for assigning Gene Ontology (GO) terms ( Figure 1B). We excluded some protein motif detections and transcript clustering procedures from our previous pipelines, making it possible to simplify the automation. Instead, one new step was introduced to identify known ncRNAs. When a transcript has no coding regions and its sequence significantly matches against a known ncRNA set retrieved from RNAdb [9], the name of its known ncRNA is transferred as its transcript description. Otherwise, the transcript description of the ncRNA becomes ''unclassifiable.'' As for the GO assignment pipeline, the most significant match was searched in the order of directly assigned Mouse Genome Informatics (MGI) markers, DNA matches, and protein matches, and then its GO assignments were transferred to the query transcript. If InterPro motifs were detected in transcripts, GO assignments on the motifs were also transferred and combined with ones for a significant match. Fourthly, new annotation items to identify problematic clones were added. In the FANTOM3 annotation system, two buttons for these problematic clones, chimeric clones and reverse clones, were introduced to simplify the annotation process. If a cDNA is deemed to be derived from two or more mRNAs or to be a contaminant from Escherichia coli, it is curated as ''chimeric clone.'' If a cDNA has evidence that implies cloning in the reverse direction, for example, having CT-AC splicing patterns rather than GT-AG ones, it is curated as ''reverse clone.'' These problematic entries are then automatically excluded from further curation and analyses.

Modification of the Curation Interface for FANTOM3

To help curators annotate accurately, the curation interface was improved from that of FANTOM2. Information such as MGI assignment, cDNA status prediction, sequence quality, expressed sequence tag mapping, genome mapping, splicing information, predicted transmembrane regions, and protein motifs was provided on the curation screen. Some information was provided in a simple graphical display to expedite rapid decisions. Moreover, additional information such as raw alignments and hyperlinks to public databases could be accessed by clicking corresponding bars in the cDNA summary image section.

In the new FANTOM3 interface, annotators were provided with an initial computational annotation that the curators were then required to accept or reject by clicking buttons. To simplify the annotation process when the computational annotation was rejected, several major reject reasons and alternative CDS predictions, CDS statuses, transcript descriptions, and GO terms were provided as a list with checkboxes, and the curators were prompted to select an appropriate one. Curators were also encouraged to add notes on each transcript, based upon their background knowledge.

The Annotation Process

The computational annotation in FANTOM3 was carried out prior to manual annotation, as in FANTOM2. The FANTOM3 annotation pipelines for assigning transcript descriptions and for GO assignments are summarized in Figure 1A and 1B, respectively. Subsequent manual annotation was carried out sequentially. The potential protein-coding transcripts, whose predicted CDS regions were longer than 100 amino acids in length based upon the computational prediction, were annotated first because these require the least input and are of greatest interest to the scientific community. Out of 30,476 FANTOM3 potential coding transcripts, 20,027 (65.7%) initial computational annotations were manually accepted, and 6,997 (23.0%) transcripts were easily annotated by choosing the alternatives provided. Thus, more than 85% of coding transcripts were easily annotated by just clicking buttons on our annotation interface, indicating that the button-based interface indeed contributed to accelerated manual annotation for potential protein-coding transcripts.

After manual curation on potential protein-coding transcripts, we next considered annotating potential nonprotein-coding transcripts. To reduce human annotation errors, potential non-protein-coding transcripts were classified into several subcategories and were released stepwise depending on their coding potential. The transcripts that completely or partially matched known genes at the DNA level were open to curators first, followed by the transcripts that showed similarity to known genes at the amino acid level. Finally, the transcripts that were just covered with expressed sequence tags were subjected to manual curation. Out of 11,555 potential non-protein-coding transcripts, 7,343

Review of Functional Annotation

To improve the quality of the functional annotation dataset, a review process was carried out following the manual curation. Expert curators were selected from all registered curators based on their performance, and they reviewed the rejected entries. In FANTOM3, computational filtration was intensively performed to lighten the burden for expert curators. Several criteria are discussed below.

In eukaryotes, nonsense-mediated mRNA decay is known as a mRNA surveillance mechanism (reviewed in [10,11]). It has been recently reported that some mRNAs that have premature termination codons are not degraded by the nonsense-mediated mRNA decay mechanism, and that the ''50 nucleotide rule'' of nonsense-mediated mRNA decay cannot always be applied to the evaluation of annotation results. However, this ''50 nucleotide rule'' was useful for a rough screening to extract the transcripts that might be incorrectly annotated. In the FANTOM3 review process, cDNA entries for which the 39 end of the curated CDS was 50 nucleotides or more upstream of the 39-most exon/intron junction were computationally extracted. These cDNA entries were intensively reviewed, and apparent misannotations were corrected by expert curators.

Flanking adenine-rich sequence at the 39 end of a transcript suggests the possibility that the cDNA could be produced by internal priming of oligo-dT primer. Therefore, we extracted the transcripts that had more than ten adenosines in the 20 flanking nucleotides by using mouse genome sequence, and these transcripts were manually reviewed by expert curators. If transcripts seemed to be produced by internal priming of coding transcripts, they were curated as coding/immature.

In FANTOM3, we also developed a genomic element browser by customizing the generic genome browser [12,13] to review annotations based on their genomic loci. In this browser, all FANTOM transcripts are aligned on their genomic loci, accompanied by information on annotation (e.g., curated coding region, gene name, coding/noncoding, clone ID, and strand orientation). This browser allowed the expert curators to compare all transcripts that were located at the same loci and to correct annotation when necessary.

Conclusions

In FANTOM3, 42,031 transcripts were newly annotated and the functional annotation of 4,347 FANTOM2 transcripts was updated with the improved annotation system. Combining the results of FANTOM2 and FANTOM3, 102,801 cDNAs were functionally annotated by the international effort. Out of these, 47,761 and 8,961 transcripts were annotated as complete coding and truncated coding, respectively, and 34,030 transcripts were annotated as non-protein-coding (Table 2). Our FANTOM3 annotation system largely contributed to the prompt and precise annotation that was accomplished, and this system could be a model for other mammalian transcriptome projects.

The curated annotation data are available at http:// fantom3.gsc.riken.jp/db and ftp://fantom3.gsc.riken.jp/ fantomdb/3.0.

Materials and Methods

Sequence set and annotation. We annotated 102,801 sequences derived from RIKEN mouse full-length enriched cDNA libraries [3]. The set consists of 60,770 FANTOM2 [2,14] and 42,031 novel isolated sequences. Of the 60,770 FANTOM2 sequences, 932 were updated after the FANTOM2 meeting. For the FANTOM2 sequences that were not curated during the FANTOM3 period, the gene names assigned in FANTOM2 were transferred as their transcript descriptions.

Mapping of transcript sequences to the mouse genome. Transcript sequences were mapped to the mouse genome (assembly mm5) in several stages. In the first stage, the sequences were aligned to the genome using BLAT version 30 [15] with options -ooc ¼ 11.ooc -fine -q ¼ rna. Low-quality alignments were then removed using pslReps (distributed with BLAT; http://www.soe.ucsc.edu/;kent/exe/linux/ blatSuite.zip) with options -minAli ¼ 0.96 -nearTop ¼ 0.005. Next, the alignments were post-processed by an algorithm designed to extend transcript-to-genome alignments by using information about exon positions from neighboring alignments (P. G. Engströ m and B. Lenhard, unpublished data). Subsequently, the highest-scoring alignment or alignments, according to the following formula, were retained for each transcript: round (20,000 3 identity þ 100 3 coverage þ 2 3 number of introns), where identity ¼ number of matches/(number of matches þ number of mismatches þ number of non-intron gaps), coverage ¼ number of matches/transcript sequence size, and introns are gaps of at least 20 bp in the transcript sequence only. Ties were broken in favor of assembled chromosomes over unassembled genomic sequence. If there were still two highestscoring alignments for a transcript, both were displayed in the annotation interface. Finally, adjacent alignment blocks were connected if they appeared to belong to the same exon. The criteria for deciding that blocks belonged to the same exon were adopted from the Sim4 program [16]: (1) gap lengths of less than 50 bp and (2) differences in gap lengths between genome and cDNA sequences of less than 9 bp. In merging blocks, gapped regions were aligned with the stretcher program in the EMBOSS package [17].

Computational analysis for data preparation. Assembled fulllength cDNA sequences were first masked using RepeatMasker (http:// repeatmasker.org) to exclude regions containing known repetitive sequences. FANTOM3 query sequences were searched against mouse non-expressed-sequence-tag mRNA sequences in the MGI database [18] (http://www.informatics.jax.org), against the mouse sequences in dbEST [19] (http://www.ncbi.nih.gov/dbEST), and against known ncRNA sequences in RNAdb (http://research.imb.uq.edu.au/rnadb) [9]. DNA searches were performed using BLASTN [20] with the -F option, which turns off filtering of the query sequences, for the MGI and dbEST database searches and with the default option for RNAdb searches. Protein databases were searched using the FASTY program [21] in the FASTA3 package. The FANTOM3 sequences were searched against the UniProt database [22]. Open reading frames in the cDNA sequences were predicted using DECODER, and those with predicted CDS regions were subjected to an InterPro motif prediction analysis. InterProScan was used to search the InterPro database [23] (http:// www.ebi.ac.uk/interpro).

Coding potential classification. We flagged sequences ''with coding potential'' when at least one of the following conditions was satisfied: a protein was matched with greater than 50% identity and greater than 50% length of the target protein, a named InterPro domain was found in the predicted protein sequence, a transmembrane region was detected with the TMHMM program [24], a coiled coil region was predicted with the NCOILS program, a signal peptide was identified with the SignalP program [25], or a CDS longer than 100 amino acids was predicted.

Annotation pipeline programs and curation interface. The annotation pipeline programs were implemented as a Perl script that evaluated the evidence at each stage in the process and made a decision at each stage, writing the appropriate annotation to the database using the appropriate controlled vocabulary terms.

The cDNA annotation (curation) interface was implemented as a Web-based application using mod_perl and the gd graphics library on a Linux system running an Apache 2.0 server. All curated annotations and annotation histories were stored in a custom database implemented in a Sybase (http://www.sybase.com) relational database management system. Other data such as similarity search alignments and clone sequences were stored in indexed flat files. &

Table 1 .
1Annotation Items in FANTOM3Annotation Category
Annotation Item
Description

Chimeric clone status
Chimeric clone status
Chimeric clone or not
Chimeric clone note
Note about the chimeric clone status
Reverse clone status
Reverse clone status
Reverse clone or not
Reverse clone note
Note about the reverse clone status
CDS
CDS
Location of CDS on transcript, existence of
frameshift errors, and unexpected stop codons
CDS status
CDS status (immature, polycistronic, mitochondrial, or selenoprotein)
CDS note
Note about the CDS
Transcript description
Transcript description
Brief explanation about the transcript
Transcript symbol
Symbol of the transcript
Transcript synonyms
Synonyms of the transcript
Transcript description note
Note about the transcript description
GO assignment
GO assignment
Assigned GO terms
GO assignment note
Note about the GO assignment

DOI: 10.1371/journal.pgen.0020062.t001

If a query was assigned to category 14, its transcript description was ''hypothetical [InterPro domain name] containing protein.'' Query sequences assigned to category 17 and 19 were annotated as ''hypothetical protein'' and ''unclassifiable,'' respectively. Query sequences grouped into category N1 or N2 were assigned the description of the matched target ncRNA entry. For query sequences falling into category N2, the keyword ''homolog of'' was appended to the matching ncRNA name. (B) Pipeline for GO terms. DB, database.

Table 2 .
2Summary of Annotation in FANTOM3Curation Category
Subcategory FANTOM3
All

FANTOM3
New

Total
102,801
42,031
Artifact
Potential contaminant
26
24
Chimera of several
mRNAs

178
120

Artifactually reversed
cDNA clone

85
62

CDS
Complete
47,761
23,572
With frameshift
errors

7,672
3,924

Mitochondrial
18
18
Polycistronic
8
8
Selenoprotein
75
40
59-truncate
5,191
2,248
With frameshift
errors

640
299

Mitochondrial
10
10
Polycistronic
2
2
Selenoprotein
1
0
39-truncate
3,464
1,682
With frameshift
errors

553
299

Mitochondrial
13
13
Polycistronic
3
3
59/39-truncate
306
173
With frameshift
errors

43
27

Mitochondrial
8
6
59 UTR only
124
68
39 UTR only
1,724
785
Immature
9,912
4,735
No CDS
34,030
8,562

DOI: 10.1371/journal.pgen.0020062.t002

PLoS Genetics | www.plosgenetics.org April 2006 | Volume 2 | Issue 4 | e62
AcknowledgmentsWe thank all annotators for their contribution to functional annotation, especially Yumi Onodera for her exceptional contribution. Funding. This study was supported by the research grant for the RIKEN Genome Exploration Research Project from the Ministry of Education, Culture, Sports, Science and Technology of Japan to Y. Hayashizaki, and a research grant for the Genome Network Project from the Ministry of Education, Culture, Sports, Science and Technology of Japan to Y. Hayashizaki. This work was also supported by the research grant for Preventure Program C of Japan Science and Technology Agency to Y. Hayashizaki.Competing interests. The authors have declared that no competing interests exist.
Functional annotation of a full-length mouse cDNA collection. J Kawai, A Shinagawa, K Shibata, M Yoshino, M Itoh, Nature. 409Kawai J, Shinagawa A, Shibata K, Yoshino M, Itoh M, et al. (2001) Functional annotation of a full-length mouse cDNA collection. Nature 409: 685-690.

Analysis of the mouse transcriptome based on functional annotation of 60,770 fulllength cDNAs. Y Okazaki, M Furuno, T Kasukawa, J Adachi, H Bono, Nature. 420Okazaki Y, Furuno M, Kasukawa T, Adachi J, Bono H, et al. (2002) Analysis of the mouse transcriptome based on functional annotation of 60,770 full- length cDNAs. Nature 420: 563-573.

The transcriptional landscape of the mammalian genome. P Carninci, T Kasukawa, S Katayama, J Gough, M C Frith, Science. 309Carninci P, Kasukawa T, Katayama S, Gough J, Frith MC, et al. (2005) The transcriptional landscape of the mammalian genome. Science 309: 1559- 1563.

Development and evaluation of an automated annotation pipeline and cDNA annotation system. T Kasukawa, M Furuno, I Nikaido, H Bono, D A Hume, Genome Res. 13Kasukawa T, Furuno M, Nikaido I, Bono H, Hume DA, et al. (2003) Development and evaluation of an automated annotation pipeline and cDNA annotation system. Genome Res 13: 1542-1551.

CDS annotation in full-length cDNA sequence. M Furuno, T Kasukawa, R Saito, J Adachi, H Suzuki, Genome Res. 13Furuno M, Kasukawa T, Saito R, Adachi J, Suzuki H, et al. (2003) CDS annotation in full-length cDNA sequence. Genome Res 13: 1478-1487.

CRITICA: Coding region identification tool invoking comparative analysis. J H Badger, G J Olsen, Mol Biol Evol. 16Badger JH, Olsen GJ (1999) CRITICA: Coding region identification tool invoking comparative analysis. Mol Biol Evol 16: 512-524.

Computational gene prediction using multiple sources of evidence. J E Allen, M Pertea, S L Salzberg, Genome Res. 14Allen JE, Pertea M, Salzberg SL (2004) Computational gene prediction using multiple sources of evidence. Genome Res 14: 142-148.

Amino acid translation program for fulllength cDNA sequences with frameshift errors. Y Fukunishi, Y Hayashizaki, Physiol Genomics. 5Fukunishi Y, Hayashizaki Y (2001) Amino acid translation program for full- length cDNA sequences with frameshift errors. Physiol Genomics 5: 81-87.

RNAdb-A comprehensive mammalian noncoding RNA database. K C Pang, S Stephen, P G Engstrom, K Tajul-Arifin, W Chen, Nucleic Acids Res. 33Pang KC, Stephen S, Engstrom PG, Tajul-Arifin K, Chen W, et al. (2005) RNAdb-A comprehensive mammalian noncoding RNA database. Nucleic Acids Res 33: D125-D130.

A perfect message: RNA surveillance and nonsense-mediated decay. M W Hentze, A E Kulozik, Cell. 96Hentze MW, Kulozik AE (1999) A perfect message: RNA surveillance and nonsense-mediated decay. Cell 96: 307-310.

Nonsense-mediated mRNA decay: Splicing, translation and mRNP dynamics. L E Maquat, Nat Rev Mol Cell Biol. 5Maquat LE (2004) Nonsense-mediated mRNA decay: Splicing, translation and mRNP dynamics. Nat Rev Mol Cell Biol 5: 89-99.

The generic genome browser: A building block for a model organism system database. L D Stein, C Mungall, S Shu, M Caudy, M Mangone, Genome Res. 12Stein LD, Mungall C, Shu S, Caudy M, Mangone M, et al. (2002) The generic genome browser: A building block for a model organism system database. Genome Res 12: 1599-1610.

CAGE Basic/Analysis Databases: The CAGE resource for comprehensive promoter analysis. H Kawaji, T Kasukawa, S Fukuda, S Katayama, C Kai, Nucleic Acids Res. 34Kawaji H, Kasukawa T, Fukuda S, Katayama S, Kai C, et al. (2006) CAGE Basic/Analysis Databases: The CAGE resource for comprehensive promoter analysis. Nucleic Acids Res 34: D632-D636.

Targeting a complex transcriptome: The construction of the mouse full-length cDNA encyclopedia. P Carninci, K Waki, T Shiraki, H Konno, K Shibata, Genome Res. 13Carninci P, Waki K, Shiraki T, Konno H, Shibata K, et al. (2003) Targeting a complex transcriptome: The construction of the mouse full-length cDNA encyclopedia. Genome Res 13: 1273-1289.

BLAT-The BLAST-like alignment tool. W J Kent, Genome Res. 12Kent WJ (2002) BLAT-The BLAST-like alignment tool. Genome Res 12: 656-664.

A computer program for aligning a cDNA sequence with a genomic DNA sequence. L Florea, G Hartzell, Z Zhang, G M Rubin, W Miller, Genome Res. 8Florea L, Hartzell G, Zhang Z, Rubin GM, Miller W (1998) A computer program for aligning a cDNA sequence with a genomic DNA sequence. Genome Res 8: 967-974.

EMBOSS: The European Molecular Biology Open Software Suite. P Rice, I Longden, A Bleasby, Trends Genet. 16Rice P, Longden I, Bleasby A (2000) EMBOSS: The European Molecular Biology Open Software Suite. Trends Genet 16: 276-277.

The Mouse Genome Database (MGD): The model organism database for the laboratory mouse. J A Blake, J E Richardson, C J Bult, J A Kadin, J T Eppig, Nucleic Acids Res. 30Blake JA, Richardson JE, Bult CJ, Kadin JA, Eppig JT (2002) The Mouse Genome Database (MGD): The model organism database for the laboratory mouse. Nucleic Acids Res 30: 113-115.

Tolstoshev CM (1993) dbEST-Database for ''expressed sequence tags. M S Boguski, T M Lowe, Nat Genet. 4Boguski MS, Lowe TM, Tolstoshev CM (1993) dbEST-Database for ''expressed sequence tags.'' Nat Genet 4: 332-333.

Basic local alignment search tool. S F Altschul, W Gish, W Miller, E W Myers, D J Lipman, J Mol Biol. 215Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ (1990) Basic local alignment search tool. J Mol Biol 215: 403-410.

Comparison of DNA sequences with protein sequences. W R Pearson, T Wood, Z Zhang, W Miller, Genomics. 46Pearson WR, Wood T, Zhang Z, Miller W (1997) Comparison of DNA sequences with protein sequences. Genomics 46: 24-36.

The Universal Protein Resource (UniProt). A Bairoch, R Apweiler, C H Wu, W C Barker, B Boeckmann, Nucleic Acids Res. 33Bairoch A, Apweiler R, Wu CH, Barker WC, Boeckmann B, et al. (2005) The Universal Protein Resource (UniProt). Nucleic Acids Res 33: D154-D159.

The InterPro database, an integrated documentation resource for protein families, domains and functional sites. R Apweiler, T K Attwood, A Bairoch, A Bateman, E Birney, Nucleic Acids Res. 29Apweiler R, Attwood TK, Bairoch A, Bateman A, Birney E, et al. (2001) The InterPro database, an integrated documentation resource for protein families, domains and functional sites. Nucleic Acids Res 29: 37-40.

Predicting transmembrane protein topology with a hidden Markov model: Application to complete genomes. A Krogh, B Larsson, G Von Heijne, E L Sonnhammer, J Mol Biol. 305Krogh A, Larsson B, von Heijne G, Sonnhammer EL (2001) Predicting transmembrane protein topology with a hidden Markov model: Application to complete genomes. J Mol Biol 305: 567-580.

Machine learning approaches for the prediction of signal peptides and other protein sorting signals. H Nielsen, S Brunak, G Von Heijne, Protein Eng. 12Nielsen H, Brunak S, von Heijne G (1999) Machine learning approaches for the prediction of signal peptides and other protein sorting signals. Protein Eng 12: 3-9.


PRISMA: Program of Research to Integrate the Services for the Maintenance of Autonomy. A system-level integration model in Quebec

PhDMargaret Macadam email@example.com
Adjunct
Factor-Inwentash Faculty of Social Work
Factor-Inwentash Faculty of Social Work
University of Toronto
University of Toronto
TorontoCanada

PRISMA: Program of Research to Integrate the Services for the Maintenance of Autonomy. A system-level integration model in Quebec
Integrated Care Case Correspondence to: Margaret MacAdam, PhD, Associate Professor (Adjunct),PRISMAintegrationelderlycoordination
The Program of Research to Integrate the Services for the Maintenance of Autonomy (PRISMA) began in Quebec in 1999. Evaluation results indicated that the PRISMA Project improved the system of care for the frail elderly at no additional cost. In 2001, the Quebec Ministry of Health and Social Services made implementing the six features of the PRISMA approach a province-wide goal in the programme now known as RSIPA (French acronym). Extensive Province-wide progress has been made since then, but ongoing challenges include reducing unmet need for case management and home care services, creating incentives for increased physician participation in care planning and improving the computerized client chart, among others. PRISMA is the only evaluated international model of a coordination approach to integration and one of the few, if not the only, integration model to have been adopted at the system level by policy-makers.BackgroundThe PRISMA Research project was established to address the problem of lack of continuity to care experienced by elderly people with chronic conditions in the Estrie region of Quebec. Its objective was to evaluate the implementation of an Integrated Service Delivery Network (ISD French acronym) to improve the health, empowerment and satisfaction of frail older people and to change health and social service utilization without increasing caregiver burden[1]. The features of the Integrated Service Delivery were service coordination, single entry point, case management, individualized service plan, a single functional assessment tool and a shared information system. A unique feature of the PRISMA approach was its coordination and case management feature, which called for the participating agencies to share responsibility for clients but did not require merger of any providers. The PRISMA Project took place in three areas of the region [Sherbrooke (urban), Granit (semi-rural) and Coaticook (rural)].The PRISMA Project was implemented using a three level governance model with clear responsibility at each level. At the strategic level, a local governance table was composed of representatives from the hospitals, the health and social service agency (somewhat similar to provincial home care and case management agencies in other parts of Canada but with a wider mandate), physicians and local community agencies. A major feature of the work of the local governance table was to shift strategic planning from a client focus to a population basis. The local governance table was the "owner" of the project and responsible for its activities. At the tactical level, managers of the direct service agencies formed a local management committee responsible for the administration of the project. At this level,

staffing issues, budgeting problems and communications among agencies were discussed and adjustments put in place as needed. At the clinical level, local clinicians formed multidisciplinary teams to manage client care, ensure the care plan was being followed and to make adjustments to the plan based on client needs. While all seniors in need were eligible for Project services, the criteria for admission included age (over 65), presence of significant functional disabilities (score over 15 on the functional assessment or be in an Iso-SMAF profile of 4 or over), have support for care at home and need formal care (three or more health care or social services) [2]. There was no financial means test for admission to the Project.

Each geographical area was responsible to manage the total client caseload using its local resources for services, including physician, hospital, residential long-term care and home and community-based care. The local caseload varied by the size of the elderly population in each area.

Services ranged from inpatient, emergency room and outpatient hospital care, primary care, specialized geriatric care, rehabilitation, in-home nursing, therapy, personal care, home support, home-delivered meals, day centres, pharmacy, equipment, supplies and domestic care.

The evaluation of the PRISMA Project followed subjects and their principle caregivers in the three experimental areas in Estrie and in three comparison areas elsewhere in Quebec for four years. Subjects were predominantly women (62%) with an average age of 83. At baseline and throughout the four-year study, the characteristics of both experimental and comparison groups were similar. After four years, the results of the evaluation indicated that, compared to the system of care in the three comparison areas, the PRISMA model produced significant reductions in the prevalence and incidence of functional decline, lowered unmet needs, reduced emergency room visits and had an almost significant effect on reducing hospitalizations. It also found a significant increase in client satisfaction and empowerment. There was no effect on rates of institutionalization, consultations with health professionals or on the use of home care services. There was also no statistically significant change in costs; thus the results of the evaluation indicate that PRISMA was cost effective because it produced improved results at no additional cost [3].

Although the earliest results of the evaluation were not published until 2006, in 2001, the Minister of Health and Social Services required all regions in Quebec to start to implement the PRISMA Integrated Service Delivery features. Each region has flexibility in how they are implemented. Today the PRISMA project does not exist per se; it is part of a Quebec-wide programme called Réseau de Services Intégrés aux Personnes Âgées (RSIPA) which has become the normal system of care for the elderly in Quebec.

Early implementation challenges in PRISMA, and now in RSIPA, included gaining tangible cooperation from agency administrators: that is, moving from talking about cooperation to changing agency internal priorities and procedures to make it happen. Operationally, obtaining funding for case management and for a shared information system was critical to implementing PRISMA. These issues were solved for the PRISMA research project but are ongoing problems in the RSIPA programme.

One of the strengths of PRISMA was the sustained commitment of the regional health leadership. Local health and social service leaders formed a partnership among university researchers, the provincial government, regional health and social service planning and funding authorities as well as managers from the home and community care service centres. This group was instrumental in creating the model of care and making resources and staff available to work on it. Perhaps most importantly, key leaders remained in place over a number of years, enabling the integration goal to remain central in the face of financial problems and external changes that could have derailed the project.

Another strength of the PRISMA approach was the adoption of an evidence-and population-based approach. The long-term goal is improved care for all seniors, not just the management of services for the frail elderly.

PART 1: the model of integrated care Introduction The Program of Research to Integrate the Services for the Maintenance of Autonomy (PRISMA) began with a research project in three parts of the Estrie (known as the Eastern Townships in English) region of Quebec in 1997. The PRISMA Research Project no longer exists because its features have become the regular approach to care of the frail elderly in the Province. The PRISMA approach is now embedded in a province-wide programme called RSIPA or Network of Integrated Services for the Elderly. The province-wide programme is referred to as RSIPA in this report.

Background

The Estrie region is located 150-km east of Montreal. It has a population of 310,000, with about 13% being aged 65 or over. The region is divided into seven areas; each with its own set of health and social service providers.

PRISMA emerged out of an earlier project in Bois Francs region by a team of researchers at Laval University. This project was designed to alleviate emergency room and other hospital problems in the area. Evaluation results indicated positive results in reducing the desire to institutionalize and in caregiver burden [4]. Features later incorporated in PRISMA originated in this first project.

The objective of the PRISMA Project was to evaluate the implementation of an Integrated Service Delivery Network. The Integrated Service Delivery was designed to improve the health, empowerment and satisfaction of frail older people and to change health and social service utilization without increasing caregiver burden [5].

The PRISMA project was established to address the problem of lack of continuity to care experienced by elderly people with chronic conditions. Lack of continuity affects timely access to care and the efficiency of health care service delivery [6]. Specifically, the local health and social care system exhibited the following problems for elders with ongoing and variable care needs: multiple entry points, services needs determined by the provider rather than by independently assessed client needs, redundant assessments, inappropriate utilization of costly resources, poor hospital to home transitions, delays in receipt of services, poor information sharing amongst professionals and unmet needs [7].

Client group

The criteria for admission to the PRISMA Research Project were selected based on a review of integration projects for the frail elderly. While all seniors in need were eligible for Project services, only those meeting the criteria below were eligible for intensive case management:

. Age: over 65 years old; . Significant functional disabilities of a score of over 15 or an Iso-SMAF profile of 4 or higher;

. Support for care at home; and . Need for formal care: three or more health care or social services [8].

Today, the regions may adjust the eligibility criteria in the RSIPA programme. For example, in the Sherbrooke area, the criteria for admission are stricter: clients must have a risk classification profile of 5 or above, and those under age 65 can be admitted if their abilities and needs meet the criteria for those aged 65 and above (interview).

Clients did not have to meet the Quebec nursing home admission criteria for admission to the PRISMA project and do not have to meet them in the RSIPA programme. There was, and still is, no financial means test for entry to the programme.

There were no formal criteria for discharging clients from the PRISMA Project and there are none today in RSIPA. In Sherbrooke, about 25% of clients are discharged each year because of death or admission to a long-term care facility (interview).

In Sherbrooke, a city in the Estrie region and one of the research areas for PRISMA, there were 3181 clients receiving home care services in 2011. Of these, the average age was 84.6 (ranging from 57 to 106) and 63% were women. Approximately, 40% of home care clients received case management. Using a classification system based on the SMAF assessment data, the RSIPA Sherbrooke client population was categorized as shown in Table 1. Clients in the PRISMA Research Project, and now RSIPA, are referred to a single entry system but through a variety of mechanisms: they may self-refer, be referred by their physicians or by other professionals in the community, such as hospital discharge planners, home care providers or housing and social care providers.

Approach to care

The response to the problems (listed above) being experienced by clients, professionals and health system managers in Estrie was to develop an Integrated Service Delivery Network comprised of six main features. These features, evaluated in the PRISMA Project, are now being implemented throughout Quebec in RSIPA.

Coordination among services

A distinguishing feature of the PRISMA Research Project and now RSIPA is its emphasis on agency coordination and shared responsibility for clients; it does not require the organizational merger of any providers but it does require agencies to give up some of their autonomy to meet coordination goals.

The foundation for coordination rests upon three levels of governance: strategic, tactical and clinical. See the governance section below for more detail on the stratgetic and tactical levels. At the clinicial level, a multidisciplinary team, composed of representatives of the participating agencies, meets to solve clinical problems. At the time of the PRISMA research project, health and social care providers included hospitals, rehabilitation services, specialized geriatric programmes, public long-term care institutions, private residential care such as retirement homes and group homes, case management, in-home nursing, therapy, social work, equipment and supplies, pharmacy, primary care physicians and other local voluntary services such as home support and day programmes. Today, the local hospitals (with the exception of teaching hospitals), residential and long-term care facilities, and the home and community care agencies (CLSCs) have been merged into one organization, the Centre for Health and Social Services (CSSS French acronym). Representatives of the CSSS meet with the other partners to plan care at the clinical level.

Single point of entry

The flow chart below depicts the steps and flow of eligible clients through PRISMA and now RSIPA ( Figure 1). Selfreferred clients are screened at the time of initial contact with the programme. A seven question screening tool (PRISMA-7) was developed to quickly identify seniors who should receive the full assessment [9]. A score of 4 positive answers on the PRISMA-7 triggers a referral for a full assessment to determine eligibility for integrated care services.

Those who are not eligble for case management are referred to a local direct service provider. All eligible clients are referred to the case managers.

Case management

As is the case with most integrated care programmes, case management was a central feature of the PRISMA programme and now RSIPA. In the PRISMA programme, case managers were staff with special training, were independent of service providers and responsible to the local governance committee. Today, the case managers are co-located with other service providers within the CSSS and may be responsible to the local governance committee or to the CSSS, depending on the region.

The role of the case manager is to perform the basic assessment, develop the care plan based on client needs and family input, coordinate care with the primary care physicians, refer to other professionals as required, ensure that the required services are delivered from participating agencies, advocate for the client as required and conduct periodic re-assessments as part of the monitoring of the client's situation. The service providers are responsible to the case manager for the delivery of care as described in the care plan although the case manager does not have direct authority over service providers.

The case manager is responsible for managing the full array of services needed by the client and also for maintaining close relationships with the primary care physician who takes on the medical management role for the client. The primary care physician ensures that specialist medical is delivered as needed and coordinates supportive care for medical services, such as transportation with the case manager. All primary care physicians have input to the care plan. Typically, the case manager communicates about the care plan or new medical issues with the primary care physician by telephone or FAX. Currently, most community physicians do not have access to the computerized care plan. If primary care physicians are formally associated with CSSS or the local hospital, they have access to the This article is published in a peer reviewed section of the International Journal of Integrated Care computerized care plan. Over time, the goal is to enable all primary care physicians in Quebec to more actively participate with case managers.

The case manager is also responsible for managing client transitions from home to hospital and vice versa. If a PRISMA programme patient arrived unexpectedly in the hospital emergency room, the case manager was notified. If the client was admitted, the case manager became part of the discharge planning team to ensure that community supports were available upon discharge. Today, the connection between the case managers and hospital staff can vary from region to region. In Sherbrooke, for example, a list of case-managed clients who have been admitted to hospital within the past 24 hours or visited the Emergency Department is sent to the case management team on a daily basis.

Most case managers are social workers or nurses although other disciplines such as physio-or occupational therapists can take on the case management role. Case managers can provide direct services, such as nursing, but on a very limited basis. The caseload goal is 40-45 clients per worker although it is currently about 50 clients in some areas. There can be wait lists for case management.

Individualized service plan

The service plan is developed after the assessment is completed. One of the responsibilities of the case manager is to achieve consensus among other service providers to ensure that the needed services will be delivered in a timely manner. The service providers work with the case manager to help solve issues that arise in adhering to the service plan. Reassessment of the service plan takes place based on the needs of the client; very stable, light care clients may be reassessed less than once a year while clients with very heavy needs are reassessed more frequently. In some CSSS areas, the reassessment process is being revised based on new data on the stability of clients in the classification profiles (interview). Unique assessment tool The SMAF (French acronym for Functional Autonomy Measurement System [11]) is a 29-item assessment tool used to ascertain functioning in five areas: ADLs, IADLs, mobility, communication and mental abilities. The functional items are based on the WHO classification of disabilities. Data from the SMAF are analysed to produce 14 classification profiles which are used to monitor service planning and usage across similar types of clients [2]. The profiles are also used to adjust admission criteria. In the future, they could also be used to estimate the funding allocation for the programme.

Information tool

The computerized client chart is used to track assessment and reassessment data, the care plan and provider interventions. The computerized client chart can be assessed by all professionals and institutions participating in the programme via the Quebec Health and Social Services Network. Strict privacy and security protocols are in place. In addition to monitoring programme operational activities, data from the computerized client chart are available for research studies. An issue with the computerized client chart is that it is not available to physicians in private practice who are not affiliated with a CSSS or otherwise have access to the Province's Health and Social Service information system. Nor does the current computerized system provide a user-friendly way to input the care plan into the system (interview).

Service delivery system

Staff at the single entry point emphasize the client's ability to self-manage their care for those clients not eligible for case management. Clients might be referred to direct service providers such as home care staff within the CSSS or local community partners. When used, partner agencies are reimbursed by the CSSS. The vast majority of clients receive their services directly from the CSSS. 1

Quality assurance

As noted above, one of the goals of PRISMA (and now of RSIPA) is improving the empowerment and satisfaction of older people. Case managers place increased emphasis on including client and family preferences in care planning. Today, client/family surveys are conducted every three years to get feedback on satisfaction and experiences with care.

Depending on the area, standardized guidelines for care are used but they are not included in the computerized system.

The Guide OSIRSIPA (Outil de suivi de l'implantation du RSIPA) measures the degree of implementation of the RSIPA. It is filled out and sent to the Ministry of Health and Social Services once a year and contains a number of process measures such as the extent to which clients receive the services they need, the range of services available and implementation of the PRISMA features.

By 2015, the Ministry expects that an average of 55% of all components of PRISMA/RSIPA will be implemented in Québec (interview). It is expected that the newly appointed Minister of Health and Social Services (the former Dean of the Faculty of Medicine and Health Sciences, University of Sherbrooke and the former PRISMA Research Director at the Centre of Aging) will increase the implementation goal from 55% to 70% (interview).

PART 2: implementation and organisation Implementation

The PRISMA Project took place in three areas of the region (Sherbrooke (urban), Granit (semi-rural) and Coaticook (rural). A comparison group was selected from three areas in other parts of Quebec that had similar characteristics 1 In some regions, a direct payment option is available for certain clients. In Sherbrooke, clients with decreasing independence, low income and living in private retirement homes can receive a direct payment option which allows them to organize their own care providers. Otherwise, they receive services under the usual system (from the CSSS and/or community partners). A full assessment is required. If clients opt for the direct payment option, usually they use the payment to pay for extra care from the staff of the retirement home. Clients receiving the direct payment option are followed by a case manager or other staff (if they do not meet case management criteria). They are assessed annually or more frequently as required. In all cases, the wishes of the client and family are part of the care planning process (interview). but had not implemented the features of PRISMA. To implement PRISMA, two key features had to be implemented: first; there needed to be adequate funding for case management and second, a shared electronic record, the computerized client chart, needed to be available to the network providers (interview). Both of these challenges were overcome in the PRISMA project but they continue to be issues in implementing RSIPA.

Governance

A key feature of PRISMA Model is the network model of service delivery. In PRISMA, each provider retains its own organizational and governance independence but the participants recognize the interdependence of the service delivery system and the importance of cooperation in order to improve client and system outcomes. Coordination of care and adherence to the care plan and system goals are supported by three levels of governance.

Strategic

During the operations of the PRISMA Project, at the local level, all key decision-makers had a representative on the local governance table. These agencies included the hospitals, the health and social service agency (somewhat similar to provincial home care and case management agencies in other parts of Canada but with a wider mandate), physicians and local voluntary and private service agencies. A major feature of the work of the local governance table was to shift strategic planning from a client focus to a population basis. The local governance table was the "owner" of the project and is responsible for its activities. Prior to the start of operations, the agencies on the local governance table made commitments to support the goals of the programme and to make service and administrative funds available for its operations. If there were serious problems with provider performance that were not solved at either the clinical or tactical levels, it was the responsibility of the local governance table to resolve the issue. Today the local CSSS is responsible for the governance of RSIPA.

Tactical

Services managers of the direct service agencies are members of a local management committee which is responsible for the administration of the project. At this level, staffing issues, budgeting problems and communications among agencies are discussed and adjustments put in place as needed.

Clinical

Local clinicians form multidisciplinary teams to manage client care, ensure the care plan is being followed and to make adjustments to the plan based on client needs. Physicians are not able to bill for care coordination; they receive no extra financial incentives to participate. Physicians do not normally attend the clinical team meetings but are contacted separately by the case manager if required.

Organization

Care for the elderly in Quebec is typically delivered through publically funded public agencies, such as hospitals and a variety of not-for-profit and for profit agencies. Case management and service planning takes place within the available health and social care resources of each area. The care package ranges from medical to hospital to rehabilitation to home care services, day programmes and social services such as meals on wheels and home support.

The services provided, the types of agency delivering each service and whether there is a cost to the clients are shown in Table 2.

RSIPA, as was PRISMA before it, is driven by quality and cost concerns. One aspect of improving quality was to include family concerns in care planning and service management. Today, care for the frail elderly is more patientcentred than it has been in the past. Families are actively involved in the care planning process.

Financial context

The PRISMA project was funded from a variety of sources. Clients receive primary and hospital services at no charge under Canada's universal public health insurance programme covering medically required services. Pharmacy costs are covered under Quebec's universal pharmacy programme at no cost to clients except for a small co-payment. Professional home care such as nursing is available free to clients while supportive care such as homemaking is provided according to Quebec's sliding fee programme. Clients receiving residential housing services, such as nursing homes or retirement homes, pay fees depending on their income and type of care they receive. For example, clients living in nursing homes pay for their accommodations while their health, social and medical costs are provided through public funds.

It is important to note that RSIPA operates from the regular allocation of funding from the Ministry of Health and Social Services. When there is insufficient funding for full operations, case management caseloads can become higher than desired and waiting lists for case management and direct services develop.

Accountability for RSIPA spending and results is established through funding and performance contracts between the CSSSs and the regional authorities using measures established by the Ministry of Health and Social Services. If there are shortages of particular services, the regions might decide to make additional funding to expand the service(s). In Estrie, for example, funding of case managers became a priority for the Regional Authority and the CSSSs when new funds were made available by the Ministry.

The Ministry of Health and Social Services has an established rate setting mechanism for provider remuneration, which establishes payment rates for RSIPA services.

PART 3: impact and sustainability

Evidence of impact

The evaluation of the PRISMA Project followed subjects and their principle caregivers in the three experimental areas in Estrie (Sherbrooke, Granit and Coaticook) and in three comparison areas from the Chaudière-Appalaches region for four years. Subjects were predominantly women (62%) with an average age of 83. At baseline and throughout the four year study, the characteristics of both experimental and comparison groups were similar. After four years, the results of the evaluation indicated that, compared to the normal system of care, the PRISMA model produced significant reduction in the prevalence and incidence of functional decline, on meeting unmet needs, and reduced Emergency Department visits and had an almost significant effect on reducing hospitalizations. It also found a significant increase in client satisfaction and empowerment. There was no effect on rates of institutionalization, consultations with health professionals or on the use of home care services. There was also no statistically significant change in costs; thus the results of the evaluation indicate that the Project was cost effective because it produced improved results at no additional cost [3]. . Statistical analyses showed no significant difference between the two groups for total annual costs.

Researchers concluded that the PRISMA model is more efficient considering the outcomes on functional decline, satisfaction, empowerment and unmet needs; as efficient considering functional autonomy; and less efficient considering social functioning and caregiver burden [13].

The positive results of PRISMA were not reached until year 3 of the evaluation. By year 3, the rate of implementation of all of the key features of the model in the research areas approached 80%. The rate of physician participation was 73% [12]. Today, it is not possible to compare outcome and cost data because the features of the PRISMA project are being implemented in the RSIPA programme across the province. Results from the PRISMA project indicated that effects of the network are felt when the implementation level reaches 70%. Thus the Ministry of Health and Social Services is placing emphasis on the regions to reach at least 55% implementation by 2015 (interview). Some consider this goal as too low and expect it to be raised (interview).

Sustainability and spread

As mentioned above, PRISMA features are now embedded in RSIPA, which is becoming the normal system of care for the frail elderly in the Province. The decision to adopt the PRISMA model across the province is the key test of sustainability of any demonstration.

The province is currently monitoring the degree of implementation of the RSIPA features in every region. From 2008 to 2011, there has been impressive improvement in implementing the key features of the programme as can be seen from Table 3.

It can be seen that there has been little progress in implementing the service plan. Every client has a service plan but the issue reflected in this chart is that the service plan cannot be added to the shared computerized record in a userfriendly manner.

Performance measures are shared among the network providers. There are no financial or other special incentives to reach outcome targets. If there are shortages of particular services, the local governance board might decide to make additional funding to expand the service(s). In Estrie, the board has made additional funding for case management and home support services, for example (interview).

Additionally, there is a PRISMA-type replication being implemented for frail elderly people underway in France; thus the model may spread in some form to at least one other country [14].

PART 4: barriers and facilitators to effective implementation Systemic and contextual factors Leadership

A key feature of the PRISMA Research Project was the partnership among university researchers, the provincial government and regional health and social service planning and funding authorities as well as managers from the home and community care service centres (CLSCs French acronym). Two research teams (Research Centre on Aging in Sherbrooke and Laval University Geriatric Research Team in Quebec City), the Ministry of Health and Social Services, the Health and Social Services Boards in Estrie, Mauricie et du Centre-du-Québec, Laval, Montérégie, Capitale-Nationale and the Sherbrooke Geriatric University Institute came together to plan, implement and evaluate the effectiveness of the initiative. The importance of this partnership cannot be overestimated because, in addition to supporting the design, implementation, operations and evaluation of the PRISMA project, it was instrumental in simultaneously moving an integrated care agenda ahead throughout the entire region. The work was promoted and sustained for over 15 years through the commitment of the key leaders in the Region.

Moving beyond talking to action From the outset, the PRISMA project, operating in three areas of the Estrie, was embedded within a health system culture that was working to improve integrated care for the elderly population throughout the region. However, it took time to move from talking about cooperation for agencies to change their procedures. Without the sustained leadership and focus of key players, the development of shared accountability for care would not have happened (interview).

PRISMA/RSIPA is a coordination model which depends on the ability of local providers to voluntarily participate in the programme by giving up some of their own autonomy. The PRISMA Research Project built upon efforts to increase coordination of services in the Estrie Region which began in 1995, when, in six of the seven areas of the region, the local hospital, the home and community care service centre and residential and long-term care centres merged to form a single entity. In 1998, the Estrie Regional Health Authority recommended implementing integrated networks across the region (which were being tested in the three PRISMA research sites in Estrie). These networks were to include a joint triage system, a case management approach and the use of individualized service plans. Over time, the key elements of integrated networks were put into place as recommended by the researchers involved with the design of the PRISMA project. During this period, the Quebec Ministry of Health and Social Services implemented changes designed to encourage integrated care for the elderly in the Province. For example, the Ministry issues guidelines for coordinated care of the elderly in 2001; in 1998, the home and community care centres (the CLSCs) and later other agencies started to use a Multiclientele Assessment Tool (OEMC French acronym); and centralized single entry points for care in each of the areas were established in 2001. The use of a common assessment instrument allowed the transmission of client data among providers. In 2001-2002, case management was implemented in the region. In 2002, the PRISMA-7 instrument was put in place to locate frail elders [15]. In Estrie, PRISMA was not viewed as a one-time research project designed only to test research hypotheses but rather as part of a larger system change designed to improve care for the frail elderly.

Flexibility at the regional and local levels One of the strengths of the implementation of the changes in Estrie, including the PRISMA project, and now RSIPA has been the understanding that a range of local and regional differences affect implementation of policy goals. For example, it took longer to implement the PRISMA features in the rural demonstration area than in the urban and semi-rural areas. A key decision has been to allow regions flexibility in implementing the features of RSIPA in ways that are appropriate for local areas.

Organizational factors

The health and social care governance structure in Quebec is already more integrated than in other Canadian provinces. Quebec is the only province in which all social and health care services are administered by one Ministry.

Reorganization

There was an externally mandated reorganization of the entire health care system in Quebec in 2004. The reorganization affected the pace of implementing integrated care in Estrie and the model itself. It is still impacting the system in Quebec.

In 2004, the Quebec Legislature passed Bill 25, an Act Respecting Local Health and Social Services Network Development Agencies which called for the reorganization of the health and social service system in the Province. The reorganization called for three levels of governance:

Regional. Eighteen regional agencies are responsible for planning, coordination and allocation funding for the local system of care;

Area networks. Within the regions, local health and social service networks were created which include all health and social care providers at the local level and are accountable for the health and social care of the population within their area; and

Health and social care centres. Each area network created a health and social care centre from the merger of the local hospital(s) (teaching hospitals were exempted), publicly owned long-term care facilities and the home care programme (CLSC). The Centres (CSSS) are the focal points of care within each network but they work in partnership with other agencies in the network. There are now 95 health and social service networks and centres (CSSS) across the Province.

The extent of the merger temporarily diverted the energy of health care managers away from ongoing integration or other activities to deal with the governance, human resources, clinical care and funding repercussions of the merger. This was just as true in Estrie as it was elsewhere. The merger activity is proceeding at different speeds across the Province; today some areas have completed the merger while others are still in process.

The results of the merger have been both positive and negative in Estrie. On the positive side, there are fewer different agencies involved in service coordination and decision making at the strategic, clinical and tactical levels: thus problems can be resolved more quickly. But, there is concern about the balance of power between organizations of different sizes and budgets. An ongoing issue is that agencies outside of the CSSS may not receive adequate funding to support their services (interview). As well, the merger upset the established relationships amongst providers; new relationships are being developed (interview).

Even with the merger, there is still silo budgeting for individual types of services (interview). The merger might make it easier to implement new funding models, such as capitation, that could encourage resources to be used more efficiently.

During the period of the merger, clinical integration was set back because of the time and resources needed to create a new culture and operations within the newly merged organization. Researchers associated with PRISMA stress that mergers are not necessary for the model to be successful (interview).

Some of the other possible advantages of the merger have yet to be realized In addition to the funding possibilities. The merger might help to reorient the system towards a more preventative approach. One researcher suggested making aging per se the focal point for planning and funding at the strategic level (interview).

Operational/service delivery factors

. The model features. The PRISMA model places emphasis on service coordination, single entry point, case management, individualized service plan, the unique assessment tool and the electronic information tool. Researchers and managers feel that each feature is essential to the overall success of integrated care for the frail elderly (interview). As the model has matured, it has become more evident that service partners need to include areas outside of the Ministry of Health and Social Services, such as the Ministry of Transportation and the Ministry of Municipal Affairs (interview). . Information systems. Another key externally mandated change which affects the delivery of care has been the implementation of a new province-wide information system, SOLUTION RSIPA. Its' platform is more stable than the earlier system used for PRISMA but it does not yet carry all of the features available under the earlier system. For example, it is not possible to enter the Service Plan into the system in a user-friendly manner. All key players stress the importance of a shared electronic record to promote integration and the need to allocate funding to improve the SOLUTION RSIPA (interview). The Ministry is expected to have improved the shared electronic record, especially with regard to the ease of entry of the care plan and medical evaluation by April 2014 (interview). . Unmet needs. The province has not made new money specifically available for case management (interview). As a Province, over time, Quebec has made proportionately less funding available for home and community care than other provinces [16]. As a result, there can be waiting lists for case management and home care services. The size and type of waiting list varies across the Province depending on local conditions and decisions made within the region and in the local CSSS. For example, in Sherbrooke, there is a waiting list for case management and home care services. Work is underway to optimize case management and home care services by regrouping and reorganizing them according to the level of service intensity required [low intensity, short term, high intensity (case management)]. To manage high need care, the single point of entry team ensures that services that are immediately required are put in place while the client is on the waiting list for case management. If needed, the single entry point staff will also do follow-ups ("relances") during the waiting period (interview). Other regions may be taking similar steps to manage the demand for care with available resources. The Ministry is working to assist regions to reduce wait times for services and to help them shift more care in the community and away from hospitals and long-term care institutions (interview). . Physician remuneration. Most primary care physicians in Quebec are paid on a fee-for-service basis which does not reimburse them for their time in care planning and service coordination. Even though in the PRISMA Project, 73% of physicians participated in the new model of care for their frail patients [17]; a new reimbursement system could increase their level of activity (interview). The Ministry is expecting to have implemented new agreements with physicians regarding both their ability to make home visits and to interact with case managers by the summer of 2013 (interview). . Training and coaching. The pace of change and staff turnover can quickly result in redirection of staff at all levels away from the goals of RSIPA or any other system change. Senior managers stress the importance of ongoing coaching and training about the goals of the RSIPA model. It is not enough to train staff at the onset of a change; managers need to be aware of the need to plan for the time and resources to maintain staff adherence to the goals of the change (interview). At the case management level, it is not an automatic activity of health professionals to consider the possible role of other providers, especially those at the home support level. There is a need to train case managers in the coordination features of their roles in RSIPA's integrated approach to care. Ongoing coaching is required to ensure that care plans reflect the full range of appropriate services, especially those that can be provided by the smaller community agencies (interview). The Ministry is seeking ways to assist the regions with staff training and coaching such as by producing training tools that can be used across the province. Some of these tools might also address best practice issues and coordination of services (interview). . Case management. In Quebec, case management for the elderly had been seen as a social work role. In PRISMA, other disciplines, such as nurses and rehabilitation therapists became case managers which provided an interdisciplinary approach. In other regions, there has been resistance by social workers to including other disciplines as case managers (interview). At the moment there is no one model of case management in Quebec. Researchers are working with the Ministry to develop a uniform case management model which may be completed in June 2013 (interview). . Quality management. The approach to quality assurance in Quebec has been to focus on process measures because it is very difficult to attribute outcomes to any particular intervention when so many variables are involved. At some point, it may be possible to compare the impact of RSIPA on other systems of care for the elderly in other jurisdictions.

Conclusions

The results of the PRISMA Project directly influenced public policy relating to the care of frail elderly people in Quebec. The key factors for success were rooted in the composition and role of the group at the strategic level which not only provided sustained leadership but also included researchers who grounded the features of the intervention in a thorough review of the integration literature. Their recommendations were supported by local decision-makers who were motivated to improve the quality of care as well as to contain costs (interview).

The basic features of the PRISMA model are not very different from those used in many other integration efforts. But PRISMA did not require either vertical or horizontal mergers or the creation of a new or greatly modified entity, such as the PACE projects require. It remains the only example of an evaluated coordination-type model as described by Leutz [18].

One of the lessons learned from other integration projects is that they need time to develop because of the range of service providers (from the very large, such as hospitals to the very small, such as local home delivered meals programmes). The PRISMA evaluation was a four-year longitudinal study. The wisdom of the choice of four years was seen in the fact that it took until year 3 to reach a 70% implementation rate of all the features. The impact of the model was not apparent until year 3.

There are 10 provinces in Canada and three territories (the material below does not include information about the territories). All provinces are implementing PRISMA features to varying degrees although not necessarily because they were influenced by PRISMA. Almost every province has a single entry point and case management service that includes assessment, care planning, ongoing monitoring and re-evaluation. Eight also have coordinated discharge planning with hospitals. However, most do not have shared electronic information systems. Nor do they have strong linkages with primary care physicians [19]. Administratively, as noted above, only Quebec administers all health and social services under one Ministry. The differences between the approach taken in Quebec and the rest of the provinces are:

. Implementing mechanisms for strong coordination across the range of service providers, include primary care physicians and hospitals; . The emphasis on a population-based approach, not just the management of services for the frail elderly.

These last two features are unique strengths of PRISMA and now RSIPA.

A feature of many case management models such as PRISMA/RSIPA is the separation between efforts to plan and deliver a wide range of health and social services to clients with chronic conditions and efforts to improve medical care for these same individuals. In Canada, reform of primary and specialized medical care has been viewed by policy-makers as independent of the local community care system. Because of issues in Canada related to waiting periods for care, primary care reforms have largely focused on increasing the supply of primary care physicians and on efforts to more efficiently manage patient loads, although there are chronic disease improvement projects such as those for people with diabetes, heart disease or Alzheimer Disease, to name a few. These projects tend to be led by specialist physicians supported by other health professionals but usually have weak links to case managed community care programmes. Yet both the community care programmes and the disease-based medical programmes typically have the same goals: to improve access to and quality of care for patients/clients and to be cost effective, usually by providing more care in the community. It would appear that a next step for RSIPA in Quebec and for other somewhat similar efforts in other parts of Canada would be to strengthen linkages between primary care and chronic disease reform with case managed community care reform to create truly integrated chronic care programmes.

Figure 1 .
1Steps and flow of eligible clients through PRISMA. Source: From Hébert, Raîche, Veil et al. [10]. International Journal of Integrated Care -Volume 15, 23 September -URN:NBN:NL:UI:10-1-114823http://www.ijic.org/ This article is published in a peer reviewed section of the International Journal of Integrated Care

Research funding was awarded by the Canadian Health Services Research Foundation, the Canadian Institutes of Health Research, the Fonds de la recherche en santé du Québec and Réseau Québécois de recherche sur le vieillissement. During the PRISMA research period, operational funding was available from the Ministère de la Santé et des Services sociaux du Québec and the health and social services agencies in the areas of Estrie, Montérégie, Mauricie International Journal of Integrated Care -Volume 15, 23 September -URN:NBN:NL:UI:10-1-114823http://www.ijic.org/ This article is published in a peer reviewed section of the International Journal of Integrated Care et du centre-du-Québec, Laval and Capitale-Nationale CSSS-IUGS (CSSS-Sherbrooke University Geriatrics Institute) and Université de Sherbrooke. Today operational funding for RSIPA is available from the Centres for Health and Social Services (CSSS French acronym).

Table 1 .
1Distribution of population risk classification profiles: Sherbrooke 2011Risk profile description

Table 2 .
2Services, provider type and client costs Mixture of not for profit and private agencies Yes, for accommodations in nursing homes (adjusted for income, and for care and accommodations in retirement homes and group homes This article is published in a peer reviewed section of the International Journal of Integrated CareService
Provider type
Client fees

Acute care
Public acute care hospital
No, except for private or semi-private accommodation

Primary care
Primary care physicians supported by
specialists

No

Specialized geriatric services
Public health nursing
No

Inpatient and outpatient
rehabilitation

In acute care hospitals, rehabilitation centres
or LTC facilities. Also available from private
outpatient centres.

No if provided in a public facility

In home nursing
Public nursing service
No

In home speech therapy,
occupational therapy &
physiotherapy

Public therapeutic services
No

Personal care
Public personal care staff at home
No

Private staff in private retirement homes
Yes

Homemaking
Private not for profit agencies
Perhaps, depending on income

Day centre
Public day care
No, except for a small contribution for transport and meals

Meals on wheels
Private not for profit agencies
Yes

Pharmacy
Private for profit providers
All are insured

Residential care such as nursing
homes, group homes and
retirement homes

Table 3 .
3Degree of implementation of network integration features in Quebec: 2008-2011Feature

International Journal of Integrated Care -Volume 15, 23 September -URN:NBN:NL:UI:10-1-114823http://www.ijic.org/
AcknowledgementsThis original case study paper was commissioned in 2013 as part of a project funded by the Commonwealth Fund that sought to compare and derive key lessons from seven country case examples of integrated care to older people and those with complex needs. The main report, comprising a summary of the findings across the seven case studies, was published by The King's Fund in 2014[20].
Effectiveness of the PRISMA integrated service delivery network: preliminary report on methods and baseline data. R Hébert, M-F Dubois, M Raîche, 978-2-89130-215-9Hébert R, Tourigny A, Raîche MII216Québec, Canada: EDISMIntegration of services for disabled people: research leading to actionHébert R, Dubois M-F, Raîche M, et al. Effectiveness of the PRISMA integrated service delivery network: preliminary report on methods and baseline data. In: Hébert R, Tourigny A, Raîche M, editors. Integration of services for disabled people: research leading to action. Volume II. Québec, Canada: EDISM; 2008. ISBN 978-2-89130-215-9. Chapter 11, p. 216.

Disability-based classification system for older people in integrated long-term care services: the Iso-SMAF profiles. N Dubuc, R Hébert, J Desrosiers, M Buteau, L Trottier, Archives of Gerontology and Geriatrics. 422Dubuc N, Hébert R, Desrosiers J, Buteau M, Trottier L. Disability-based classification system for older people in integrated long-term care services: the Iso-SMAF profiles. Archives of Gerontology and Geriatrics 2006;42(2):191-206.

RISPA overall performance: impacts on the use of health services, their costs and efficiency under the PRISMA-Estrie study. M Raîche, R Hébert, D Blanchette, 978-2-89130-215-923 September -URN:NBN:NL:UI:10-1-114823. Hébert R, Tourigny A, Raîche MQuébec, Canada: EDISMII560International Journal of Integrated CareRaîche M, Hébert R, Blanchette D, et al. RISPA overall performance: impacts on the use of health services, their costs and efficiency under the PRISMA-Estrie study. In: Hébert R, Tourigny A, Raîche M, editors. Integration of services for disabled people: research leading to action. Volume II. Québec, Canada: EDISM; 2008. ISBN 978-2-89130-215-9. Chapter 13, p. 560. International Journal of Integrated Care -Volume 15, 23 September -URN:NBN:NL:UI:10-1-114823 -http://www.ijic.org/ This article is published in a peer reviewed section of the International Journal of Integrated Care

Quasi-experimental study of the effectiveness of an integrated service delivery network for the frail elderly. André Tourigny, P D Bonin, L Hébert, R , Rochette L , Canadian Journal on Aging. 233André Tourigny PD, Bonin L, Hébert R, Rochette L. Quasi-experimental study of the effectiveness of an integrated service delivery network for the frail elderly. Canadian Journal on Aging 2004;23(3):231-46.

Effectiveness of the PRISMA integrated service delivery network: preliminary report on methods and baseline data. R Hébert, M-F Dubois, M Raîche, 978-2-89130-215-9Canada: EDISM. Hébert R, Tourigny A, Raîche MII216Integration of services for disabled people: research leading to actionHébert R, Dubois M-F, Raîche M, et al. Effectiveness of the PRISMA integrated service delivery network: preliminary report on methods and baseline data. In: Hébert R, Tourigny A, Raîche M, editors. Integration of services for disabled people: research leading to action. Volume II. Canada: EDISM; 2008. ISBN 978-2-89130-215-9. Chapter 11, p. 216.

PRISMA: a new model of integrated service delivery for the frail older people in Canada. R Hébert, P J Durand, N Dubuc, A Tourigny, Available from: URN:NBN:NL:UI. Hébert R, Durand PJ, Dubuc N, Tourigny A. PRISMA: a new model of integrated service delivery for the frail older people in Canada. International Journal of Integrated Care 2003(3). Available from: URN:NBN:NL:UI:10-1-100315.

Integrating services for older people: the PRISMA coordination model, implementation and impact in Quebec. Presentation. 2012, National Symposium on Integrated Care: Advancing Integrated Health Care in Canada: Practices That Work. M Raîche, R Hébert, TorontoInternetRaîche M, Hébert R, and the PRISMA group. Integrating services for older people: the PRISMA coordination model, imple- mentation and impact in Quebec. Presentation. 2012, National Symposium on Integrated Care: Advancing Integrated Health Care in Canada: Practices That Work. Toronto [Internet]. [cited 2014 Oct 10]. Available from http://www.healthcouncilcanada. ca/content.php?mnu=4&mnu1=34#.

Impact of PRISMA, a coordination-type integrated service delivery system for frail older people in Quebec (Canada): a quasi-experimental study. R Hébert, M Raîche, M-F Dubois, The Journals of Gerontology. Series B, Psychological Sciences and Social Sciences. 651Hébert R, Raîche M, Dubois M-F, et al. Impact of PRISMA, a coordination-type integrated service delivery system for frail older people in Quebec (Canada): a quasi-experimental study. The Journals of Gerontology. Series B, Psychological Sciences and Social Sciences 2010;65(1):107-18.

PRISMA-7: a case-finding tool to identify older adults with moderate to severe disabilities. M Raîche, R Hébert, M-F Dubois, Archives of Gerontology and Geriatrics. 47Raîche M, Hébert R, Dubois M-F. PRISMA-7: a case-finding tool to identify older adults with moderate to severe disabilities. Archives of Gerontology and Geriatrics 2007;47:9-18.

Integrating services for frail older people: the PRISMA coordination model implementation and impact. Presentation. Program of Research to Integrate the Services for the Maintenance of Autonomy. R Hébert, M Raîche, A Veil, Quebec, CanadaResearch Centre on Aging, University of SherbrookHébert R, Raîche M, Veil A, et al. Integrating services for frail older people: the PRISMA coordination model implementation and impact. Presentation. Program of Research to Integrate the Services for the Maintenance of Autonomy. Research Centre on Aging, University of Sherbrook, Quebec, Canada; 2009.

Frail elderly patients: new model for integrated service delivery. R Hébert, P J Durand, A Tourigny, Canadian Family Physician. 49Hébert R, Durand PJ, Tourigny A. Frail elderly patients: new model for integrated service delivery. Canadian Family Physician 2003;49:992-97.

Implementation and operation costs for an integrated services delivery network for the frail elderly in estrie. S Durand, S Blanchette, R Hébert, 2-89130-204-4Hébert R, Tourigny A, Gagnon MIIQuébec, Canada: EDISEMIntegrated service delivery to ensure person's functional autonomyDurand S, Blanchette S, Hébert R. Implementation and operation costs for an integrated services delivery network for the frail elderly in estrie. In: Hébert R, Tourigny A, Gagnon M, editors. Integrated service delivery to ensure person's functional auton- omy. Volume II. Chapter 10, Québec, Canada: EDISEM; 2008. ISBN 2-89130-204-4, pp. 199-211.

Mesures d'efficience d'un réseau intégré de services pour les personnes âgées. S Durand, Université de SherbrookeThèse de doctoratDurand S. Mesures d'efficience d'un réseau intégré de services pour les personnes âgées. Thèse de doctorat. Université de Sherbrooke; 2010.

PRISMA France: adapting the PRISMA integration model to the French health and social care system. D Somme, H Trouvé, Y Couturier, 978-2-89130-215-9Hébert R, Tourigny A, Raiche MIIQuébec, Canada: EDISMIntegration of services for disabled people: research leading to actionSomme D, Trouvé H, Couturier Y, et al. PRISMA France: adapting the PRISMA integration model to the French health and social care system. In: Hébert R, Tourigny A, Raiche M, editors. Integration of services for disabled people: research leading to action. Volume II. Québec, Canada: EDISM; 2008. ISBN 978-2-89130-215-9. Chapter 26. pp. 511-26.

Integration of services for disabled people: research leading to action. L Dieleman, P Richard, C Bureau, J Bolduc, 2-89130-204-4Canada: EDISEM. Hébert R, Tourigny A, Raîche MIIThe use of research data for management purposesDieleman L, Richard P, Bureau C, Bolduc J. The use of research data for management purposes. In: Hébert R, Tourigny A, Raîche M, editors. Integration of services for disabled people: research leading to action. Volume II. Chapter 24, Canada: EDISEM; 2008. ISBN 2-89130-204-4. Pp. 485-95.

Long-term care for the elderly in Quebec: considerations of appropriateness and quality of care. H A Palley, M-P Pomey, M-J Fleury, Occasional Papers on Public Policy. 41Palley HA, Pomey M-P, Fleury M-J. Long-term care for the elderly in Quebec: considerations of appropriateness and quality of care. Occasional Papers on Public Policy 2012;4(1):1-7.

Family physicians' perception of the integrated services network and case managers. L Voyer, R Hébert, 978-2-89130-215-9Canada: EDISM. Hébert R, Tourigny A, Raîche MIIIntegration of services for disabled people: research leading to actionVoyer L, Hébert R. Family physicians' perception of the integrated services network and case managers. In: Hébert R, Tourigny A, Raîche M, editor. Integration of services for disabled people: research leading to action. Volume II. Canada: EDISM; 2008. ISBN 978-2-89130-215-9. Chapter 8. pp. 167-78.

Five laws for integrating medical and social services: lessons from the United States and The United Kingdom. W Leutz, The Millbank Quarterly. 771Leutz W. Five laws for integrating medical and social services: lessons from the United States and The United Kingdom. The Millbank Quarterly 1999;77(1):77-110.

Progress toward integrating care for seniors in Canada. M Macadam, 11. Available from: URN:NBN:NL:UI. MacAdam M. Progress toward integrating care for seniors in Canada. International Journal of Integrated Care 2011:11. Available from: URN:NBN:NL:UI:10-1-101378.

Providing integrated care for older people with complex needs. Lessons from seven international case studies: The King's Fund. N Goodwin, A Dixon, G Anderson, W Wodchis, Goodwin N, Dixon A, Anderson G, Wodchis W. Providing integrated care for older people with complex needs. Lessons from seven international case studies: The King's Fund 2014.