BMC Musculoskeletal Disorders Efficiency of immediate postoperative inpatient physical therapy following total knee arthroplasty: an RCT

Anton F Lenssen email@example.com
Yvonne †1
Department of Physical Therapy
University Hospital Maastricht
MaastrichtThe Netherlands

Crijns Hf
Eddie †2
Department of Physical therapy
University Hospital Maastricht
The NetherlandsMaastricht

Waltjé Mh
Mike Ja †2
Department of Physical therapy
University Hospital Maastricht
The NetherlandsMaastricht

Van Steyn
Department of Orthopaedics
University Hospital Maastricht
MaastrichtThe Netherlands

Ruud Jt
Geesink †3
Department of Orthopaedics
University Hospital Maastricht
MaastrichtThe Netherlands

Piet A Van Den Brandt
Department of Epidemiology
Maastricht University
MaastrichtThe Netherlands

Rob A De Bie
Department of Epidemiology
Maastricht University
MaastrichtThe Netherlands

BMC Musculoskeletal Disorders Efficiency of immediate postoperative inpatient physical therapy following total knee arthroplasty: an RCT
10.1186/1471-2474-7-71BioMed Central Page 1 of 9 (page number not for citation purposes) Research article
Background:The main goal of physical therapy treatment (PT) in the clinical stage following total knee arthroplasty (TKA) is to prepare patients for discharge from the hospital as soon as possible after their operation. Although aggressive rehabilitation is believed to be important, evidence of effects of different exercise programmes following TKA is limited. This led to the question whether the intensity of PT (once versus twice daily) following TKA affects short-term recovery, measured as range of motion.Methods: A randomised controlled trial compared an exercise regimen of two sessions per day with a similar programme administered once daily. Primary outcome measure was ROM.Results: At the time of hospital discharge, there was no difference between the experimental and control groups in range of motion.Conclusion:This study shows that in our setting twice daily PT sessions do not produce different results as daily PT sessions. It may be questioned whether multiple daily therapy sessions are needed as an in-hospital PT regimen in OA total knee patients.

Background

Recent trends in the treatment of patients following orthopaedic surgery in the Netherlands encourage early discharge from hospitals, which implies increasing the intensity of treatment during the in-hospital phase. The strategy of choice involves mobilisation of the joints, starting on the day of surgery as well as shortening the bed rest phase and early ambulation. Physical therapy treatment (PT) plays an important role in this strategy.

In 2000 a 'Joint Care' programme was implemented at the University Hospital Maastricht. This is a clinical pathway in which the activities of physicians, nurses, physical therapists and other staff inside as well as outside the hospital are coordinated to provide the best overall care for patients with a particular diagnosis or procedure [1].

Clinical pathways are used in many institutions for patients undergoing total knee arthroplasty (TKA) and total hip arthroplasty (THA). These procedures lend themselves very well to a pathway approach, because the process of care is relatively standardized. Clinical pathways have been successful in reducing costs and length of stay in acute care hospitals, while not compromising patient outcomes [1][2][3][4][5][6][7]. The contents of the individual treatments are hospital-specific to create an optimal regimen of care tailored to that specific institution. The main goal in the clinical stage is to prepare patients for discharge from the hospital as soon as possible after their operation. Discharge depends primarily on restoring sufficient functional range of motion (ROM) [8].

Various authors [9][10][11][12] have described aggressive rehabilitation schemes incorporating more than one PT treatment session per day in the clinical phase following total knee surgery. Aggressive rehabilitation is believed to be important in preventing postoperative contracture of the soft tissue and in gaining better flexion. Nevertheless evidence of effects of different exercise programs following TKA is limited [13]. Our current PT programme is based on these rehabilitation schemes, as well as on the outcome of a nonrandomised controlled trial [14] comparing usual care with a 'Joint Care' clinical pathway. This trial found the length of hospital stay to be significantly shorter in the 'Joint Care' group. The author concluded that this was mainly caused by the greater stimulation to exercise and the number of minutes of PT per day. Patients were not only stimulated by their physical therapists to exercise, but the nursing staff also stimulated the patients to exercise and perform daily life activities themselves. Based on this result we doubled the time spent on in-hospital PT from 20 minutes to 40 minutes per patient per day, thus increasing treatment intensity to fulfil one of the goals of the programme; shortening hospital stay to four days after surgery. Patients are scheduled for discharge the fourth day after surgery if their passive flexion ROM exceeds 65°, when they are able to transfer from sitting to standing, walk independently, and are able to climb stairs, if necessary.

After two years of experience with the high intensity approach, the mean hospital stay was reduced to approximately four days after the operation. In 2003, however budget considerations meant that continuation of the extra PT sessions was being reconsidered. This led to a debate within the PT department whether or not the extra time investment should be maintained, i.e. whether the second daily visit was essential in reaching the short-term PT goals and whether having only one session of PT a day would lead to poorer range of motion (ROM) and function.

This resulted in the following research question. Does the intensity of PT treatment (once versus twice daily) in patients following TKA affect their short-term recovery, measured as range of motion?

To answer this question, a randomised controlled in-hospital trial was conducted in which a regimen consisting of four days with 40 minutes of PT a day was compared with one involving 20 minutes of PT a day.

Methods

The medical ethics committee of the Maastricht University Hospital and Maastricht University approved this prospective, randomised controlled trial.

The design of the trial is depicted in figure 1.

Subjects

A consecutive series of patients who received a primary TKA at the University Hospital Maastricht, the Netherlands, was invited to participate between 1 January and 1 June, 2004. Subjects were considered eligible for the study if they were scheduled in the 'Joint Care' programme and signed an informed consent form.

Patients undergoing knee revision surgery were excluded as were patients over 85 years of age, patients with coresearch design and outcome assessment Figure 1 research design and outcome assessment. T0 baseline assessment, two weeks before surgery, T1 assessment 4 days after surgery, T2 assessment 6 weeks after surgery, T3 assessment 3 months after surgery. R = Randomisation. morbidity influencing gait and patients who did not understand or speak Dutch.

Randomisation

An independent research assistant performed concealed randomization using a computer generated randomisation schedule in which patients were allocated according to a weekly treatment regimen. All patients are admitted into hospital on Monday evening, surgery is on Tuesdays and hospital leave is planned the next Saturday. By randomising clusters of treatment weeks instead of individual randomisation we made sure that patients within the same treatment week all received the same postoperative treatment regime, thus avoiding contamination, which might occur when patients receive different regimes in the same week of admission at the same ward.

Information on the particular treatment intensity was given on the day of surgery. In this way we prevented foreknowledge of treatment assignment and thus shielded those who enrolled participants from being influenced by this knowledge because enrolment into the study was conducted two weeks before surgery.

Interventions

Clinical phase

During the in-hospital phase daily PT was administered to all participating patients. Patients in the experimental group were treated twice daily (totalling 40 minutes per day), patients in the control group once daily (20 minutes per day). The treatment content was similar in both groups and consisted of active and passive mobilisation of the knee, strengthening of the quadriceps muscle and functional exercises including transfers from a supine position to sitting and from sitting to standing, walking and stair climbing. As a rule, patients left hospital after the morning PT sessions on the fourth day after surgery, leading to a total number of PT sessions in the experimental group of seven sessions compared to four sessions in the control group.

A detailed description of the treatment protocol following total knee surgery is available at the website of the Physical Therapy department of the Maastricht University Hospital[15] We divided the treating physical therapists into two groups. One group treated all patients in the morning, while the optional second treatment sessions were administered by other physical therapists, leaving the group of therapists who treated the patients in the morning sessions unaware whether or not patients were receiving a second PT session. Therapist received weekly work schedules. During the study period therapist shifted from morning to afternoon groups randomly over treatment weeks. These changes were introduced to balance the work load of the participating therapist rather than for study purposes.

Hospital discharge followed when patients showed wound healing, they were ambulant using a walking frame or crutches and passive flexion ROM of the knee was above 65 degrees.

Outcome measures

Primary outcome measure in this study was passive flexion ROM.

Secondary outcome measures were active ROM and passive extension ROM, functional status, length of stay (LOS) after surgery, pain over the last 24 hours (using an 11 point scale), satisfaction with treatment (using an 11 point scale from totally dissatisfied to completely satisfied) [16], satisfaction with the intermediate treatment results and global perceived effect.

ROM was measured actively as well as passively using a large goniometer following the method described by Brosseau [17]. Intratester reliability for knee flexion is 0.99, for active extension 0.97. Criterion validity for knee flexion 0.98, for extension 0.42 [17].

Functional status was measured using two scales, the disease specific Western Ontario and McMaster University Osteoarthritis index (WOMAC) [18] and the joint specific Knee Society Scale (KSS) [19].

The WOMAC is a disease specific questionnaire developed specifically for people with osteoarthritis of the hip and knee. Using visual analog scales, its 24 items probe 3 dimensions -pain (5 items), stiffness (2 items), and functional difficulty (17 items). Scale sum scores were standardised (0-100), with high values indicating less pain or better physical functioning [20].

The WOMAC questionnaire is well recognized for its good validity, reliability and responsiveness. We used the Dutch version of WOMAC [20].

The KSS is concise and easy to use. It represents a clear attempt to separate knee function from overall patient functional status. Bach et al [21] reported that reproducibility of the knee score is poor whilst the function score shows good reproducibility. The construct validity of the KSS is good [22].

At follow-up, patients were asked to judge the effect of the surgery on a 7-point likert scale from 'worse than ever' status to 'completely recovered' [23].

The primary effect measurement was scheduled for the fourth day after surgery, while follow-up measurements were scheduled at 6 weeks and 3 months after surgery. Table 1 shows the timing and measurement tools used. The outcome assessor was blinded with regard to the a clinical intervention regime administered.

We selected the KSS score as functional outcome on day four because several items in the WOMAC are not applicable during the in-hospital period. The WOMAC was assessed during the outpatient period because psychometric properties of the WOMAC are superior to the KSS.

Sample size

A 10° of passive ROM difference between groups was considered clinically important enough to eventually raise treatment frequency. The sample size needed to detect a significant difference (2-sided, α <. 05) of at least 10°R OM between groups (primary outcome) with an assumed standard deviation (SD) of 10° and a power of 0.8, was 16 subjects per group. To compensate for a maximally acceptable 20% loss of subjects between baseline and follow-up, 20 subjects per group (40 in total) were required.

Analyses

Data were checked for completeness and normality. Statistical analyses were blinded and performed according to the 'intention-to-treat' principle. Descriptive statistics were calculated of both group statuses at baseline. Post treatment scores as well as differences between pre-and post treatment scores were compared with regard to all outcome measures. Group differences and 95% confidence intervals (CI) were computed for all outcome measures. The scores on global perceived effect were dichotomized to examine the ratio of improved versus unimproved patients. Patients were scored as improved if they felt much better or had recovered completely. Student's t-test was used for continuous data to determine differences between the two treatment groups. Chi-square tests were used for analyses of categorical data and when continuous data were not normally distributed. All data were analysed using SPSS version 11.0 [24].

Results

Of the 55 subjects who were scheduled for surgery between January and June 2004, 43 were included in the study. 9 patients refused to participate and 3 were excluded because they had relevant co-morbidities. Twenty-one patients were randomly assigned to the experimental group, 22 to the control group.

The baseline characteristics of the patients in the two groups were similar in terms of clinical and demographic characteristics. They also showed comparable levels of functional ability and Quality of Life (table 2).

All 43 patients were assessed two weeks before surgery (T0) and four days after surgery (T1).

Six weeks after the operation (T2) 2 patients of the control group failed to attend the follow-up measurement, thus data from 41 patients were available for intention to treat analyses. All participants attended the follow-up measurements at 3 months.

Outcome 4 days after surgery

As can be seen from table 3, there was no detectable difference in the primary outcome measure ROM between the experimental and control groups four days after surgery. Patients in the control group all received 4 treatment sessions. Two patients in the experimental group received six of the seven possible treatment sessions. One of these Flexion contracture was about 8° in both groups and active flexion ROM was about 70° Functional status results on day four were also similar between groups.

Six-weeks and three-months follow-up

There were no significant or clinically relevant differences in the secondary outcome measures at any of the other follow-up moments (table 4). Follow-up data were comparable to those at day four after surgery. ROM in extension seemed to be a little better in the control group, as was satisfaction with the overall treatment. Recovery of flexion range of motion as well as functional status is similar in both groups.

Discussion

This trial did not detect any differences in the effects on any of the outcome parameters between regimens involving one or two physical therapy sessions a day in the clinical phase following TKA.

Remarkably, patients seemed to be very satisfied with PT treatment, irrespective of the number of sessions. Three months after surgery, the perceived effect of the total treatment was high, with 39 out of 43 patients stating that they had vastly improved or were completely cured.

De Jong [14] reported similar numbers of PT sessions and comparable hospital stay (4.97 days) after surgery com-pared to our experimental group (4.1 days). De Jong did not report on ROM and functional capacity was measured using a scale of unknown reliability and validity.

Since we did not find any other studies on the effects of different PT treatment intensities following TKA, it is only possible to compare the results of the total group with those of other studies with known treatment intensity.

Most studies on rehabilitation after TKA have focussed on the use of continuous passive motion treatment.

Several recent studies [9,[25][26][27] incorporated high intensity basic PT rehabilitation programmes and describe outcomes comparable to our data. Two studies employed standardized PT treatments twice daily, whilst Kumar et al. reported 90 minutes to two hours of PT daily and Beaupre et al. did not standardize the treatment duration but reported on six treatment sessions during the first four days after surgery.

Three [9,25,26] studies described short-term results on range of motion.

The ROM we found four days after surgery is comparable to the values reported in these studies. The level of pain found in our study is in agreement with those reported by Bennett whereas the length of stay in Bennett's sample was longer (8.8 compared to 5 days), and KSS scores at 3months follow-up were slightly better in our sample (mean KSS function score of 52 versus 69).

Our study only collected Womac data preoperative and at six-weeks and three-months follow-up. No WOMAC data were collected at hospital discharge because several items of the WOMAC score are not valid during hospital stay, items like getting in and out of a car, shopping, getting in   and out of a bath and doing domestic chores are not applicable in this phase.

Study limitations

We chose cluster randomization of treatment weeks over individual randomization. By clustering in treatment weeks we made sure that all patients operated in a week were receiving the same amount of daily treatment sessions. We believe that this led to less bias than introduced by contamination in individual randomization. Patients receive PT treatment in the 'living room' of the orthopaedic ward in which they stay during daytime. We believe contamination is likely to occur when one patient would receive a second treatment in the afternoon whilst the patient sitting next to him/her is not. Conversation is overheard and just seeing treatment might encourage the patient to exercise them self, therewith diminishing contrast between both groups.

Though randomizing in clusters theoretically might inflict bias, we believed in this case it does not have a lot of influence. It is accepted that clustering may result in P values and confidence intervals which are sufficiently biased to have a major effect if any of the following are true: the cluster size is large, the number of clusters is small, or the intra-cluster correlation coefficient is large [28]. Our study contained 22 clusters, cluster sizes were small with a mean of two patients per cluster and intracluster correlation was very small, (icc = 0,171), leading up to a design effect of 1.171 [29].

In combination with the fact that clusters were merely different weeks in a four months period in the same hospital with the same staff etc. makes us believe that clustering did not introduce a major bias in our study.

We chose passive flexion ROM as primary outcome measure because passive ROM is the only measure of function involved in hospital discharge in our situation. Therefore we based our sample size calculation on between group differences in passive flexion ROM.

When designing the study the therapists and orthopaedic surgeons involved discussed what effect would justify the effort of extra PT sessions daily. We had no information on possible effect sizes or minimally important clinical differences out of the literature, since we are unaware of a single similar study in this field. However we had some information on the effects of CPM in this same setting [30]. In this study we found an 8 degrees difference in flexion ROM when using CPM in the in hospital phase after total knee arthroplasty.

We agreed on a 10 degrees between group difference to be a result worth the extra PT effort. Although we did not detect a between group difference at all, one may state that our study was designed to find a large difference. Sample size was calculated to find this effect and therefore may be to small to detect smaller differences. Our study is underpowered to detect significantly smaller between group differences. For example, to detect a 5 degree difference sample size would a sample size of over 120 participants.

Although our study could not detect differences between groups receiving PT once or twice daily, this does not mean that individual patients might not benefit from more intensive PT. One of the primary issues in treating patients with TKA is to identify those patients that may require more intensive rehabilitation. Unfortunately, secondary analyses yielded no variables at baseline that predicted short-term postoperative recovery.

Different therapists treated patients in the morning and afternoon sessions as well as over different treatment days. This is a common procedure in our hospital which in itself might increase treatment variation. However, treatments were standardized and applied following a protocol.

In this study, PT treatment was standardized in terms of content as well as duration of the treatment sessions.

Although almost every study in this field has done the same, one might question whether this is the appropriate way to handle patient care. It might be wiser to set treatment goals per day. As soon as the target for that day has been achieved no further treatment might be necessary.

Flow chart of subjects through the trial Figure 2 Flow chart of subjects through the trial. This would mean that the number of treatments can be individualized according to the goals set for each day.

Although blinding therapists is very difficult, we attempted to blind at least the physical therapists attending the morning session. Blinding was not successful in every case, because some patients gave information about the second session during the morning sessions.

Participating physical therapists stated that not knowing whether a patient would receive a second treatment session probably raised their awareness about achieving clinical goals during the morning sessions. Although they were aware that this attitude might bias study results, their attitude towards giving optimal patient care overruled the urge to follow the treatment protocol. In this sense one might say that treating patients once daily is just as good as twice daily if the physical therapists are more focussed.

Conclusion

This study shows that in our setting twice daily PT sessions do not produce different results as daily PT sessions. It may be questioned whether multiple daily therapy sessions are needed as an in-hospital PT regimen in OA total knee patients.

Publish with Bio Med Central and every scientist can read your work free of charge"BioMed Central will be the most significant development for disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK Your research papers will be: available free of charge to the entire biomedical community peer reviewed and published immediately upon acceptance cited in PubMed and archived on PubMed Central yours -you keep the copyright Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.

Table 1 :
1outcome measures and timing ROM = Range of Motion patients did not feel well enough to participate in a second PT session on the second day after surgery. The other patient left the hospital on the third day after surgery. The surgeon approved his discharge since he had reached the short-term treatment goals and asked to return home for personal reasons. Length of stay was similar in both groups. One patient in the control group was transferred to the cardiology department because of cardiac complaints. He left the hospital 15 days after surgery.T0
T1
T2
T3

Table 2 :
2patient's baseline characteristicsVariable
Experimental group n = 21
Control group n = 22

Female/Male
15/6
17/5
Mean
sd
age
70
(8.5)
67
(7)
active ROM extension (degrees)
8°(6°)
8°(5°)
passive ROM extension (degrees)
5°(5°)
5°(5°)
active ROM flexion (degrees)
106°(19°)
102°(20°)
passive ROM flexion (degrees)
110°(19°)
114°(15°)
functional status WOMAC
44
(16.7)
40
(16.6)
WOMAC subscale pain
9.2
(2.7)
8.3
(3.5)
WOMAC subscale stiffness
3.7
(1.1)
3.3
(1.4)
WOMAC subscale difficulty
31.1
(9.3)
28.2
(11.8)
functional status KSS knee
49
(18)
43
(21)
KSS function
53
(18)
52
(21)
Pain (11 point scale)
3,7
(2,8)
3,5
(3,3)

Table 3 :
3between group differences at day 4 after surgery (scheduled hospital discharge) T1 and differences between status 14 days prior to surgery and 4 days after surgeryVariable
Experimental group
n = 21

Control group
n = 22

Mean
difference

95%CI
P-value Differences between t0 and t1
Mean
difference

95%CI
P-value

Experimental
group

Control
group
Mean
sd
Mean
sd
Mean
sd
Mean
sd

ROM active extension (degrees)
8.4
5.1
7.8
5.6
0.56
-2.7 -3.8
0.73
0.71
8.6
0.14
5.2
0.57
-3.9 -5.0
0.46

ROM passive extension (degrees)
5.5
4.5
5.8
4.9
-0.3
-3.2 -2.6
0.81
0.1
7.2
0.4
4.7
-0.3
-4.2 -3.2
0.78

Rom active flexion (degrees)
69.8
11.9
70.4
13.3
-0.5
-8.3 -7.3
0.90
36.1
22.0
31.5
17.8
4.6
-7.8 -16.9
0.79

ROM passive flexion (degrees)
77.7
11.2
77.9
12.8
-0.2
-7.7 -7.2
0.95
36.6
17.9
32.1
18.4
4.5
-6.8 -15.6
0.43

Functional status KSS knee
54.3
15.2
52.2
17.8
1.1
-8.7 -12.3
0.68
9.3
22.7
4.4
26.3
4.9
-20 -10.2
0.52

Functional status KSS function
29.3
10.7
22.9
13.2
-6.4
-1.1 -13.8
0.09
-24
23.4
-29
19.3
-5
-18 -7.9
0.43

Length of stay
4.1
0.9
4.5
1.3
0.4
-0.3 -1.0
0.34
-
-
-
-

Pain (11 point scale)
3.6
2.3
3.6
2.8
0
-1.6 -1.6
0.97
-0.14
3.1
0.1
3.5
0.04
-1.8 -2.3
0.82

Satisfaction with treatment (11-point scale)
9.1
0.8
9.1
0.7
0
-0.5 -0.5
0.99
-
-
-
-

Table 4 :
4Six weeks and three months follow-up6 weeks

(page number not for citation purposes)
AcknowledgementsWe thank all our co workers of the Orthopaedic and Physical therapy department for their support during the trial. We thank all the patients who participated in the study for their valuable input.Competing interestsThe author(s) declare that they have no competing interests.Authors' contributionsAFL participated in the design of the study, exercise intervention, assessments and follow-ups, statistical analyses and writing. YHFC, EMHW, MJAvS and RAdB participated in the study design and the interpretation of the data, and critically revised the article. RGTG and PAvdB participated in the interpretation of the data and critically revised the article. All authors read and approved the final manuscript.Pre-publication historyThe pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2474/7/71/prepub
The effect of a perioperative clinical pathway for knee replacement surgery on hospital costs. A Macario, M Horne, S Goodman, T Vitez, F Dexter, R Heinen, B Brown, Anesth Analg. 86Macario A, Horne M, Goodman S, Vitez T, Dexter F, Heinen R, Brown B: The effect of a perioperative clinical pathway for knee replacement surgery on hospital costs. Anesth Analg 1998, 86:978-984.

Clinical pathways in total knee arthroplasty: a New Zealand experience. J M Pennington, D P Jones, S Mcintyre, J Orthop Surg. 11Pennington JM, Jones DP, McIntyre S: Clinical pathways in total knee arthroplasty: a New Zealand experience. J Orthop Surg (Hong Kong) 2003, 11:166-173.

Clinical pathway management of total knee arthroplasty. J D Mabrey, J S Toohey, D A Armstrong, L Lavery, L A Wammack, Clin Orthop Relat Res. Mabrey JD, Toohey JS, Armstrong DA, Lavery L, Wammack LA: Clin- ical pathway management of total knee arthroplasty. Clin Orthop Relat Res 1997:125-133.

Total knee arthroplasty clinical pathway. K Harlan, A Meiring, Hosp Case Manag. 7Harlan K, Meiring A: Total knee arthroplasty clinical pathway. Hosp Case Manag 1999, 7:49-53.

Impact of cost reduction programs on short-term patient outcome and hospital cost of total knee arthroplasty. W L Healy, R Iorio, J Ko, D Appleby, D W Lemos, J Bone Joint Surg Am. Healy WL, Iorio R, Ko J, Appleby D, Lemos DW: Impact of cost reduction programs on short-term patient outcome and hospital cost of total knee arthroplasty. J Bone Joint Surg Am 2002, 84-A:348-353.

Clinical pathway management of total knee arthroplasty: a retrospective comparative study. S Pearson, I Moraw, G J Maddern, Aust N Z J Surg. 70Pearson S, Moraw I, Maddern GJ: Clinical pathway management of total knee arthroplasty: a retrospective comparative study. Aust N Z J Surg 2000, 70:351-354.

Effectiveness of clinical pathways for total knee and total hip arthroplasty: literature review. S Kim, E Losina, D H Solomon, J Wright, J N Katz, J Arthroplasty. 18Kim S, Losina E, Solomon DH, Wright J, Katz JN: Effectiveness of clinical pathways for total knee and total hip arthroplasty: lit- erature review. J Arthroplasty 2003, 18:69-74.

Hospital stay and discharge outcomes after knee arthroplasty: implications for physiotherapy practice. L B Oldmeadow, H Mcburney, V J Robertson, Aust J Physiother. 48Oldmeadow LB, McBurney H, Robertson VJ: Hospital stay and dis- charge outcomes after knee arthroplasty: implications for physiotherapy practice. Aust J Physiother 2002, 48:117-121.

A comparison of 2 continuous passive motion protocols after total knee arthroplasty: a controlled and randomized study. L A Bennett, S C Brearley, J A Hart, M J Bailey, J Arthroplasty. 20Bennett LA, Brearley SC, Hart JA, Bailey MJ: A comparison of 2 continuous passive motion protocols after total knee arthro- plasty: a controlled and randomized study. J Arthroplasty 2005, 20:225-233.

Effect of adjunctive range-of-motion therapy after primary total knee arthroplasty on the use of health services after hospital discharge. D M Davies, D W Johnston, L A Beaupre, D A Lier, Can J Surg. 46Davies DM, Johnston DW, Beaupre LA, Lier DA: Effect of adjunc- tive range-of-motion therapy after primary total knee arthroplasty on the use of health services after hospital dis- charge. Can J Surg 2003, 46:30-36.

Relationship between length of stay and manipulation rate after total knee arthroplasty. D R Mauerhan, J G Mokris, A Ly, G M Kiebzak, J Arthroplasty. 13Mauerhan DR, Mokris JG, Ly A, Kiebzak GM: Relationship between length of stay and manipulation rate after total knee arthroplasty. J Arthroplasty 1998, 13:896-900.

Primary total knee arthroplasty using the Genesis Total Knee Arthroplasty System: 3-to 6-year follow-up study of 105 knees. J G Mokris, S W Smith, S E Anderson, J Arthroplasty. 12Mokris JG, Smith SW, Anderson SE: Primary total knee arthro- plasty using the Genesis Total Knee Arthroplasty System: 3- to 6-year follow-up study of 105 knees. J Arthroplasty 1997, 12:91-98.

Effectiveness and practice variation of rehabilitation after joint replacement. E M Roos, Curr Opin Rheumatol. 15Roos EM: Effectiveness and practice variation of rehabilita- tion after joint replacement. Curr Opin Rheumatol 2003, 15:160-162.

Reactivatie van een THA patient volgens een óud'of een 'nieuw'protocol. J A Jong De, Maastricht University, Faculty of Health SciencesMSc thesisJong de JA: Reactivatie van een THA patient volgens een óud'of een 'nieuw'protocol. MSc thesis, Maastricht University, Fac- ulty of Health Sciences 1999.

Patient satisfaction after total knee arthroplasty: a comparison between subjective and objective outcome assessments. P H Bullens, C J Van Loon, M C De Waal Malefijt, R F Laan, R P Veth, J Arthroplasty. 16Bullens PH, van Loon CJ, de Waal Malefijt MC, Laan RF, Veth RP: Patient satisfaction after total knee arthroplasty: a compar- ison between subjective and objective outcome assess- ments. J Arthroplasty 2001, 16:740-747.

Intra-and intertester reliability and criterion validity of the parallelogram and universal goniometers for measuring maximum active knee flexion and extension of patients with knee restrictions. L Brosseau, S Balmer, M Tousignant, J P O'sullivan, C Goudreault, M Goudreault, S Gringras, Arch Phys Med Rehabil. 82Brosseau L, Balmer S, Tousignant M, O'Sullivan JP, Goudreault C, Goudreault M, Gringras S: Intra-and intertester reliability and criterion validity of the parallelogram and universal goniom- eters for measuring maximum active knee flexion and extension of patients with knee restrictions. Arch Phys Med Rehabil 2001, 82:396-402.

Validation study of the WOMAC: a health status instrument for measuring clinically important patient relevant outcomes following total hip or knee arthroplasties in osteoarthritis. N Bellamy, W W Buchanan, C H Goldsmith, J Campbell, L W Stit, J Orthop Rheumatology. 1Bellamy N, Buchanan WW, Goldsmith CH, Campbell J, Stit LW: Val- idation study of the WOMAC: a health status instrument for measuring clinically important patient relevant outcomes following total hip or knee arthroplasties in osteoarthritis. J Orthop Rheumatology 1988, 1:95-108.

Rationale of the Knee Society clinical rating system. J N Insall, L D Dorr, R D Scott, W N Scott, Clin Orthop Relat Res. Insall JN, Dorr LD, Scott RD, Scott WN: Rationale of the Knee Society clinical rating system. Clin Orthop Relat Res 1989:13-14.

Suarez-Almazor ME: Satisfactory cross cultural equivalence of the Dutch WOMAC in patients with hip osteoarthritis waiting for arthroplasty. L D Roorda, C A Jones, M Waltz, G J Lankhorst, L M Bouter, J W Van Der Eijken, W J Willems, I C Heyligers, D C Voaklander, K D Kelly, Ann Rheum Dis. 63Roorda LD, Jones CA, Waltz M, Lankhorst GJ, Bouter LM, van der Eijken JW, Willems WJ, Heyligers IC, Voaklander DC, Kelly KD, Sua- rez-Almazor ME: Satisfactory cross cultural equivalence of the Dutch WOMAC in patients with hip osteoarthritis waiting for arthroplasty. Ann Rheum Dis 2004, 63:36-42.

Scoring systems in total knee arthroplasty. C M Bach, M Nogler, I E Steingruber, M Ogon, C Wimmer, G Gobel, M Krismer, Clin Orthop Relat Res. Bach CM, Nogler M, Steingruber IE, Ogon M, Wimmer C, Gobel G, Krismer M: Scoring systems in total knee arthroplasty. Clin Orthop Relat Res 2002:184-196.

Validity and responsiveness of the Knee Society Clinical Rating System in comparison with the SF-36 and WOMAC. E A Lingard, J N Katz, R J Wright, E A Wright, C B Sledge, J Bone Joint Surg Am. Lingard EA, Katz JN, Wright RJ, Wright EA, Sledge CB: Validity and responsiveness of the Knee Society Clinical Rating System in comparison with the SF-36 and WOMAC. J Bone Joint Surg Am 2001, 83-A:1856-1864.

A comparison of Likert and visual analogue scales for measuring change in function. G H Guyatt, M Townsend, L B Berman, J L Keller, J Chronic Dis. 40Guyatt GH, Townsend M, Berman LB, Keller JL: A comparison of Likert and visual analogue scales for measuring change in function. J Chronic Dis 1987, 40:1129-1133.

M J Norusis, SPSS 9.0 guide tot data analyses. SPSS IncNorusis MJ: SPSS 9.0 guide tot data analyses. Chicago: SPSS Inc.1999. .

Exercise combined with continuous passive motion or slider board therapy compared with exercise only: a randomized controlled trial of patients following total knee arthroplasty. L A Beaupre, D M Davies, C A Jones, J G Cinats, Phys Ther. 81Beaupre LA, Davies DM, Jones CA, Cinats JG: Exercise combined with continuous passive motion or slider board therapy com- pared with exercise only: a randomized controlled trial of patients following total knee arthroplasty. Phys Ther 2001, 81:1029-1037.

Rehabilitation after total knee arthroplasty: a comparison of 2 rehabilitation techniques. P J Kumar, E J Mcpherson, L D Dorr, Z Wan, K Baldwin, Clin Orthop Relat Res. Kumar PJ, McPherson EJ, Dorr LD, Wan Z, Baldwin K: Rehabilita- tion after total knee arthroplasty: a comparison of 2 rehabil- itation techniques. Clin Orthop Relat Res 1996:93-101.

Prospective randomized clinical trial of continuous passive motion after total knee arthroplasty. S J Macdonald, R B Bourne, C H Rorabeck, R W Mccalden, J Kramer, M Vaz, Clin Orthop Relat Res. MacDonald SJ, Bourne RB, Rorabeck CH, McCalden RW, Kramer J, Vaz M: Prospective randomized clinical trial of continuous passive motion after total knee arthroplasty. Clin Orthop Relat Res 2000:30-35.

Cluster randomised trials in the medical literature: two bibliometric surveys. J M Bland, BMC Med Res Methodol. 421Bland JM: Cluster randomised trials in the medical literature: two bibliometric surveys. BMC Med Res Methodol 2004, 4:21.

CONSORT statement: extension to cluster randomised trials. M K Campbell, D R Elbourne, D G Altman, Bmj. 328Campbell MK, Elbourne DR, Altman DG: CONSORT statement: extension to cluster randomised trials. Bmj 2004, 328:702-708.

Steijn van MJA: Continuous passive motion in rehabilitation following total knee arthroplasty: a Randomised controlled trial. A F Lenssen, R A Bie De, S K Bulstra, Physical Therapy Reviews. 8Lenssen AF, Bie de RA, Bulstra SK, Steijn van MJA: Continuous pas- sive motion in rehabilitation following total knee arthro- plasty: a Randomised controlled trial. Physical Therapy Reviews 2003, 8:123-129.


Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and User Survey

Sarah L Gorst *email@example.com
Department of Biostatistics
University of Liverpool
LiverpoolUnited Kingdom

Elizabeth Gargon
Department of Biostatistics
University of Liverpool
LiverpoolUnited Kingdom

Mike Clarke
Northern Ireland Network for Trials Methodology Research
Queen's University Belfast
BelfastUnited Kingdom

Jane M Blazeby
School of Social and Community Medicine
Division of Surgery, Head & Neck
University of Bristol
University Hospitals NHS Foundation Trust
BristolUnited Kingdom

Douglas G Altman
Centre for Statistics in Medicine
University of Oxford
OxfordUnited Kingdom

Paula R Williamson
Department of Biostatistics
University of Liverpool
LiverpoolUnited Kingdom

Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and User Survey
10.1371/journal.pone.0146444RESEARCH ARTICLE
BackgroundA COS represents an agreed minimum set of outcomes that should be measured and reported in all trials of a specific condition. The COMET (Core Outcome Measures in Effectiveness Trials) initiative aims to collate and stimulate the development and application of COS, by including data on relevant studies within a publically available internet-based resource. In recent years, there has been an interest in increasing the development of COS. Therefore, this study aimed to provide an update of a previous review, and examine the quality of development of COS. A further aim was to understand the reasons why individuals are searching the COMET database.MethodsA multi-faceted search strategy was followed, in order to identify studies that sought to determine which outcomes/domains to measure in clinical trials of a specific condition. Additionally, a pop up survey was added to the COMET website, to ascertain why people were searching the COMET database.ResultsThirty-two reports relating to 29 studies were eligible for inclusion in the review. There has been an improvement in the description of the scope of a COS and an increase in the proportion of studies using literature/systematic reviews and the Delphi technique. Clinical experts continue to be the most common group involved in developing COS, however patient and public involvement has increased. The pop-up survey revealed the most common reasons for visiting the COMET website to be thinking about developing a COS and planning a clinical trial.Competing Interests: MC, JMB, DGA, and PRW are members of the COMET Management Group and coapplicants on grants to support COMET and related work. EG is a member of the COMET ManagementConclusionsThis update demonstrates that recent studies appear to have adopted a more structured approach towards COS development and public representation has increased. However, there remains a need for developers to adequately describe details about the scope of COS, and for greater public engagement. The COMET database appears to be a useful resource for both COS developers and users of COS.Choosing Important Health Outcomes: Updated Review and User Survey PLOS ONE |

Introduction

When designing clinical trials, it is important to measure appropriate outcomes, so that the results can be compared with other trials and will be as useful as possible to decision makers. At present, many studies which explore the effects of the same intervention on a specific health condition measure and report different outcomes, making it difficult to compare, contrast or combine their findings. This causes problems for people trying to use healthcare research. Inconsistency in outcome measurement and outcome reporting bias have led to avoidable waste in the production and reporting of research [1,2]. However, these issues could be reduced through the development and application of core outcome sets (COS). A COS represents an agreed minimum set of outcomes that should be measured and reported in all trials of a specific condition [3]. The existence of a COS allows the results of trials to be brought together as appropriate and ensures that all trials provide usable evidence. COS are not intended to restrict the number of outcomes in a particular trial, rather, the intention is that the outcomes in the COS will always be collected and reported, and it is fully expected that researchers will continue to explore additional outcomes [3].

The COMET (Core Outcome Measures in Effectiveness Trials) initiative aims to collate and stimulate the development, application and promotion of COS, by including data on relevant individual studies within a publically available internet-based resource. The COMET database is a unique inventory that includes published accounts of COS development, as well as planned and ongoing work. Initially, the database included studies that had been identified by ad hoc means and a systematic review which addressed outcome selection in clinical trials in children [4]. In order to bring the database content up-to-date and be comprehensive, a systematic approach was needed to identify relevant material. Gargon and colleagues [5] conducted a systematic review in 2013, the first comprehensive search for COS in health research, and identified 198 studies that determined which outcomes or domains should be measured in all clinical trials for a specific health condition. The review revealed wide variation in the methods used to develop COS, and highlighted the need for methodological guidance, including how to engage key stakeholder groups, particularly members of the public, in the development and implementation of COS. When using the term 'public' through this report we include patients, carers, health and social care service users and people from organisations who represent these groups [6]. The involvement of the public in the development of COS is particularly relevant for comparative effectiveness research where long term patient centred outcomes are often the important endpoints.

Awareness of the need for COS and knowledge of the COMET Initiative has continued to grow, reflected in website and database usage figures [7]. More than 16,500 visits were made to the website in 2014 (36% increase over 2013) and 9780 new visitors (43% increase). By December 2014, a total of 6588 searches of the COMET database had been completed; however no information had been collected about the reasons for searching the database.

Aims

To update the original systematic review [5], in order to identify any further studies where a COS has been developed, and to describe the methodological techniques used in these studies. In addition, to understand the reasons why individuals are searching the COMET database.

Methods

Systematic review update

The methods used in this updated review followed the same approach used in the original review [5].

Study selection. Inclusion and exclusion criteria. As described in detail previously [5]. Broadly speaking, studies were eligible for inclusion if they had applied methodology for determining which outcome domains or outcomes should be measured, or are important to measure, in clinical trials or other forms of health research.

Types of participants and interventions. As previously [5], studies were categorised as eligible if they related to participants of any age, with any health condition, in any setting, and assessed the effect of any or all interventions for that condition.

Identification of relevant studies. In January 2015, we searched MEDLINE via Ovid and SCOPUS (including EMBASE) without language restrictions. The search identified studies that had been published from January 2013 onwards (MEDLINE) and August 2013 onwards (SCO-PUS). The multifaceted search strategy developed in the original review using a combination of text words and index terms [8] was used in the current review, with adaptations appropriate for each database.

Selecting studies for inclusion in the review. Records from each database were combined and duplicates were removed. Titles and abstracts were read to assess eligibility of studies for inclusion in the review (stage 1). Full texts of potentially relevant articles were obtained to assess for inclusion (stage 2). Two of three reviewers (SG, JW and BG) independently checked the title and abstract of each citation. One reviewer (SG) assessed the title and abstract of all citations. A second reviewer (JW) assessed the title and abstract of the first half of the citations and a third reviewer (BG) assessed the title and abstract of the second half of the citations. Citations were retained for further checking if agreement could not be reached. One reviewer (SG) assessed each full paper for inclusion in the review and another reviewer (BG) assessed half of the full papers. Reasons for exclusion at this stage were documented for articles judged to be ineligible.

Checking for agreement between reviewers. During each stage of the review process, agreement between reviewers was assessed. Prior to independently assessing records, the three reviewers (SG, JW and BG) independently checked batches of abstracts and full papers for agreement.

Checking for correct exclusion. Of the records that had been excluded on the basis of the title and abstract, full text papers were obtained for a 1% sample and a fourth reviewer (EG) assessed correct exclusion. If any studies were identified as being incorrectly excluded, further checking was performed within the other excluded records. Of the records that had been excluded after reading their full text papers, 5% were assessed for correct exclusion at that stage.

Data extraction. As described in detail previously [5], data were extracted in relation to the study aims, health area, target population, methods of COS development and stakeholder groups involved.

Data analysis and presentation of results. As described previously [5], the results are presented descriptively.

Pop-up survey

During May-June 2015, a pop up survey consisting of one question was added to the COMET website search page, with the intention of finding out why people were searching in the COMET database. At the beginning of each search, the survey would appear, to ask people to select a response as to their reason for searching in the COMET database. The survey was designed so that it was just one question with a set of multiple-choice answers, which was fully contained within the pop-window. This meant that people could answer the question and close the window with a single click. People were asked only once to do this although they could have run a number of searches.

Results

Description of studies

Following the removal of duplicates, 4980 citations were identified in the initial database search. A total of 4551 records were excluded during the title and abstract stage, and a further 400 were excluded following the assessment of full text papers (Fig 1). S1 Table provides a summary of the reasons for exclusion of the full text papers. Twenty-nine citations met the inclusion criteria. In addition to the database search, three additional citations were identified as being eligible for inclusion in the review following reference checking. In total, 32 reports relating to 29 new studies were included (S2 Table).

Included studies

Year of publication. The figure displaying the year of first publication of each COS that was included in the original review has been updated to include the 29 new studies identified in this updated review (Fig 2). The figure confirms that there has been a general increase in the number of COS over the years, and shows that there has been a consistently higher number of COS published annually in recent years than in most years before 2010. Of the 29 studies identified in this update, 28 studies were published between 2013 and 2014, and one study was published in 2010. This study was identified in the searches for the original review [5], but the lack of information on the COS in the title and abstract meant that it was excluded in the initial stages of that review. This study was identified in the updated review, through the checking of references in the included studies, thus highlighting the value of checking reference lists for locating studies in systematic reviews [9].

Scope of core outcome sets. Fig 3 shows  There has been an improvement in the description of the scope of a COS. In the original review, only 25% (49/198) of studies were found to clearly define the population characteristics, as compared to 55% (n = 16/29) of studies in the updated review. The majority, 58% (n = 115/ 198), of studies in the original review neglected to report intervention characteristics, compared to 17% (n = 5/29) in the update.

Methods used to select outcomes. COS have been developed using a variety of methods [5]. The inclusion of a literature/systematic review to inform the development has increased from 33% (n = 66/198) of studies in the original review to 72% (n = 21/29) of studies in the updated review. The use of the Delphi technique for assessing and developing consensus has risen from 15% (n = 29/198) in the original review to 31% (n = 9/29) in the updated review.

People involved in selecting outcomes. The stakeholder groups regarded as key to developing a COS will vary between clinical areas. Clinical experts continue to be involved in almost all studies (see S4 Table), but there has been a trend towards greater involvement of patient and public representatives, increasing from 18% (31/174) in the original review, to 59% (13/22) in this updated review. Table 2 describes the degree of patient representation reported in COS studies included in this updated review.

Pop-up survey

During the four-week period of the survey, 396 different people searched the database. The survey achieved a 52% (206/396) response rate. Table 3 shows the frequency of the reasons selected. The most common were 'I am thinking about developing a core outcome set' and 'I am planning a clinical trial'.

Discussion

As in the original review, studies identified in this update covered various areas of health, with cancer and rheumatology being the most common. In addition to identifying the areas where COS are continuing to be developed, the review identified that COS had been developed for the first time in the areas of rehabilitation and eyes & vision. Regular review updates will allow us to identify the areas where there is an ongoing absence of COS, which may highlight future opportunities for COS developers.

It is apparent that many studies still do not report important characteristics pertaining to the scope of the COS. The development and implementation of a reporting guideline, may facilitate improved reporting of COS, a project currently underway [10].

A variety of methods continue to be used to develop COS. However, there has been an increase in the proportion of studies using literature/systematic reviews and the Delphi technique. As in the original review, the updated review found clinical experts to be the most common group involved in developing COS. However, the proportion of studies that reported patient and public involvement in the process has increased. Since the systematic search was completed in January 2015, we are aware of a further six COS studies that have been published, and of these six studies, 100% have included patient and public representatives. In addition, patient and public representatives are also involved in 89% (64/72) of the planned and ongoing studies registered in the COMET database. It is important that clinical trials include outcomes that are considered to be important to patients and other users of health and social care. It appears that the issue of patient and public involvement is no longer about whether to involve, but rather how. In response to this need, COMET has set up the PoPPIE (People and Patient Participation, Involvement and Engagement) Working Group to develop resources in this area. As an example, plain language summaries are being developed in partnership with patients, http://www.comet-initiative.org/resources/PlainLanguageSummary, and a research agenda around methods for patient engagement has been set.

Implications

The studies identified in this updated review have been added to the COMET database; a freely accessible, publically available, searchable online resource that shows what work has been done in relation to outcomes to measure in effectiveness trials. This update has ensured that the database stores the most current research, and provides readily available information about existing COS, which will assist researchers, practitioners and other stakeholders, including the public, when designing new clinical trials, systematic reviews and clinical guidelines. Moreover, it will also help to avoid unnecessary duplication of efforts in COS development.

The results of the pop up survey demonstrate that people thinking about developing a COS are checking the COMET database to see whether a COS exists in their area of interest to avoid such duplication, hence emphasising the importance of keeping the database current. The popup survey also shows that the COMET database is being used for a variety of reasons by people besides COS developers, including clinical trialists, systematic reviewers, auditors, COS funders, and people who have been asked to participate in the development of COS. Interestingly, nearly one sixth of respondents were searching for reasons not anticipated, and further research is needed to understand those. Although 48% of visitors to the website did not

Limitations

There are a number of limitations of this review that should be noted. First, although two wellestablished, large bibliographic databases were searched to identify studies meeting the inclusion criteria, we may have missed some relevant studies within these databases, either because the reports of these studies are not indexed in these databases or because the information in the indexed record is insufficient to show that the report is eligible. It would be beneficial for bibliographic databases to introduce an indexing term to facilitate easier identification of COS. As shown by both our original review [8] and this update, the identification of COS is a complex task and we will keep the search strategy under review and refine it as necessary to minimise the possibility of missing eligible studies in the future. A second limitation of the review is that the lack of an assessment tool means that there has been no formal quality assessment of the included studies. Determining the quality of a COS is not straightforward. A high quality COS would be one that leads to improved health outcomes but this would be difficult to measure. A tool is needed to assess how the COS developers minimised biases which would otherwise undermine the ability of the COS to have a positive impact on patient care and outcomes.

Conclusion

We have provided an updated review of studies that have addressed the development of COS for measurement and reporting in clinical trials. The updated review has demonstrated that COS are continuing to be developed across a range of health areas. Recent studies appear to have adopted (or, at least, reported) a more structured approach towards COS development, and public representation has increased. However, although general reporting quality has increased, there remains a need for developers to adequately describe details about the scope of COS, and for greater public engagement. The COMET database is likely to be a useful resource for both COS developers and users of COS including clinical trialists and systematic reviewers.

Supporting Information S1 PRISMA Checklist. PRISMA checklist for content of a systematic review.

(DOC) S1 Table. Reason for exclusion at stage 2 (assessment of full text reports).

Fig 1 .
1Identification of studies. doi:10.1371/journal.pone.0146444.g001 Choosing Important Health Outcomes: Updated Review and User Survey PLOS ONE | DOI:10.1371/journal.pone.0146444 January 19, 2016

Fig 2 .
2Year of first publication of each COS study (n = 227).doi:10.1371/journal.pone.0146444.g002

Fig 3 .
3Number of COS developed in each disease category (n = 227). doi:10.1371/journal.pone.0146444.g003

the number of COS developed according to disease category. The classification of 227 published COS studies are presented in S3Table.The scope of published COS studies is summarised inTable 1and includes both the 198 COS included in the original review and the 29 new COS that have been added by this updated review. This includes study aims, setting for intended use, population characteristics and intervention characteristics.

Table 1 .
1The scope of included studies (n = 227).respond to the survey, closing down the pop-up box immediately, this may reflect a typical response rate for such surveys.Original review n (%)
Updated review n (%)
Combined n (%)

Table 1 .
1(Continued)    Original review n (%)
Updated review n (%)
Combined n (%)

Table 2 .
2Public involvement detail where reported (n = 6).Methods used
Total number of participants
Number of public participants
% Public participants

Table . Table
.of reports included in updated review (n = 32). (DOCX) S3Table. Classification of COS according to condition (n = 227 published studies). (DOCX) S4 Table. Participant groups involved in selecting outcomes in new studies identified in the review update (n = 29). (DOCX)
PLOS ONE | DOI:10.1371/journal.pone.0146444 January 19, 2016
PLOS ONE | DOI:10.1371/journal.pone.0146444 January 19, 2016 12 / 12
AcknowledgmentsThe authors would like to thank Jennifer Weston and Binu Gurung for their assistance in screening studies for inclusion in the review.Author ContributionsMeeting and teleconferences (mixed)61 17% We are considering funding a clinical trial 2 I am involved in auditing a condition 6As part of a general educational activity 29General interest 27None of the above 32Did not respond 190
Avoidable waste in the production and reporting of research evidence. I Chalmers, P Glasziou, 10.1016/S0140-6736(09)60329-9Lancet. 37419525005Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009; 374:86-89. doi: 10.1016/S0140-6736(09)60329-9 PMID: 19525005

Biomedical research: increasing value, reducing waste. M R Macleod, S Michie, I Roberts, U Dirnagl, I Chalmers, Jpa Ioannidis, 10.1016/S0140-6736(13)62329-6Lancet. 383Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JPA, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014; 383:101-104. doi: 10.1016/S0140-6736(13)62329-6 PMID: 24411643

Developing core outcome sets for clinical trials: issues to consider. P R Williamson, D G Altman, J M Blazeby, M Clarke, D Devane, E Gargon, 10.1186/1745-6215-13-132Trials. 13132Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core out- come sets for clinical trials: issues to consider. Trials. 2012; 13:132. doi: 10.1186/1745-6215-13-132 PMID: 22867278

A systematic review of studies that aim to determine which outcomes to measure in clinical trials in children. I Sinha, L Jones, R L Smyth, P R Williamson, 10.1371/journal.pmed.005009618447577PLoS Med. 596Sinha I, Jones L, Smyth RL, Williamson PR. A systematic review of studies that aim to determine which outcomes to measure in clinical trials in children. PLoS Med. 2008; 5:e96. doi: 10.1371/journal.pmed. 0050096 PMID: 18447577

Choosing important health outcomes for comparative effectiveness research: a systematic review. E Gargon, B Gurung, N Medley, D G Altman, J M Blazeby, M Clarke, 10.1371/journal.pone.009911124932522PLoS ONE. 9699111Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, Clarke M, et al. Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS ONE. 2014; 9(6):e99111. doi: 10.1371/journal.pone.0099111 PMID: 24932522

Briefing notes for researchers: involving the public in NHS, public health and social care research. Eastleigh: INVOLVE. INVOLVE. Briefing notes for researchers: involving the public in NHS, public health and social care research. Eastleigh: INVOLVE; 2012.

The COMET Initiative database: progress and activities from. E Gargon, P R Williamson, D G Altman, J M Blazeby, M Clarke, Trials. 151279Gargon E, Williamson PR, Altman DG, Blazeby JM, Clarke M. The COMET Initiative database: prog- ress and activities from 2011 to 2013. Trials. 2014; 15(1):279.

Collating the knowledge base for core outcome set development: developing and appraising the search strategy for a systematic review. E Gargon, P R Williamson, M Clarke, 10.1186/s12874-015-0019-9BMC Medical Research Methodology. 1525888523Gargon E, Williamson PR, Clarke M. Collating the knowledge base for core outcome set development: developing and appraising the search strategy for a systematic review. BMC Medical Research Meth- odology. 2015; 15:26. doi: 10.1186/s12874-015-0019-9 PMID: 25888523

Checking reference lists to find studies for systematic reviews. T Horsley, O Dingwall, M Sampson, Cochrane Databaser Syst Rev. 826Horsley T, Dingwall O, Sampson M. Checking reference lists to find studies for systematic reviews. Cochrane Databaser Syst Rev. 2011; 8:MR000026.

COS-STAR: a reporting guideline for studies developing core outcome sets (protocol). J J Kirkham, S Gorst, D G Altman, J Blazeby, M Clarke, D Devane, 10.1186/s13063-015-0913-9Trials. 16373Kirkham JJ, Gorst S, Altman DG, Blazeby J, Clarke M, Devane D, et al. COS-STAR: a reporting guide- line for studies developing core outcome sets (protocol). Trials. 2015; 16:373. doi: 10.1186/s13063- 015-0913-9 PMID: 26297658