Abstract
Background: The maintenance of standards is a problem for postgraduate medical examinations, particularly if they use norm-referencing as the sole method of standard setting. In each of its diets, the MRCP(UK) Part 1 Examination includes a number of marker questions, which are unchanged from their use in a previous diet. This paper describes two complementary studies of marker questions for 52 diets of the MRCP(UK) Part 1 Examination over the years 1985 to 2001 to assess whether standards have changed.
Background
The role of postgraduate medical examinations is to set standards of practice and thus to assure the public and the medical profession that doctors have the knowledge and expertise required to diagnose and treat patients, and to progress in their medical careers. An important part of running "high-stakes" qualifying examinations is the setting of a pass mark, and many postgraduate examinations in the UK have relied on norm-referencing, in which a fixed proportion of candidates passes at each occasion [1] .
Although administratively straightforward, norm-referencing has several problems. The absolute performance across different diets of the exam varies, firstly due to asking questions of different difficulty, and secondly as a result of candidates differing in their ability. These factors of question difficulty and candidate ability are entirely confounded when only overall examination performance is considered, and norm-referencing cannot assess whether candidate ability varies from occasion to occasion. The result, as was shown in an important and influential analysis of the American Board of Internal Medicine's examination, was a slide in standards over the time-period 1983 to 1988 [2] . On that basis the Board implemented a process of criterion-referencing, in which examiners set a pass mark by assessing the content of each individual question on the examination. Having said that, we also acknowledge that there are arguments in some situations for the use of norm-referencing [3] , and as a result there are even stronger arguments for the use of compromise methods [4, 5] .
In this paper we primarily wish to assess whether the absolute standard of candidates taking the MRCP(UK) examination changed over the period 1985 to 2002, when the format of the examination and the method for setting its pass mark were relatively constant. The assessment of the true ability of candidates, independently of question difficulty, requires a process of equating to establish whether candidates on one occasion are of equivalent ability to those on another occasion. Statistical equating can be carried out if marker questions are available, the same questions being used on two different diets [6] .
Background to Study 1
In this paper we describe two complementary analyses of marker questions used in the MRCP(UK) Part1 Examination between 1985 and the first diet of 2002 (2002/1). Three diets of the Examination were held each year, and in all diets the pass rate was norm-referenced at 35% of those candidates taking the Examination on their first four attempts at UK examination centres. Two separate studies are described: 
Study 2: Comparison of 1996/2 and 2001/3 diets
Although Study 2 follows on from Study 1, conceptually and in practice, the requirements of the journal are that the method of each is presented before the results of each, and then the discussion of each. This is somewhat confusing, and in particular it is necessary to present the background and justification to study 2 before the results of Study 1. Readers who are confused by this layout are referred to the Additional File, where the text of the paper is ordered in a more logical format.
Background to Study 2
As will become apparent, Study 1 demonstrates that performance on marker questions declined during the period 1997 to 2002. However there are several possible interpretations of that result, not least because the data are aggregated across all candidates and do not allow analysis of sub-groups of candidates, such as UK graduates on first and subsequent attempts. Study 2 therefore analysed raw data at the level of responses to individual questions from individual candidates, thereby allowing a detailed comparison of the two diets.
Methods

Methods: Study 1
The format of the Examination, which was held three times a year, was unchanged until 2002/1, consisting of 60 five-part Multiple True-False questions (a total of 300 items). Negative marking was used in scoring the examination, and results were expressed as a 'corrected percentage correct', which takes guessing into account. The only substantive change in the Examination Regulations was that candidates were allowed to make an unlimited number of attempts from 1999/2 onwards, whereas previously they had been limited to four attempts.
Marker items were defined as any items included for a second time in any of the 43 diets of the MRCP(UK) Part 1 Examination held between 1988/1 and 2002/1, and which had been used in the previous diet with the stem, the item and the correct answer unchanged. Documents for tracing markers prior to 1988 were not readily available. The dates of the two diets, and the proportion of candidates getting each item right, wrong or not answering it, were recorded for each marker item.
Statistical analysis: Study 1
The central issue in a marker question analysis is whether, on average, aggregate performance on items has increased or decreased between the first and second usage. By assessing how such a change relates to the dates when the items were used, one can estimate the changing overall performance of candidates in the Examination. The study is a variant of an incomplete paired comparison design [7, 8] .
If the percentages of candidates who get an item correct or wrong on the first and second occasion are c 1 , w 1 and c 2 , w 2 , then the change in performance of the item is:
It should be noted that ∆ is independent of the proportion of candidates choosing not to answer an item since those not answering are effectively neutral, neither gaining a mark from a correct answer nor losing one due to a wrong answer. Given the different performance of the marker questions at different diets, one can reconstruct the changing true ability of the candidates by a process analogous to triangulation, a process that can be carried out using multiple regression (for a technical explanation see below). The method takes one arbitrarily chosen date as the reference category (the first diet of 2000 was used) and estimates the ability of the candidates at each other diet relative to the reference diet. The estimates, like all regression coefficients, have a standard error and confidence intervals.
Statistical method, Study 1: details of technique
Although there are many articles and books devoted to test equating, they usually consider only the problem of a large set of overlapping items occurring in two adjacent diets of an examination. The marker questions for Part 1 are large in number but are distributed across many diets, making a different problem for estimation. It is interesting to note that the regression solution described here is very general in its applicability, and was previously used in an entirely different context [9] .
The exam is taken on n occasions. Let the standard of the candidates vary, such that for diet d, the true standard, relative to some arbitrary reference diet, r, is s d (i.e. s r = 0). If the same item (question) were to be used on every diet, then on diet 1, a proportion s 1 would get the item correct, on diet 2 s 2 would get the item correct, etc.
A diet has a variable number of marker items, which have been used unchanged in a number of previous diets. An individual marker item, m, in diet k, will have been used previously in, say, diet j. The statistical analysis is required to derive the true standard of the candidates at the various diets, s i (i = 1, n; s r = 0), from the performance on the marker items.
Let the difference in performance on marker item m between diets j and k be expressed as ∆ m . In the specific case of a true-false examination with negative marking, let c j % get the item correct at time j, w j % get the item wrong at time j, and na j % not answer the item at time j (and with equivalent symbols for time k). ∆ m is then calculated as:
Note that if the performance of the candidates on the question is the same at diets j and k then ∆ m = 0.
A series of dummy variables, v 1 to v n , is then created for each marker variable, which take the values : SPSS version 11.5 was used for statistical analysis of the data. For statistical analysis, diets 1, 2 and 3 in a year were set at 0, .33 and 0.67 of the year.
Methods: Study 2
Raw data were available only for the 1996/2 diet and the diets from 1997/1 onwards, and of these only the 1996/2 and 2001/3 diets shared sufficient marker items for analysis (see figure 1 ). Since the 1996/2 diet took place just before the apparent decline in performance seen in figure  2 , whereas 2001/3 took place afterwards, a comparison is appropriate.
The 1996/2 and 2001/3 diets had 28 items in common, which were scored on the basis of +1 for a correct answer, -1 for a wrong answer and 0 for no answer, giving a 'marker score' expressed as a corrected percentage score. All analyses were restricted to 'UK graduates', defined as those with primary registrable qualifications from United Kingdom medical schools. after which ability appears to decline fairly steeply. That hypothesis was formalised by using non-linear regression to fit two straight lines to the data, each with its own independent slope, with a 'dog-leg' mid-way through the data, the date of the dog-leg itself being a free parameter. The dog-leg curve shown in figure 2 fits the data well (R 2 = .468), and is a significant improvement over a simple linear regression (F(2,48) = 18.28, p < 0.001). The inflexion of the dog-leg rebeccadiaz@example.org (equivalent to the 1997/2 diet), with 95% confidence intervals of 1996.0 to 1998.6 (equivalent to diets 1996/1 to 1998/3). The slope before 1997 is +0.18%/year (95% CI 0.03% to 0.33%), and the slope after 1997 is -1.34%/year (95% CI -1.93% to -0.75%).
Results


Results: Study 2
The 1996/2 diet was taken by 2132 candidates of whom 852 were UK graduates. The 2001/3 diet was taken by 2051 candidates of whom 557 were UK graduates. The 28 marker items were combined into a single scale for which
The estimated true ability of candidates taking the examination at each diet Figure 2 The estimated true ability of candidates taking the examination at each diet. The solid points are the estimates from the regression (s d ). The thick black line is a fitted lowess line (locally weighted regression), and the thin black lines show lowess lines through the one standard error confidence intervals for the points. The thick dashed line is the 'dog-leg' curve (see text). The vertical dashed line indicates the date when unlimited attempts were allowed in the Part 1 examination. alpha was 0.593, which is equivalent, using the Spearman-Brown formula [10] , to a reliability of 0.940 for a full-length 300 item test, somewhat higher than the mean reliability across 54 diets of the examination as a whole of 0.865 [11] .
Overall performance on marker items: Study 2
In the 1996/2 diet, the 852 UK graduates have a mean marker score of 69.4 (SD 14.5), compared with a mean marker score of 59.6 (SD 15. (table 1) and better on 2 items, and performance was not significantly different for 9 items (see Figure 5) . The median change in performance was -7.65% (inter-quartile range -1.42% to -15.73%).
Overall performance on the 28 marker items for UK graduates taking the 1996/2 and 2001/3 diets of the examination, on their first and subsequent attempts Figure 3 Overall performance on the 28 marker items for UK graduates taking the 1996/2 and 2001/3 diets of the examination, on their first and subsequent attempts. Until diet 1999/2 candidates were only allowed to take the exam four times, whereas after 1999/2 they were allowed unlimited attempts. Fewer than 20 candidates were on their 6 th or higher attempt, and they have been omitted from the analysis. 
Conclusion

Conclusions: Study 1
In the analysis of marker items in Study 1, candidates showed a gradual improvement in performance on the MRCP(UK) Part 1 examination for the twelve years between 1985 and 1997. Such a change could either result from a genuine increase in candidates' knowledge or from candidates becoming more 'test-wise', perhaps due to attending crammer courses and becoming more aware of questions from previous papers. However, the latter explanation seems unlikely in view of the sharp decline in performance of candidates after 1997.
If the decline in performance after 1997 reflects decreasing candidate ability, then that has important implications for medical education and training, and possible confounders and artefacts must be excluded. Separate analyses, not reported in full here but available in the Additional File, have investigated the following possibilities:
Change in number of attempts allowed
From 1999/2 onwards, candidates were not restricted to four attempts at the Examination. Figure 2 , however, shows that the decline in performance started well before that date.
Changes in strategy in relation to negative marking
The MRCP(UK) Part 1 examination was negatively marked, wrong answers incurring a greater penalty than unanswered questions. If candidates changed their strategy, that may have produced an apparent decline in performance. We have modelled the number of notanswered questions, and although this did show a decline in between 1990 and 1993, the timing was unrelated to the change shown in Figure 2 . This study of 4,405 marker items suggests the standard of candidates taking the Part 1 MRCP(UK) Examination may have changed over time, rising gradually until about 1997, and then declining rather more rapidly. The discontinuity around 1997 does not seem to be related to any obvious change in the structure of the Examination or the composition of the candidates taking it.
Changes in the mix of candidates taking the Examination

Conclusion: Study 2
Study 2 examined the performances of individual candidates on marker items and confirms the earlier finding that performance dropped between 1996/2 and 2001/3. In particular, fewer UK graduates who passed the Examination in 2001/3 gave correct answers than had equivalent candidates in 1996/2. The median change in performance on items of -7.75% was similar to the expected change of -7.1%, based on the -1.34% per year shown in figure 2 . The results of figure 3 are probably not therefore distorted or biased by inadvertent differences in candidate mix between the diets.
We found that the performances on individual marker items between 1996/2 and 2001/3 had dropped for 21 of the 28 items. None of the questions is about recondite, obscure or unimportant areas of knowledge for a general physician in training, and none of the changes are likely to reflect changes in the importance of knowledge, in understanding of disease mechanisms, or in treatment strategies. They are therefore acceptable marker questions. The largest decreases were on the electrocardiography and anatomy of the Wolff-Parkinson-White syndrome, aseptic meningitis in infectious mononucleosis, and bone marrow biopsy findings in the anaemia of chronic infection, all of which are important clinical problems. The only two significant increases in knowledge are on questions concerning C-reactive protein (a relatively recently introduced clinical test), and on the diagnosis of organic brain disease. 
Discussion: General

iii. Changes in clinical experience
In recent years the working hours of junior doctors have declined, in part due to changes in Government regulations, and clinical experience and hence examination performance may also have declined. In the absence of good measures of clinical experience this hypothesis is difficult to test. There is, however, evidence that the undergraduate clinical experience of UK doctors qualifying in 1996 was lower than that of doctors qualifying a decade earlier [14] , and that more recent medical graduates have less knowledge of basic clinical science [15] .
iv. Changes in undergraduate medical training
Undergraduate medical training in the UK has been continually changing for nearly four decades, dating back primarily to the Royal Commission of 1968 [16] , and supported by subsequent recommendations from the General Medical Council [17, 18] ; new subjects were introduced into the curriculum, and traditional subjects such as anatomy were de-emphasised. Particularly dramatic changes followed the General Medical Council's Tomorrow's Doctors [19] of 1993, as a result of which many medical schools introduced major curricular changes, often involving problem-based learning. A number of medical schools also merged, and most medical schools became larger. Although these latter innovations might have caused changes in the knowledge-base of graduates, they are unlikely to explain changes we describe here, which began in 1997 and hence relate to students entering medical school in 1990 or 1991, before the publication of Tomorrow's Doctors. A key question concerns whether the standards of undergraduate examinations, both basic medical sciences and finals, have been maintained; however, the patchy use of marker questions, frequent changes in undergraduate examination formats, and the absence of a UK national medical licensing examination make it unlikely that the question can be answered easily. Indeed, the only reliable evidence on the absolute standard of undergraduate training may have to come from performance in postgraduate examinations.
In summary, we have provided evidence of a decline in the performance of candidates taking the MRCP(UK) Part 1 Examination between 1997 and 2001. In addition, as a result of the reliance on norm-referencing, there was also a decline in the standard of those passing the Examination. Criterion referencing is now included as a central part of the MRCP(UK) Examination standard setting process. The reasons for the declining standard of UK graduates are not clear, but on balance are more likely to reflect changes in undergraduate training than changes in postgraduate medical education. More research into the standard of other postgraduate examinations, as well as undergraduate assessments, is urgently needed.

{'paper_id': '12458550', 'title': 'Changes in standard of candidates taking the MRCP(UK) Part 1 examination, 1985 to 2002: Analysis of marker questions', 's2_url': 'https://api.semanticscholar.org/CorpusID:12458550', 'source': 'S2ORC'}