The testing effect for mediator final test cues and related final test cues in online and laboratory experiments

Leonora C Coppens
Peter P J L Verkoeijen
Samantha Bouwmeester
Remy M J P Rikers
The testing effect for mediator final test cues and related final test cues in online and laboratory experiments
10.1186/s40359-016-0127-2R E S E A R C H A R T I C L E Open Access
Background: The testing effect is the finding that information that is retrieved during learning is more often correctly retrieved on a final test than information that is restudied. According to the semantic mediator hypothesis the testing effect arises because retrieval practice of cue-target pairs (mother-child) activates semantically related mediators (father) more than restudying. Hence, the mediator-target (father-child) association should be stronger for retrieved than restudied pairs. Indeed, Carpenter (2011) found a larger testing effect when participants received mediators (father) than when they received target-related words (birth) as final test cues. Methods: The present study started as an attempt to test an alternative account of Carpenter's results. However, it turned into a series of conceptual (Experiment 1) and direct (Experiment 2 and 3) replications conducted with online samples. The results of these online replications were compared with those of similar existing laboratory experiments through small-scale meta-analyses. Results: The results showed that (1) the magnitude of the raw mediator testing effect advantage is comparable for online and laboratory experiments, (2) in both online and laboratory experiments the magnitude of the raw mediator testing effect advantage is smaller than in Carpenter's original experiment, and (3) the testing effect for related cues varies considerably between online experiments. Conclusions: The variability in the testing effect for related cues in online experiments could point toward moderators of the related cue short-term testing effect. The raw mediator testing effect advantage is smaller than in Carpenter's original experiment.

Background

Information that has been retrieved from memory is generally remembered better than information that has only been studied. This phenomenon is referred to as the testing effect. The widely investigated testing effect has proven to be a robust phenomenon as it has been demonstrated with various final memory tests, materials, and participants (see for recent reviews [1][2][3][4][5][6][7][8]).

Although the testing effect has been well established empirically, the cognitive mechanisms that contribute to the emergence of the effect are less clear. Carpenter [9] suggested that elaborative processes underlie the testing effect (see [10] for a similar account). According to her elaborative retrieval hypothesis, retrieving a target based on the cue during practice causes more elaboration than restudying the entire pair. This elaboration helps retrieval at a final memory test because it causes activation of information which is then coupled with the target, hence creating additional retrieval routes. To exemplify the proposed theoretical mechanism, consider a participant who has to learn the word pair mother -child. Retrieving the target when given the cue (i.e., mother) is more likely to lead to the activation of information associated with that cue (e.g., love, father, diapers) than restudying the entire word pair. As a result, the activated information is associated with the target (i.e., child) thereby providing additional retrieval routes to the target. As a consequence, targets from previously retrieved word pairs are more likely to be retrieved than targets from restudied word pairs: the testing effect arises. However, Carpenter [11] noted that the elaborative retrieval hypothesis was not specific about what related information is activated during retrieval practice. To address this issue, she turned to the mediator effectiveness hypothesis put forward by Pyc and Rawson [12,13]. Based on the mediator effectiveness hypothesis, Carpenter proposed that semantic mediators might be more likely to get activated during retrieval practice than during restudying (henceforth denoted as the semantic mediator hypothesis). Carpenter defined a semantic mediator as a word that according to the norms of Nelson, McEvoy, and Schreiber [14] has a strong forward association with the cue (i.e., when given the cue people will often spontaneously activate the mediator) and that is easily coupled with the target. For instance, in the word pair mother-child, the cue (mother) will elicitat least for a vast majority of people -the word father. The word father can easily be coupled with the target child. Hence, father is a semantic mediator in case of this particular word pair. The semantic mediator hypothesis predicts that the link between the semantic mediator father and the target child will be stronger after retrieval practice than after restudying.

Carpenter [11] (Experiment 2) tested this prediction using cue-target pairs such as mother -child. These word pairs were studied and then restudied once or retrieved once. After a 30-min distractor task, participants received a final test with one of three cue types: the original cue, a semantic mediator or a new cue that was weakly related to the target: a related cue. The latter two are relevant for the present study. Carpenter's results showed a testing effect in the original cue condition. Moreover, at the final test the advantage of retrieval practice over restudying was greater when participants were cued with a mediator (father) than when they were cued with a related cue (birth). Furthermore, targets from the retrieval practice condition were more often correctly produced during the final test when they were cued with mediators than when they were cued with related words. This difference in memory performance between mediator-cues and relatedcues was much smaller for restudied items.

These results of Carpenter's second experiment are important because they provide direct empirical support for a crucial assumption of the semantic mediator hypothesis: the assumption that the link between a mediator and a target is strengthened more during retrieval practice than during restudying. However, there might be an alternative explanation for the findings of Carpenter's [11] second experiment. We noted that some of the mediators used in this study were quite strongly associated with the cue. For example, one of the word pairs was motherchild with the mediator father and the related cue birth. In this case, there is a strong cue-mediator association from mother to father (and no forward association from mother to birth), but the mediator father is also strongly associated with the original cue mother (.706 according to the norms of Nelson et al. [14]). Now it might be possible the larger testing effect on a mediator-cued final test (father -_ ) as opposed to a related word-cued final test (birth -_ ) was caused by mediators with strong mediator-cue associations. That is, when given the mediator father at the final test, participants can easily retrieve the original cue mother. Because it is easier to retrieve the target from the original cue after retrieval practice than after restudying (in Carpenter's Experiment 2, final test performance after a relatively short retention interval was better for tested than for restudied items; cf. [15][16][17]), activation of the original cue through the mediator will facilitate retrieval of the target more after retrieval practice than after restudying. By contrast, the related final test cues in Carpenter's experiment did not have an associative relationship with the original cues, and therefore it was harder to retrieve the original cue from a related final test cue than from a mediator final test cue. If the testing effect emerges due to a strengthened cue-target link then related final test cues are less likely to produce a testing effect than mediator final test cues. Thus, strong mediator-cue associations in Carpenter's stimulus materials in combination with a strengthened cue-target link might explain why the testing effect was larger for mediator final test cues than for related final test cues.

To test this alternative explanation of the results of Carpenter's Experiment 2, we repeated the experiment with new stimuli. We created two lists of 16 word sets that consisted of a cue, a target, a mediator, and a related cue (see Fig. 1). In both the stimuli lists, there was a weak cue-target association, a strong cue-mediator association and a weak association between the related cue and the target. The difference between the two stimuli lists was the mediator-cue association. In one stimuli list, there was a strong mediatorcue association (as illustrated in the left part of Fig. 1). This corresponds with the situation in some of the stimuli of Carpenter [11], such as motherchild with the mediator father. In the other stimuli list, there was no mediator-cue association (as illustrated in the right part of Fig. 1). An example of such a word set is the pair anatomy -science with the mediator body. There is no pre-existing association from body to anatomy. Therefore, if the proposed mediator body is not activated during learning it will not activate the original cue anatomy and the alternative route from the mediator through the original cue to the target is blocked.

If our alternative account is correct and the larger testing effect in the mediator-cued final test condition is caused by a strong mediator-cue association, then the stimuli with a strong mediator-cue association should yield a replication of the pattern Carpenter [11] found: a larger testing effect on a mediator-cued final test than on a related-cue-cued final test. By contrast, for stimuli without a mediator-cue association the magnitude of the testing effect should not differ between mediator final test cues and related final test cues. It should be noted that Carpenter's semantic mediator hypothesis predicts a larger testing effect on a mediator-cued final test than on a related-cue-cued final test for both stimuli lists.

Experiment 1

Methods Participants

For Experiment 1, we recruited participants via Amazon Mechanical Turk (MTurk; http://www.mturk.com). MTurk is an online system in which requesters can open an account and post a variety of tasks. These tasks are referred to as human intelligence tasks, or HITS. People who register as MTurk workers can take part in HITS for a monetary reward. Simcox and Fiez [18] list a number of advantages of the MTurk participants pool as compared to the (psychology) undergraduates participants pool from which samples are traditionally drawn in psychological research. First, MTurk participants are more diverse in terms of ethnicity, economic background and age, which benefits the external validity of MTurk research. Second, MTurk provides a large and stable pool of participants from which samples can be drawn year round. Third, experiments can be run very rapidly via MTurk. A disadvantage, however, is that the workers population might be more heterogeneous than the undergraduate population and that they complete the online task under less standardized conditions. This generally leads to more within subject variance which in turnceteris paribus -deflates the effect-size.

Participants in Carpenter's [11] original experiment were undergraduate students instead of MTurk workers. Hence, our sample is drawn from a different population than hers. However, we think this difference is not problematic for a number of reasons. For one, nowhere in the original paper does Carpenter indicate that specific sample characteristics are required to obtain the crucial finding from her second experiment. Also, evidence is accumulating that cognitive psychological findings translate readily from the psychological laboratory to the online Mechanical Turk platform (e.g., [19][20][21][22][23]). In addition, replicating Carpenter's findings with a sample from a more heterogeneous population than the relatively homogeneous undergraduate population would constitute evidence for the robustness and generality of Carpenter's findings. This in turn would rule out that Carpenter's findings are restricted to a specific and narrow population.

Two hundred thirty-five (235) United States residents completed the experiment via Mechanical Turk. Participants were paid $1.50 for their participation. The data of 9 participants were not included in the analysis because their native language was not English, leaving 226 participants (142 females, 84 males, age range 19-66, mean age 35.4, SD = 11.7). Participants were randomly assigned to conditions.

Materials and design

A 2 (list: strong mediator-cue association vs. no mediator-cue association) × 2 (learning condition: restudy vs. retrieval practice) × 2 (final test cue: mediator vs. related) between-subjects design was used. To investigate the effect of the mediator-cue association, we used the association norms of Nelson et al. [14] to create two lists of 16 word sets (see Appendix A). Each word set consisted of a cue and a target (weak cue-target association, .01 -.05), a mediator (strong cue-mediator association, >.5) and a related cue (weak related word-target association, .01 -.05). The difference between the two lists was the mediator-cue association. In one of the lists, the mediator-cue association in each word set was higher than .5. In the other list, the mediator-cue association in each set was 0 (see Fig. 1).

The experiment was created and run in Qualtrics [24] in order to control timing and randomization of stimuli. In the strong mediator-cue association condition (left), there was a strong association between the mediator and the cue. In the no mediator-cue association condition (right), there was no association between the mediator and the cue

Procedure

The procedure was identical to that of Experiment 2 of Carpenter [11] with the exception of the original cue final test condition, which we did not include because it was not relevant to the current research question. The experiment was placed as a task on MTurk with a short description of the experiment ('this task involves learning word pairs and answering trivia questions'). When a worker was interested in completing the task, she or he could participate in the experiment by clicking on a link and visiting a website.

The welcome screen of the experiment included a description of the task and questions about participants' age, gender, mother tongue, and level of education. In addition, participants rated three statements about the testing environment on a 5-point Likert scale. After the participant answered these questions, the learning phase began. In the learning phase all 16 cue-target pairs in one of the lists were shown in a different random order for each participant. The cue was presented on the left side of the screen and the underlined target was presented on the right. The task of the participants was to judge how related the words were on a scale from 1 to 5 (1 = not at all related -5 = highly related), and to try to remember the word pairs for a later memory test. The study trials were self-paced. After the study trials, there was a short filler task of 30 s, which involved adding single-digit numbers that appeared on the screen in a rapid sequence. Then the cue-target pairs were presented again in a new random order during restudy or retrieval practice trials. Restudy trials were the same as study trials; participants again indicated how related the words were on a scale from 1 to 5. In retrieval practice trials, only the cue was presented and participants had to type the target in a text box to the right of the cue. Both the restudy and retrieval practice trials were selfpaced, as was the case in Carpenter's [11] Experiment 2.

After a filler task of 30 min, in which participants answered multiple-choice trivia questions (e.g., 'What does NASA stand for? A. National Aeronautics and Space Administration; B. National Astronauts and Space Adventures; C. Nebulous Air and Starry Atmosphere; D. New Airways and Spatial Asteroids'), the final test began. Participants were informed that they would see words that were somehow related to the second, underlined word of the word pairs they saw earlier, and that their task was to think of the target word that matched the given word and enter the matching word in a text box. An example, using words that did not occur in the experiment, was included to elucidate the instructions. During the final test, participants were either cued with the mediator or with the related cue of each word pair. The cue was presented on the left side of the screen and participants entered a response into a text box on the right side of the screen. The final test was self-paced.

To end the experiment, participants rated five concluding statements about the clarity of instructions, motivation, effort, and concentration on a 5-point Likert scale. The duration of the entire experiment was about 45 min.

Results

An alpha level of .05 was used for all statistical tests reported in this paper. Minor typing errors in which one letter was missing, added or in the wrong place were corrected before analysis.

Working conditions

The three statements about working conditions of the participants were rated as follows: 'I'm in a noisy environment': mean rating 1.5 (SD = 0.77), 'There are a lot of distractions here': mean rating 1.52 (SD = 0.74), 'I'm in a busy environment': mean rating 1.34 (SD = 0.66). The statements at the end of the experiment were rated as follows: ' All instructions were clear and I was sure of what I was supposed to do': mean rating 4.02 (SD = 1), 'I found the experiment interesting': mean rating 4.02 (SD = 1), 'The experiment was difficult': mean rating 4.06 (SD = 0.98), 'I really tried to remember the word pairs': mean rating 4.51 (SD = 0.79), 'I was distracted during the experiment': mean rating 1.83 (SD = 0.98).

To make sure the working conditions of the MTurk workers resembled those of participants in the laboratory as much as possible we only included those participants in the subsequent analyses who scored 1 or 2 on the last question (i.e., "I was distracted during the experiment"). The resultant sample consisted of 181 participants.

Intervening test

In the list with no mediator-cue associations the mean proportion of correct targets retrieved on the intervening test was .91 (SD = .12) in the mediator final-test condition and .84 (SD = .23) in the related final-test condition. In the list with strong mediator-cue associations, the mean proportion of correct targets retrieved on the intervening test was .97 (SD = .09) in the mediator final-test condition and .94 (SD = .09) in the related final-test condition.

Final test

The proportion of correctly recalled targets on the final test for the no mediator-cue (no MC) association list and the strong mediator-cue association list (strong MC) are presented in the second and third row of Table 1.

No mediator-cue association A 2 (learning condition: restudy vs. retrieval practice) × 2 (final test cue: related vs. mediator) between-subjects analysis of variance (ANOVA) on the proportion correctly recalled targets on the final test yielded a small, marginally significant main effect of learning condition, F(1,83) = 3.416, p = .068, η 2 p = .040. Overall, mean target retrieval was higher for cue-target pairs learned through retrieval practice than through restudying, i.e., a testing effect. The effect of final test cue was very small and not significant, F(1,83) = 0.10, p = .919, η 2 p < .01. This suggests that mean target retrieval did not differ between related final test cues and mediator final test cues. Furthermore, the Learning Condition × Final Test Cue interaction was small and not significant, F(1,83) = 0.875, p = .352, η 2 p = .010. For the crucial Learning Condition × Final Test Cue interaction effect, it is also useful to look at the difference in the testing effect between mediator cues and related cues. In this case, the difference was .08 indicating that the testing effect (mean proportion correct for tested targets -mean proportion correct for restudied targets) was about 14 % points higher for mediator final test cues than for related cues. The direction of this mediator testing effect advantage is in line with Carpenter's results (i.e., a larger testing effect on a mediator-cued final test than a related word-cued final test), but in her study the advantage was much larger, i.e., 23 % points.

Strong mediator-cue association A 2 (learning condition: restudy vs. retrieval practice) × 2 (final test cue: related vs. mediator) between-subjects ANOVA revealed a significant small sized main effect of learning condition, F(1,90) = 6.330, p = .0104, η 2 p = .066: mean target retrieval was higher for cue-target pairs learned through retrieval practice than through restudying (i.e., a testing effect). Furthermore, we found a small significant main effect of final test cue, F(1,90) = 8.190, p = .005, η 2 p = .083. The mean final test performance was better for mediator final test cues than for related final test cues. The Learning Condition × Final Test Cue interaction was small and not significant, F(1,90) = 1.024, p = .314, η 2 p = .011. The testing effect for mediator cues was about 14 % points smaller than for related cues. This mediator testing effect disadvantage is inconsistent with Carpenter's [11] mediator testing effect advantage.

Discussion

The results of Experiment 1 revealed no significant interaction effect between final test cue and learning condition in either of the two lists. The pattern of sample means showed, however, a larger testing effect for mediator final test cues than for related final test cues in the list with no mediator-cue associations. This pattern of results is similar to the one observed by Carpenter [11] in her second experiment. By contrast, in the list with strong mediator-cue associations, the testing effect was larger for related final test cues than for mediator final test cues. Taken together, these findings are not in line with the predictions based on our alternative account of the findings from Carpenter's second experiment. Reasoning from this account, we expected to replicate Carpenter's finding in the list with the strong mediator-cue associations. In addition, with respect to the list with no mediator-cue associations, we predicted similar testing effects for the mediator final test cues and the related final test cues. However, the findings from Experiment 1 are also inconsistent with the semantic mediator hypothesis. According to this hypothesis mediator final test cues ought to produce a larger testing effect than related final test cues both in the strong mediator-cue association list and in the no mediator-cue association list. The outcomes of Experiment 1, which failed to corroborate the semantic mediator hypothesis, casts some doubt on the reliability of Carpenter's [11] results. This doubt was amplified because Carpenter's second experiment had a 2 × 2 between subjects design with only 10 participants per cell. Such a small sample is problematic because all other things being equal (i.e., alpha level, effect size and the probability of the null hypothesis being true), the probability that a significant result reflects a Type-1 error increases with a smaller sample size [25]. Consequently, it is important to assess the replicability of Carpenter's findings. To this aim, we conducted a replication of Carpenter's experiment, using the same procedure and learning materials.

Experiment 2

Methods Participants

One hundred seventy-three (173) United States residents who had not participated in Experiment 1 completed the experiment via MTurk (http://www.mturk.com). Participants were randomly assigned to conditions of the factorial design mentioned below. They were paid $1.60 for their participation. Eight participants were excluded from further analysis because their native language was not English, leaving 165 participants (99 females, 66 males, age 18-67, mean age 34.6, SD = 12.2). Of these participants, 82 learned the word pairs through restudy and 83 learned the word pairs through retrieval practice. Forty-four participants in the restudy condition and 47 participants in the retrieval practice condition completed the final test with mediator cues. Thirty-eight participants in the restudy condition and 36 participants in the retrieval practice condition completed the final test with related cues.

Materials and design

We used a 2 (learning condition: restudy vs. retrieval practice) × 2 (final test condition: mediator vs. related) between-subjects design. Participants studied the same word pairs Carpenter [11] used (see Appendix B). The experiment was programmed and run in Qualtrics [24].

Procedure

The procedure was identical to that of Experiment 1.

Results and discussion

Working conditions

The three statements about the current working environment of the participants were rated as follows: 'I'm in a noisy environment': mean rating 1.35 (SD = 0.59), 'there are a lot of distractions here': mean rating 1.38 (SD = 0.57), 'I'm in a busy environment': mean rating 1.32 (SD = 0.66). The statements at the end of the experiments were rated as follows: 'I only participated in this experiment to earn money': mean rating 3.25 (SD = 1.2), 'I found the experiment interesting': mean rating 3.88 (SD = 1.01),'The experiment was boring': mean rating 2.58 (SD = 1.14), 'The experiment was difficult': mean rating 3.45 (SD = 1.14), 'I really tried to remember the word pairs': mean rating 4.71 (SD = 0.52), 'I was distracted during the experiment': mean rating 1.63 (SD = 0.89).

To make sure the working conditions of the MTurk workers resembled those of participants in the lab as much as possible, we only included those participants in the subsequent analyses who scored 1 or 2 on the last question (i.e., "I was distracted during the experiment"). The resultant sample consisted of 141 participants.

Intervening test

On the intervening test, participants correctly retrieved .89 (SD = .19) of the targets on average in the related final test cue condition, and .93 (SD = .17) in the mediator final test condition.

Final test

The fourth row of Table 1 shows the proportion correctly recalled targets on the final test per condition. A 2 (learning condition: restudy vs. retrieval practice) × 2 (final test cue: mediator vs. related) between-subjects ANOVA with the proportion correctly recalled final test targets as dependent variable yielded a small but significant main effect of learning condition, F(1,137) = 6.914, p = .010, η 2 p = .048, indicating that final test performance was better for retrieved than restudied word pairs (i.e., a testing effect), and a small main effect of final test cue, F(1,137) = 8.852, p = .003, η 2 p = .069, indicating better final test performance with related cues than with mediator cues. There was a very small nonsignificant Learning Condition × Final Test Cue interaction, F(1,137) = 0.067, p = .796, η 2 p < .001, indicating that the effect of learning condition did not differ between final test cue conditions. Furthermore, and contrary to Carpenter's [11] results, the testing effect for mediator cues was numerically even smaller than for related cues.

In sum, the results from our Experiment 2 are inconsistent with Carpenter's [11] second experiment, and with the semantic mediator hypothesis for that matter. However, our sample was drawn from a different population than Carpenter's sample, and although there is no reason to expect that this should matter it might be possible that the effect under interest is much smaller or even absent in the population of MTurk workers. Alternatively, it might be that there is a meaningful effect in the MTurk population but that we were unlucky enough to stumble on an extreme sample and our results reflect a Type II error. To gain insight into what happened, we aimed to assess the robustness of our findings by conducting a replication of our Experiment 2 and hence of Carpenter's original experiment.

Experiment 3

Methods Participants

One hundred eighteen (118) United States residents who had not participated in Experiment 1 or Experiment 2 completed the experiment via MTurk (http://www.mturk.com). Participants were randomly assigned to conditions. They were paid $1.33 for their participation. Two participants were excluded from further analysis because their native language was not English, leaving 116 participants (78 females, 38 males, age 19-67, mean age 33.4, SD = 11.9). Of these participants, 59 learned the word pairs through restudy and 57 learned the word pairs through retrieval practice. Thirty participants in the restudy condition and 26 participants in the retrieval practice condition completed the final test with mediator cues. Twenty-nine participants in the restudy condition and 31 participants in the retrieval practice condition completed the final test with related cues.

Materials, design, procedure

Materials, design, and procedure were the same as in Experiment 2.

Results and discussion

Working conditions

The three statements about the current working environment of the participants were rated as follows: 'I'm in a noisy environment': mean rating 1.48 (SD = 0.74), 'there are a lot of distractions here': mean rating 1.44 (SD = 0.62), 'I'm in a busy environment': mean rating 1.40 (SD = 0.8). The statements at the end of the experiments were rated as follows: 'I only participated in this experiment to earn money': mean rating 3.56 (SD = 1.11), 'I found the experiment interesting': mean rating 3.79 (SD = 0.99), 'The experiment was boring': mean rating 2.85 (SD = 1.21),'The experiment was difficult': mean rating 3.37 (SD = 1.11), 'I really tried to remember the word pairs': mean rating 4.68 (SD = 0.54), 'I was distracted during the experiment': mean rating 1.78 (SD = 0.99).

As in Experiment 1 and 2, we only included participants in the subsequent analyses who scored 1 or 2 on the latter question. This led to a final sample of 95 participants.

Intervening test

On the intervening test, participants correctly retrieved .94 (SD = .12) of the targets in the related final test cue condition and .95 (SD = .09) in the mediator final test cue condition.

Final test

The fifth row of Table 1 shows the proportion correctly recalled targets on the final test per condition. A 2 (learning condition: restudy vs. retrieval practice) × 2 (final test cue: mediator vs. related) between-subjects ANOVA on these proportions yielded a small significant main effect of learning condition, F(1,80) = 4.935, p = .029, η 2 p = .058, indicating that final test performance was better for retrieved than restudied word pairs (i.e., a testing effect).

There was a small significant main effect of final test cue, F(1,80) = 4.255, p = .042, η 2 p = .051, indicating that performance was better for mediator than for related final test cues. Furthermore, there was a small significant Learning Condition × Final Test Cue interaction, F(1,80) = 6.606, p = .012, η 2 p = .076, indicating that the effect of learning condition (i.e., the testing effect) was larger for mediator than for related final test cues. This pattern is consistent with Carpenter's [11] pattern although the mediator testing effect advantage was much smaller in the current experiment than in Carpenter's study.

Small-scale meta-analyses

The present study resulted in four estimates of the interaction effect between learning condition (retrieval practice vs. restudy) and final test cue (mediator vs. related): two in Experiment 1, and one each in Experiments 2 and 3. The estimates of the interaction effect revealed a larger testing effect for mediator cues than for related cues in two cases (i.e., in the no-mediator-cue association list of Experiment 1, and in Experiment 3), whereas Experiment 2 and the strong mediator-cue association list in Experiment 1 demonstrated a reversed pattern. With the exception of Experiment 3, regardless of the direction, the observed interaction effects appeared to be smaller than in Carpenter's [11] second experiment.

However, we obtained our results with MTurk participants through online experiments whereas Carpenter's [11] original findings were obtained in the psychological laboratory with undergraduate students. To examine whether the experimental setting (MTurk/online vs. psychological laboratory) might be associated with the interaction between cue type (mediator vs. related) and the magnitude of the testing effect, we conducted two small-scale meta-analyses (see [26,27]) in which we included the findings from Carpenter's original study as well as findings from four highly similar unpublished experiments we were aware of (i.e., two by Rawson, Vaughn, & Carpenter [28], and two by Brennan, Cho, & Neely [29]).

The two experiments by Rawson and colleagues (see Appendix B of their paper) used Carpenter's 16 original word pairs plus 20 new word pairs. Their experimental procedure was identical to Carpenter's original procedure. Yet, contrary to Carpenter's entirely betweensubjects experiment, Rawson and colleagues' experiments had a 2 Final Test Cue (mediator vs. related) × 2 Learning (restudy vs. testing) mixed design with repeated measures on the first factor.

Brennan and colleagues used two sets of materials in their experiment: Carpenter's original materials (Set A) and a set of new materials (Set B). Participants learned both sets of materials according to Carpenter's original procedure with restudy and retrieval practice being manipulated between subjects and with a final test involving only mediator cues. Table 1 provides further information on the studies included in the small-scale meta-analyses as well as relevant descriptive statistics. It should be noted that all experiments in Table 1 employed extralist final test cues, i.e., cues not presented during the learning phase, which is not a standard procedure in testing effect research In addition, the final tests were always administered after a relatively short retention interval, while the testing effect usually only emerges after a long retention interval. However, apart from the related cue condition in our Experiment 3, the mean performance for items learned through testing is numerically better than the mean performance for items learned through restudy regardless of whether the final test involves mediator cues or related cues. Consequently, it seems that these extralist final test cues can reliably produce short-term testing effects. Furthermore, the standard deviations of the final test scores tend to be larger for the MTurk experiments than for the Lab experiments. To the extent that these standard deviations reflect error variance, this shows that the error variance is larger in the MTurk experiments than in the Lab experiments: a finding that does not come as a surprise given that the MTurk participants completed the experiments in less standardized settings (which leads to more unsystematic variance in final test scores) than participants in a psychological laboratory. Figure 2 presents the mean advantage of testing over restudying and the 95 % Confidence Interval (CI) of the mean for each experiment from Table 1 for mediator final test cues. Two random-effects meta-analyses were conducted to estimate the combined mean testing effect for lab experiments (i.e., estimation based on Carpenter However, the estimation accuracy (width of the CI) is somewhat higher in the Lab experiments than in MTurk. Furthermore, the heterogeneity index Q indicates that the variance in the four MTurk testing effects can be attributed to sampling error, Q(3) = 2.520, p = .471. By contrast, the five Lab testing effects showed some heterogeneity, Q(4) = 9.004, p = .06, suggesting that the samples might have been drawn from populations with different mean testing effects. However, these heterogeneity indices should be considered with extreme caution because they are based on a very small sample of studies.

Mediator-cue testing effect

Related cue testing effect Figure 3 presents the mean advantage of testing over restudying and the 95 % Confidence Interval (CI) of the mean for each experiment from Table 1 for related final test cues. The two random-effects meta-analyses suggest that (marginally) significant testing effects can be found However, the combined testing effect estimate is somewhat smaller and much more accurate (i.e., a narrower CI) in Lab experiments than in MTurk experiments. Also, there is a clear indication of heterogeneity for the MTurk testing effects, Q(3) = 10.209, p = .017, but not for the Lab testing effects, Q(2) < 1, p = .824. Again due to the small number of involved studies, these heterogeneity indices should be considered with extreme caution.

The combined means from the small-scale meta-analyses demonstrate that the short-term testing effect is larger for mediator cues than for related cues both in MTurk experiments (combined mediator cue testing effect = 0.153; combined related cue testing effect = 0.105) and in Lab experiments (combined mediator cue testing effect = 0.129; combined related cue testing effect = 0.070). Furthermore, the mediator testing effect advantage is about 5 % points in MTurk experiments and in Lab experiments. However, the testing effect for related cues appears to vary substantially across MTurk experiments and this makes it more difficult to find a Learning (restudy vs. retrieval practice) × Final Test Cue (mediator vs. related) interaction effect.

General discussion

Direct association hypothesis

Recently, Carpenter [11] proposed that when people learn cue-target (C-T) pairs they are more likely to activate semantic mediators (M) during retrieval practice than during restudy. In turn, due to this mediator activation, retrieval practice is assumed to strengthen the M-T link more than restudying. Hence, if people receive mediator cues during the final test, the probability of coming up with the correct target will be higher following retrieval practice than following restudy. Also, this testing effect will be smaller when related words are used as cues during the final test, which were presumably not activated during retrieval practice. Consistent with these predictions, Carpenter found in her second experiment that the testing effect was indeed larger for mediator cues than for related cues. However, it might be possible that retrieval practice does in fact not strengthen the M-T link but only the C-T link. Now, if there is also a strong pre-existing association from the mediator to the cue, people will be able to reinstate the original cue (C) on the basis of a mediator final test cue. Subsequently, if retrieval practice strengthens the C-T link more than restudying, the use of mediator final test cues will result in a testing effect. Furthermore, the testing effect will be smaller with related final test cues that have no (or a much smaller) pre-existing association to the original cue. This line of reasoning, which Brennan, Cho and Neely [29] dubbed the direct association hypothesis, may provide an alternative account of the findings from Carpenter's [11] second experiment because for some of her materials there were strong mediator-cue associations. To assess our alternative explanation of Carpenter's findings, we replicated Carpenter's design using cue-target pairs with no mediator-cue association (No-MC List) and cue-target pairs with strong mediator-cue associations (Strong-MC List). If Carpenter's findings arose through mediator-cue associations, her pattern of results should emerge in the Strong-MC List but not in the No-MC List. However, the results from our Experiment 1 were not in line with these predictions. In the No-MC list, we found an interaction effect that was much smaller, but similar to the effect Carpenter found, with the testing effect being larger for mediator cues than for related cues. By contrast, in the Strong-MC list, the magnitude of the testing effect was comparable for mediator and related final test cues. Hence, the findings from Experiment 1 failed  to corroborate the direct association hypothesis (see also [29]).

Direct replication attempts

We did not find empirical evidence for our alternative explanation of Carpenter's [11] result. However, our results were also not consistent with the semantic mediator account, which predicts a larger testing effect for mediator than for related final test cues for both lists. Because our findings were not consistent with this prediction, we followed up on Experiment 1 with two direct replications of Carpenter's second experiment. Before we discuss the outcomes of our experiments, we will address the power of our experiments as well as the degree of similarity between our experiments and the original one.

An important requirement for replications (but ironically notor hardly everfor original studies) is that they are performed with adequate power. To determine the sample size associated with an adequate power level, one needs to know the minimal effect size in the population that is assumed to be theoretically relevant. However, in psychological research, such an effect size is almost never provided. Carpenter's experiment is a point in case because neither the expected sizes of the two main effects (in a factorial ANOVA these effects are important since they determine in part the power associated with the interaction effect) nor the expected size of the crucial interaction effect were specified. Therefore, replicators often use the effect size in the original study for their power calculations. However, this is problematic because due to publication bias reported effect sizes are likely to overestimate the true effect size in the population (e.g., [30]). For example, in Carpenter's original experiment almost 50 % of the variance in the dependent variable was accounted for by the linear model with the two main effects and the interaction. This effect is extraordinarily large even for laboratory research.

Given the problems associated with determining the theoretically relevant minimal effect size, Simonsohn [31] proposed to infer it from the original study's sample size. The assumption is the original researcher(s) drew their sample to have at least some probability to detect an effect if there is actually an effect in the population. Simonsohn suggests but he admits this is arbitrarythat the intended power of studies was at least 33 %. If we assume the original study had an intended power of 33 %, and given the original study's sample size n, it is possible to determine the minimally relevant effect size. Simonsohn denotes this effect size as d33%. A replication should be powerful enough to allow for an informative failure; this means it should be able to demonstrate that the effect of interest is smaller than the minimally relevant effect size d33%. Simonsohn shows through a mathematical derivation that the required n "to make the replication be powered at 80 % to conclude it informatively failed, if the true effect being studied does not exist" (page 16 of the supplement; [31]) is approximately 2.5 times the original sample size. Therefore, a replication attempt of Carpenter's [11] second experiment would require at least 2.5*40 = 100 participants. Experiment 2 and Experiment 3 of the present study had respectively 141 and 95 participants, so they met Simonsohn's criterion for an adequately powered study.

The present experiments were set up as direct replications meaning that we tried to reinstate the methods and materials of the original experiment as closely as possible. However, there are always differences between an original experiment and a replication, even when the original researcher carries out the replication. An important question in the evaluation of replication attempts is whether existing differences render a replication uninformative regarding the reproducibility of the original results. In our view, the answer to this question depends on the strengths of the theoretical and/or practical arguments as to why the differences should matter. With respect to our experiments, one might note that testing participants online is problematic because it increases the unsystematic variance as compared to testing participants in the psychological laboratory. However, if more unsystematic variance is the only problemimplying that the raw effect of interest is the same online as in the laboratorythen it can be easily resolved by testing more participants than in the original study. We reasoned that a direct replication in addition to the original materials and procedure would require English-speaking participants who are not distracted while doing the task. Our experiments meet these criteria at least if we assume we can trust participants' self-reports on their native language and on the conditions under which they did the experiment (another way to possibly reduce the variability would be to exclude participants based on for example catch trials or variability of response latencies, which unfortunately was not possible with our data because we did not include catch trials and could not reliably measure response latencies). Nevertheless, other researchers might hold other criteria for evaluating the comparability between our experiments and the original. The easiest way to resolve issues pertaining to comparability is to require researchers to argue (and not simply report without elaboration) in their papers for a range of tolerances on the method and sample parameters of their experiments. The more restrictive they are, the more they reduce the generality and scopeand consequently the interestof their claims. Hence, researchers would be encouraged to be as liberal as possible in their methods parameters in order to increase the generality of their effect. Furthermore, if researchers routinely specify a range of allowable method and sample parameters it would become very easy to determine whether a direct replication attempt would qualify as such.

Thus, the direct replications of Carpenter's [11] experiment, i.e., our Experiments 2 and 3 were adequately powered and in our view they should be considered as methodologically valid direct replications attempts. The results of the experiments were mixed. Experiment 2 was largely inconsistent with Carpenter's original experiment whereas Experiment 3 clearly reinforced Carpenter's findings. It is not clear however whether the inter-experiment variability reflects the operation of an unknown moderator to the interaction effect or whether the sample was extreme in one of the experiments (or in both but that would be unlikely).

Small-scale meta-analyses

Taken together, the results of the present series of experiments were mixed. We found patterns similar to the results of Carpenter [11] in the No-MC list of Experiment 1 and in Experiment 3, but not in the Strong-MC list of Experiment 1 and in Experiment 2. However, our experiments were conducted online with MTurk participants, whereas Carpenter tested undergraduate psychology students in the laboratory. To examine whether this might have yielded different outcomes, we used small-scale meta-analyses to calculate combined estimates of the mean testing effect for related cues and for mediator cues both in online experiments (i.e., the four experiments from the present study) and laboratory experiments (i.e., Carpenter original experiment and four similar experiments). The outcomes of these analyses consistently revealed short term testing effects for mediator cues and related test cues. More important, however, was the finding that the mediator testing effect advantage is about 5 % points in both online experiments and in laboratory experiments. Hence, the raw mediator testing effect advantage is highly similar in online and laboratory settings. It should be noted though that this raw advantage is much smaller than in Carpenter's original experiment, which revealed a mediator testing effect advantage of 23 % points.

In addition, we found that the mean testing effect for related cues varied considerably across online experiments, but much less across laboratory experiments. As a result, it may be more difficult to find mediator testing advantages in online experiments than in laboratory experiments. Further research needs to be conducted to assess whether the related-cue testing effect variability reflects regular random sample fluctuation or the operation of moderators. Should the latter be the case, this will either spur the further development of the semantic mediator hypothesis of the testing effect or it might lead to the refutation of the hypothesis in favor of an alternative (e.g., [4,32,33]).

Conclusions

The experiments in the present study can be seen as conceptual (Experiment 1) and exact (Experiments 2 and 3) replications of Carpenter's [11] original experiment. Recently, replication of results from psychological research has received a lot of attention (e.g., [34]) and most researchers would probably agree that replications are important. However, replication attempts are scarce and if they are performed, they are hard to publish [35,36]. This is unfortunate, because replications inform researchers in a field about the extent to which a finding remains stable across similar experiments [35]. The current paper does exactly that and the tentative conclusions are that (1) related cues and mediator cues produce reliable short-term testing effects, (2) the magnitude of the raw mediator testing effect advantage is comparable for online and laboratory experiments, (3) in both online and laboratory experiments the magnitude of the raw mediator testing effect advantage is smaller than in Carpenter's [11] original experiment and (4) the testing effect for related cues varies considerably between online experiments. This variability might be theoretically relevant if it points towards moderators of the related cue short-term testing effect. Furthermore, the findings of the present study are methodologically relevant to researchers who aim to build on Carpenter's original findings: when designing their experiments, they should keep in mind that the raw mediator testing effect advantage is much smaller than in Carpenter's experiment and that the mediator testing effect advantage may vary particularly in online samples.

Ethics approval and consent to participate

The following ethics statement applies to all experiments in the present study. In Dutch legislation the law on medical-scientific research on humans (Wet Medisch Wetenschappelijk Onderzoek met mensen; WMO) protects people from maltreatment and experimentation. The WMO applies to research in which people are submitted to a medical or physical intervention, or to research in which a certain mode of behavior is imposed on people. According to the WMO, approval from an ethics committee is not required for certain strictly behavioral studies (note that it is almost always required for studies involving a medical or physical intervention).

We consulted the chair of the Ethics Committee Psychology of the Erasmus University Rotterdam, the Netherlands, to determine whether a formal approval of the current study was required. She concluded that a formal approval by the Ethics Committee was not necessary because the procedure was noninvasive, participants were given full disclosure of the experimental procedure, they received a payment proportionate to the task at hand, and the results of the experiments were analyzed anonymously.

The participants in all experiments were United States citizens who voluntarily subscribed for online participation in the described experiments. We did not obtain written informed consent from the participants.

Fig. 1
1Word associations in Experiment 1.

Exp2 through Brennan et al. Set B) and for MTurk experiments (i.e., estimation based on Coppens et al.'s experiments). The estimates are presented as combined effects in Fig. 2, and they show comparable (in terms of mean difference and statistical significance) testing effects in Lab experiments (Combined M = 0.129, 95 % CI [0.066; 0.192]) and in MTurk experiments (Combined M = 0.153, 95 % CI [0.073; 0.232].

in Lab experiments (Combined M = 0.070, 95 % CI [0.019; 0.121]) and in MTurk experiments (Combined M = 0.105, 95 % CI [−0.005; 0.213].

Fig. 2
2Forest plot of the 95 % confidence intervals of the mean testing advantage (final test proportion correct for tested pairsfinal test proportion correct for restudied pairs) obtained with mediator final test cues for the Lab experiments (Carpenter Exp2 through Brennan et al. Set B) and the MTurk experiments (Coppens et al. Exp1 No-Mc through Coppens et al. Exp3). The combined estimates for the Lab Experiments and the MTurk experiments and the 95 % confidence intervals are also presented

Fig. 3
3Forest plot of the 95 % confidence intervals of the mean testing advantage (final test proportion correct for tested pairsfinal test proportion correct for restudied pairs) obtained with related final test cues for the Lab experiments (Carpenter Exp2 through Rawson et al. Exp2) and the MTurk experiments (Coppens et al. Exp1 No-Mc through Coppens et al. Exp3). The combined estimates for the Lab Experiments and the MTurk experiments and the 95 % confidence intervals are also presented

Table 1
1Setting, Design, Sample Size and Results of the Experiments in the Small-Scale Meta AnalysesExperiment
Setting Design
Total
n

M testing
mediator
(SD)

M restudy
mediator
(SD)

M testing
related
(SD)

M restudy
related (SD)

Coppens et al.
Exp1 No-Mc

Online 2 retrieval cue (mediator vs. related) × 2 learning (restudy
vs. testing) between subjects

87
0.26 (0.26)
0.13 (0.24)
0.21 (0.21) 0.16 (0.17)

Coppens et al.
Exp1 Strong Mc

Online 2 retrieval cue (mediator vs. related) × 2 learning (restudy
vs. testing) between subjects

94
0.50 (0.46)
0.40 (0.38)
0.38 (0.23) 0.14 (0.13)

Coppens et al.
Exp2

Online 2 retrieval cue (mediator vs. related) × 2 learning (restudy
vs. testing) between subjects

141
0.36 (0.31)
0.24 (0.25)
0.50 (0.27) 0.37 (0.26)

Coppens et al.
Exp3

Online 2 retrieval cue (mediator vs. related) × 2 learning (restudy
vs. testing) between subjects

95
0.57 (0.33)
0.29 (0.27)
0.31 (0.21) 0.32 (0.24)

Carpenter 2011
Exp2

Lab
2 retrieval cue (mediator vs. related) × 2 learning (restudy
vs. testing) between subjects

40
0.58 (0.23)
0.23 (0.12)
0.29 (0.18) 0.18 (0.16)

Rawson et al.
Appendix B long
lag

Lab
2 retrieval cue (mediator vs. related) × 2 learning (restudy
vs. testing) mixed with retrieval cue within subjects

65
0.28 (0.25)
0.15 (0.19)
0.18 (0.17) 0.11 (0.15)

Rawson et al.
Appendix B short
lag

Lab
2 retrieval cue (mediator vs. related) × 2 learning (restudy
vs. testing) mixed with retrieval cue within subjects

63
0.28 (0.26)
0.12 (0.18)
0.15 (0.18) 0.09 (0.12)

Brennan, Cho &
Neely Set A

Lab
Mediator cue only, learning (restudy vs. testing)
manipulated between subjects

68
0.27 (0.20)
0.19 (0.16)

Brennan, Cho &
Neely Set B

Lab
Mediator cue only, learning (restudy vs. testing) between
subjects

68
0.14 (0.12)
0.06 (0.08)

Table 3
3Stimuli used in Experiments 2 and 3Cue
Target
Mediator
Related
C-T
C-M
M-C
R-T

Weapon
Knife
Gun
Ax
0,075
0,592
0,024
0,046

Coffee
Table
Tea
Banquet
0,020
0,442
0,369
0,020

Mother
Child
Father
Birth
0,010
0,597
0,706
0,015

Soil
Earth
Dirt
Continent
0,040
0,717
0,055
0,041

Availability of data and materialsThe datasets supporting the conclusions of this article are available in the Open Science Framework repository https://osf.io/dxwz4/.The materials used in the experiments described in this article are included within the article (and its Appendix A and Appendix B). Appendix BAppendix AAbbreviations C-T: cue-target; M-C: mediator-cue; M-T: mediator-target; MTurk: Amazon Mechanical Turk.Competing interestsThe authors declare that they have no competing interests.Authors' contributions LC conceptualized and designed the study, collected the data, performed statistical analyses and interpretation of data, prepared the draft manuscript and critically revised the manuscript. PV performed statistical analyses and interpretation of data, prepared the draft manuscript and critically revised the manuscript. SB contributed to analysis and interpretation of data and provided input into the draft and revised manuscript. RR contributed to interpretation of data and critically revised the manuscript. All authors read and approved the final manuscript. Note. C-T indicates cue-to-target association strength, C-M indicates cue-to-mediator association strength, M-C indicates mediator-to-cue association strength, and R-T indicates related-to-target association strength. Mediator-to-target association strength and related-to-cue association strength was always 0• We accept pre-submission inquiries • Our selector tool helps you to find the most relevant journal Submit your next manuscript to BioMed Central and we will help you at every step:
Tests enhance retention and transfer of spatial learning. S K Carpenter, Psychon Bull Rev. 19Carpenter SK. Tests enhance retention and transfer of spatial learning. Psychon Bull Rev. 2012;19:443-8.

Spacing and testing effects: A deeply critical, lengthy, and at times discursive review of the literature. P F Delaney, Ppjl Verkoeijen, A Spirgel, Psychol. Learn. Motiv. Adv. Res. Theory. Ross BHDelaney PF, Verkoeijen PPJL, Spirgel A. Spacing and testing effects: A deeply critical, lengthy, and at times discursive review of the literature. In: Ross BH, editor. Psychol. Learn. Motiv. Adv. Res. Theory. 2010. p. 63-147.

Retrieval-based learning: Active retrieval promotes meaningful learning. J D Karpicke, Curr Dir Psychol Sci. 21Karpicke JD. Retrieval-based learning: Active retrieval promotes meaningful learning. Curr Dir Psychol Sci. 2012;21:157-63.

Retrieval-based learning: An episodic context account. J D Karpicke, M Lehman, W R Aue, Psychol. Learn. Motiv. Ross BH61Karpicke JD, Lehman M, Aue WR. Retrieval-based learning: An episodic context account. In: Ross BH, editor. Psychol. Learn. Motiv, vol. 61. 2014. p. 237-84.

The critical role of retrieval practice in long-term retention. Iii Roediger, Hl, A C Butler, Trends Cogn Sci. 15Roediger III HL, Butler AC. The critical role of retrieval practice in long-term retention. Trends Cogn Sci. 2011;15:20-7.

The power of testing memory: Basic research and implications for educational practice. Iii Roediger, Hl, J D Karpicke, Perspect Psychol Sci. 1Roediger III HL, Karpicke JD. The power of testing memory: Basic research and implications for educational practice. Perspect Psychol Sci. 2006;1:181-210.

The effect of testing versus restudy on retention: A metaanalytic review of the testing effect. C A Rowland, Psychol Bull. 140Rowland CA. The effect of testing versus restudy on retention: A meta- analytic review of the testing effect. Psychol Bull. 2014;140:1432-63.

About practice: Repetition, spacing, and abstraction. T C Toppino, E Gerbier, Psychol. Learn. Motiv. Ross BHToppino TC, Gerbier E. About practice: Repetition, spacing, and abstraction. In: Ross BH, editor. Psychol. Learn. Motiv. 2014. p. 113-89.

Cue strength as a moderator of the testing effect: The benefits of elaborative retrieval. S K Carpenter, J Exp Psychol Learn Mem Cogn. 35Carpenter SK. Cue strength as a moderator of the testing effect: The benefits of elaborative retrieval. J Exp Psychol Learn Mem Cogn. 2009;35: 1563-9.

A short-term testing effect in cross-language recognition. Ppjl Verkoeijen, S Bouwmeester, G Camp, Psychol Sci. 23Verkoeijen PPJL, Bouwmeester S, Camp G. A short-term testing effect in cross-language recognition. Psychol Sci. 2012;23:567-71.

Semantic information activated during retrieval contributes to later retention: Support for the mediator effectiveness hypothesis of the testing effect. S K Carpenter, J Exp Psychol Learn Mem Cogn. 37Carpenter SK. Semantic information activated during retrieval contributes to later retention: Support for the mediator effectiveness hypothesis of the testing effect. J Exp Psychol Learn Mem Cogn. 2011; 37:1547-52.

Why testing improves memory: Mediator effectiveness hypothesis. M A Pyc, K A Rawson, Science. 330Pyc MA, Rawson KA. Why testing improves memory: Mediator effectiveness hypothesis. Science. 2010;330:335-5.

Why is test-restudy practice beneficial for memory? An evaluation of the mediator shift hypothesis. M A Pyc, K A Rawson, J Exp Psychol Learn Mem Cogn. 38Pyc MA, Rawson KA. Why is test-restudy practice beneficial for memory? An evaluation of the mediator shift hypothesis. J Exp Psychol Learn Mem Cogn. 2012;38:737-46.

The University of South Florida word association, rhyme, and word fragment norms. D L Nelson, C L Mcevoy, T A Schreiber, Nelson DL, McEvoy CL, Schreiber TA. The University of South Florida word association, rhyme, and word fragment norms [Internet]. 1998. Available from: http://w3.usf.edu/FreeAssociation/.

The influence of retrieval on retention. M Carrier, H Pashler, Mem Cognit. 20Carrier M, Pashler H. The influence of retrieval on retention. Mem Cognit. 1992;20:633-42.

When does testing enhance retention? A distribution-based interpretation of retrieval as a memory modifier. V Halamish, R A Bjork, J Exp Psychol Learn Mem Cogn. 37Halamish V, Bjork RA. When does testing enhance retention? A distribution-based interpretation of retrieval as a memory modifier. J Exp Psychol Learn Mem Cogn. 2011;37:801-12.

Mnemonic benefits of retrieval practice at short retention intervals. C A Rowland, E L Delosh, Memory. 23Rowland CA, DeLosh EL. Mnemonic benefits of retrieval practice at short retention intervals. Memory. 2015;23:403-19.

Collecting response times using Amazon Mechanical Turk and Adobe Flash. T Simcox, J A Fiez, Behav Res Methods. 46Simcox T, Fiez JA. Collecting response times using Amazon Mechanical Turk and Adobe Flash. Behav Res Methods. 2014;46:95-111.

Why interleaving enhances inductive learning: The roles of discrimination and retrieval. M S Birnbaum, N Kornell, E L Bjork, R A Bjork, Mem Cognit. 41Birnbaum MS, Kornell N, Bjork EL, Bjork RA. Why interleaving enhances inductive learning: The roles of discrimination and retrieval. Mem Cognit. 2013;41:392-402.

Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. Mjc Crump, J V Mcdonnell, T M Gureckis, Plos One. 857410Crump MJC, McDonnell JV, Gureckis TM. Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. Plos One. 2013;8, e57410.

Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. L Germine, K Nakayama, B C Duchaine, C F Chabris, G Chatterjee, J B Wilmer, Psychon Bull Rev. 19Germine L, Nakayama K, Duchaine BC, Chabris CF, Chatterjee G, Wilmer JB. Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychon Bull Rev. 2012;19:847-57.

Is spacing really the "friend of induction. Ppjl Verkoeijen, S Bouwmeester, Front Psychol. 5259Verkoeijen PPJL, Bouwmeester S. Is spacing really the "friend of induction"? Front Psychol. 2014;5:259.

Revisiting mental simulation in language comprehension: Six replication attempts. R A Zwaan, D Pecher, Plos One. 751382Zwaan RA, Pecher D. Revisiting mental simulation in language comprehension: Six replication attempts. Plos One. 2012;7, e51382.

Qualtrics (version 31984). Qualtrics Labs, IncProvo, UTQualtrics (version 31984). Provo, UT: Qualtrics Labs, Inc. 2012.

Sailing from the seas of chaos into the corridor of stability practical recommendations to increase the informational value of studies. D Lakens, Erk Evers, Perspect Psychol Sci. 9Lakens D, Evers ERK. Sailing from the seas of chaos into the corridor of stability practical recommendations to increase the informational value of studies. Perspect Psychol Sci. 2014;9:278-92.

Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. G Cumming, New York, NY: RoutledgeCumming G. Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY: Routledge; 2012.

The New Statistics: Why and How. G Cumming, Psychol Sci. 25Cumming G. The New Statistics: Why and How. Psychol Sci. 2014;25:7-29.

Does the benefit of testing depend on lag, and if so, why? Evaluating the elaborative retrieval hypothesis. K A Rawson, K E Vaughn, S K Carpenter, Mem Cognit. 43Rawson KA, Vaughn KE, Carpenter SK. Does the benefit of testing depend on lag, and if so, why? Evaluating the elaborative retrieval hypothesis. Mem Cognit. 2015;43:619-33.

The role of mediators in the testing effect in paired-associate learning. M K Brennan, K W Cho, J H Neely, Toronto, Brennan MK, Cho KW. Neely JH. Toronto: The role of mediators in the testing effect in paired-associate learning; 2013.

A vast graveyard of undead theories: Publication bias and psychological science's aversion to the null. C J Ferguson, M Heene, Perspect Psychol Sci. 7Ferguson CJ, Heene M. A vast graveyard of undead theories: Publication bias and psychological science's aversion to the null. Perspect Psychol Sci. 2012;7:555-61.

Small telescopes: Detectability and the evaluation of replication results. U Simonsohn, Psychol Sci. 26Simonsohn U. Small telescopes: Detectability and the evaluation of replication results. Psychol Sci. 2015;26:559-69.

Toward an episodic context account of retrieval-based learning: Dissociating retrieval practice and elaboration. M Lehman, M A Smith, J D Karpicke, J Exp Psychol-Learn Mem Cogn. 40Lehman M, Smith MA, Karpicke JD. Toward an episodic context account of retrieval-based learning: Dissociating retrieval practice and elaboration. J Exp Psychol-Learn Mem Cogn. 2014;40:1787-94.

Separate mnemonic effects of retrieval practice and elaborative encoding. J D Karpicke, M A Smith, J Mem Lang. 67Karpicke JD, Smith MA. Separate mnemonic effects of retrieval practice and elaborative encoding. J Mem Lang. 2012;67:17-29.

Editors' introduction to the special section on replicability in psychological science: A crisis of confidence?. H Pashler, E-J Wagenmakers, Perspect Psychol Sci. 7Pashler H, Wagenmakers E-J. Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspect Psychol Sci. 2012;7:528-30.

Rewarding replications: A sure and simple way to improve psychological science. S L Koole, D Lakens, Perspect Psychol Sci. 7Koole SL, Lakens D. Rewarding replications: A sure and simple way to improve psychological science. Perspect Psychol Sci. 2012;7:608-14.

Replications in psychology research: How often do they really occur?. M C Makel, J A Plucker, B Hegarty, Perspect Psychol Sci. 7Makel MC, Plucker JA, Hegarty B. Replications in psychology research: How often do they really occur? Perspect Psychol Sci. 2012;7:537-42.


BMC Pediatrics The surged faradic stimulation to the pelvic floor muscles as an adjunct to the medical management in children with rectal prolapse

Simmi K Ratan
Department of Pediatric Surgery
Post Graduate Institute of Medical Sciences
RohtakHaryanaIndia

Nain Kamal
Rattan firstname.lastname@example.com
Department of Pediatric Surgery
Post Graduate Institute of Medical Sciences
RohtakHaryanaIndia

Poonam Jhajhria
Department of Physiotherapy
Post Graduate Institute of Medical Sciences
RohtakHaryanaIndia

Yogesh Parshad Mathur
Department of Physiotherapy
Post Graduate Institute of Medical Sciences
RohtakHaryanaIndia

Atul Jhanwar
Department of Pediatric Surgery
Post Graduate Institute of Medical Sciences
RohtakHaryanaIndia

Dimple Kondal
Department of Physiotherapy
Post Graduate Institute of Medical Sciences
RohtakHaryanaIndia

Department of Biostatistics
Institute of Medical Sciences
All IndiaNew DelhiIndia

BMC Pediatrics The surged faradic stimulation to the pelvic floor muscles as an adjunct to the medical management in children with rectal prolapse
10.1186/1471-2431-9-44BioMed Central Page 1 of 6 (page number not for citation purposes) Research article Dimple Kondal -email@example.com * Corresponding author
Background:To assess the role of the surged faradic stimulation to the pelvic floor muscles as an adjunct to the conservative management in the children of idiopathic rectal prolapseMethods: Study design: ProspectiveSetting: Pediatric Surgery Department, Pt BD Sharma, Post Graduate Institute of Medical Sciences, Rohtak Subjects: 47 consecutive children with idiopathic rectal prolapse attending the Pediatric Surgery out patient department from Methodology: The information pertaining to duration and the extent of rectal prolapse, predisposing or associated medical conditions, results of local clinical examination were noted. Surged faradic stimulation using modified intraluminal rectal probe, was given on the alternate days. The conventional conservative medical management was also continued. The extent of relief and the number of the sittings of faradic stimulation required were noted at various stages of follow-ups Statistical Methods: Mean values between those completely cured and others; poor responders and others were compared with t-test and proportions were compared with Chi square test. The pvalue < 0.05 was considered statistically significant.Results:The mean number of sittings in the completely cured group (n = 28(64%)) was (12.4 ± 7.8) and was comparable with very poor responder (n = 6(13%). There was higher percentage of relief (76%) at the first follow up (at 15 days) in completely cured Vs other (37%) and also the poor responders showed (20%) Vs other (68%) and was statistically significant.Conclusion:With use of faradic stimulation, even the long-standing rectal prolapse can be fully cured. The follow up visit at 2 weeks is very important to gauge the likely success of this modality in treatment of the patients with rectal prolapse. Those showing poor response at this stage may require alternative treatment or take a long time to get cured

Background

Rectal prolapse is a common condition in children in the underdeveloped nations, albeit with unclear etiology. The commonest predisposing factors are the diseases leading to malnutrition (e.g. amebiasis, giardiasis, worms) leading to disappearance of the ischio-rectal fat and causing lack of rectal support. A persistently increased intraabdominal pressure e.g. due to constipation, frequent cough, has also been implicated. Among the anatomical factors that had to the development of rectal prolapse in children are the vertical course of rectum, a low-position of rectum in relation to other pelvic organs and lack of levator support [1,2]. There are other authors who believe that the rectal prolapse in children mainly involves the mucosa; and the muscle coats are either not involved or are involved at a later stage [3]. This reason is forwarded to explain the noted higher frequency of rectal prolapse in infants [4]. It is, therefore, on this ground that the prolapse in children is supposedly different from the adults (as in adults the weakness of the pelvic floor is the predominant etiological factor) [5].

Based on these principles, the conservative treatment has remained the initial cornerstone of the rectal prolapse management in children and the interventional therapies (injection treatment and surgical operations) have been reserved for the non-responders [2,4]. By and large, the conservative treatment includes use of stools softeners, strapping of the buttocks to prevent recurrence of prolapse and defecation in lateral position [2].

In the present study, we added one more component to the prevalent conservative treatment and that is, the use of surged faradic current for stimulating external anal sphincter and the pubo-rectalis sling. This was contemplated seeing the fair results of using this current in our patients where the fecal incontinence had resulted due to poorly developed or neurologically weak pelvic floor muscles. Similar observations had been earlier reported by various workers also [6][7][8]. The surged faradic stimulus was given with the help of rectal probe that was designed so as stimulate the external anal sphincter and the weak pelvic floor muscles (when introduced passed intraluminally). The rationale of the therapy is that the strengthened external anal sphincter and pelvic floor muscles would support the rectum adequately causing amelioration of the rectal prolapse. Though initially employed in the children presenting with failed conservative management, the therapy was subsequently added as adjunct to the conservative management for every case with rectal prolapse, so as to expedite the relief. To the best of our knowledge, no similar study has been reported in the English literature till date.

Methods

Forty seven consecutive children with symptomatic rectal prolapse from July 2005 to June 2006 were included in the study. A detailed history was obtained from the parents and was adjudged to be reliable for all patients. Thereafter they were examined by the members of pediatric surgical team having more than 10 years of experience in specialty of Pediatric Surgery. Most study subjects had long standing prolapse and were awaiting their turn for the definitive surgical treatment while the surged faradic stimulation as was added as an adjunct to conservative management. The data was collected on their sex, age, duration and extent of prolapse (length and the coats of rectum involved), anal tone, the preceding illness, if any, family history, duration of conservative management and the extent of relief obtained thereby. Prior conservative management was said to have been given if the child had taken this treatment at a stretch at least for four weeks. All the early presenters (n = 12) in the study (those presenting within 2 weeks of initiation of symptoms), that is, with conservative management of lesser than 4 weeks, were considered to have taken no prior conservative management. The extent of prolapse was adjudged mainly on the subjective assessment of the reporting parent. However, an objective assessment was also done by the physiotherapist while child was being subjected to faradic stimulation. It was rated as complete if all rectal coats could be visualized through the entire circumference of the prolapsed segment. The assessment of anal tone was done subjectively at the initial examination by experienced pediatric surgeons. Only gross deviation from normal could be picked up in this manner in the outpatient setting. as invariably the children were apprehensive and tending to contract the entire pelvic floor musculature in an attempt to resist anal examination. The children were given surged faradic stimuli to the external anal sphincter and the pelvic floor muscles while continuing with the conservative management (stool softeners, high protein high caloric diet). A good reassurance and a mild sedative for more apprehensive child were enough to derive cooperation from them. The older children (5 years or more) were taught the abdominal and pelvic floor muscle exercises; whereas, for the younger children, the mothers were taught the passive maneuvers to achieve the same goal. They subjects and their parents were instructed to strictly adhere to this regime of treatment or face failure or relapse. No fixed period for this adjunct treatment was contemplated and the final response was adjudged as the one present at the 'exit point' of the patient from the study.

The procedure

The child was made to lie in prone position. Electrical stimulation was given using Timpac muscle stimulator TMS-10 with intrarectal pen electrode (our modification) [Fig 1], while inactive electrode [ fig 2] was placed under patient's pelvis The electrical stimulation was achieved with pulses of duration of 1 ms with a frequency of 30 Hz(faradic type pulses) Inactive electrode(large pad electrode) was placed under lower abdomen. Active electrode (Pen electrode) was used for the external anal sphincteric stimulation and for pelvic floor muscle stimulation. The electrode was inserted for over a short distance into the rectum(by1/4 inch or so). Intensity of the current used was variable with each patient. It was gradually increased till observable muscle contraction was observed i.e. anal wink. The patient was asked to do voluntary anal contraction with each stimulus. About 90 contractions were given in each session to avoid muscle fatigue, over 3 minutes (30 contractions per minute). Each patient was given1 session per day 3 times a week (on alternate days). Meanwhile the patient was asked to do pelvic floor exercises; mainly the tightening of the abdominal muscles, the thigh adductors and gluteals and drawing up the pelvic floor muscles for 5 sec and then rest. Other exercises included strengthening of spinal extensors and abdominal muscles. The patients were asked to do them as frequently as possible with a minimum of twice daily.

The relief was categorized as 50%, 75% or 100% depending both on the reduction in the size of the prolapsed rectum and the frequency of prolapse (spontaneous or while straining; everytime with straining or occasional), as deduced from the combined assessment done from information delivered by the parents and the examination findings of the physiotherapists The initial response at first follow up at 2 weeks of therapy was also noted. The response was gauged with respect to the average number of sittings undertaken for getting 100% cure of rectal prolapse. The first instance when the parents reported the prolapse not to have appeared even after straining was marked. It was considered as the 'end point' if the prolapse did not appear for two consecutive weeks. Any appearance of the prolapse after 2 weeks after its disappearance qualified for a 'relapse'. Thereafter, follow up visits were advised at 4, 8 and 12 week after stopping of treatment to confirm the complete amelioration. In those who failed to turn up on the due date, the same was enquired telephonically. However, in subjects who exited the therapy short of complete relief, the degree of relief and the number of sittings at the exit point were taken for computation purpose. Any telephonic report of full relief of prolapse in these subjects was ignored for the want of substantiation of the same by the supervising clinicians or physiotherapists. For the non responders (50% or less) the cause of failure was noted.

Statistical Methods

STATA version 9.0 was used. The responders and non responders were identified as two groups. The difference of mean for different parameters between the two groups was tested by 't' test and difference between the propotions by chi-square test. A p value of < 0.05 was taken as statistically significant.

Results

The mean age of subjects was 4.5 year, ranging from 4 months to 12 years. The mean duration of symptoms being 9.3 month, ranging from 1 day to 4 years. There were 12 early presenters. The extent of prolapse was partial, that is, not involving the all coats and circumference Intrarectal pen electrode used for giving faradic stimulation Figure 1 Intrarectal pen electrode used for giving faradic stimulation. Figure 2 The inactive electrode.

The inactive electrode

of rectum, in 40% (n = 19) and total in 60% (n = 28). The prior history of illness present were as: bleeding (n = 4) diarrhea (n = 14), constipation (n = 14), other illness (n = 9). The family history was positive in 9% patients (n = 4) only. The prior conservative management was taken by about 70% children (n = 35) and only 32% (n = 15) of them had shown partial response to this treatment. Most children (n = 44, 94%) had average build. A decreased anal tone was noted among 9 children (19%) only. The extent of prolapse was less than 3 cm in 64% (n = 30) and the in others it was longer than 3 cm. The symptomatic relief for associated symptoms as bleeding and pain, was quite quick with above stated regime, with all subjects getting relieved of the associated symptoms by the first follow up at 2 weeks. The children tolerated this therapy regime quite well, though towards the beginning of therapy they were apprehensive and reluctant. A painless therapy with quick amelioration of their symptoms proved to be morale boosters.

Twenty eight children responded fully; while 19 showed partial incomplete response. There were 10 children who had < 50% relief. Of them 4 had taken only 6 sittings of faradic stimuation (only 2 weeks of therapy). Only three patient had taken faradic stimulation for beyond 4 weeks and still got relieved lesser than 50% relief. Of them, two had suffered the illness for 2, 6 months respectively while the third subject had a shorter duration (10 days) The maximum number of sitting to reach 100% relief was 36; while the minimum number was 1, surprisingly. The tables showing the distribution of 'extent of relief' with respect to 'duration of the presenting symptoms' (Table 1) and the 'number of faradic stimulation sittings' (Table 2) are given. Briefly, they show a better response for subjects who had long standing rectal prolapse and were therapycompliant. In fact, 6 of 9 patients with 75% relief on therapy affirmed to have been totally cured on telephonic enquiry 12 weeks after the exit point from therapy, but the same could not be used for computation of results in the present study in absence of substantiation of the same by the supervising clinicians. No patient reported recurrence of symptoms after achieving complete relief.

The mean number of sitting in these fully cured (12.4 ± 7.8) did not differ much from those with poor response i.e. <50% cure (11.75 ± 6.9). Also, no statistically significant difference was noted between the number sittings showing < 50% response (6.7 ± 4.1) and those showing >50% response (8.4 ± 6.0). However, the former showed a higher percentage of relief at the first follow up visit (76% ± 20%) in contrast to (20 ± 24%) for the non responders at 2 weeks of beginning of the therapy. This difference was found to be significant (p = 0.01). However, as expected all non-compliance emerged out to be the strongest predictor of non-response to this treatment (p = 0.001).

Discussion

The surged faradic current has found wide application to improving the strength and vascularity of various groups of muscles including external anal sphincter [9]. In the present study, besides stimulating and strengthening the external anal sphincter, an improvision was done to stimulate the pelvic floor muscle as well. This was to take care of the causative factor of poor pelvic floor muscle support in rectal prolapse in children. As we noted that most children presented to us had average health, long duration of rectal prolapse of 9.3 month (average) that had involved all coats of rectum. Therefore, our study supports time conclusion of Fowler that the common form of rectal prolapse involve all coats of rectum [10]. However, another reason could be that the initial mucosal prolapse, over the time, involved all the layers of rectum as only 17% (n = 2 out of 12) of those early presenters (within days) had total rectal prolapse. On a wider perspective, it can be deduced that the chronically present rectal prolapse leads to weakness/atony of the pelvic floor muscles that required to be improved by their stimulation and/or exercise. Hence, inclusion of faradic stimulation of the pelvic floor muscles should form an integral part of the conservative therapy for the rectal prolapse.

We could cure 28 (64%) children fully using this strategy; of which 21 (70%) whom had received the conventional conservative management for rectal prolapse for longer than 3 months and would have required some form of interventional therapy. This shows that the judicious application of a simple non-invasive technique of faradic stimulation can cure long standing rectal prolapse and obviates any need of any other form of intervention. As expected, the cure rate was poor in the non complaint children. On the other hand, results were gratifying in those with long standing prolapse of over 1 year duration (Table 1). These subjects were found to be most compliant for this therapy as well. Perhaps chronic suffering and a failure to respond to the conventional conservative management acted as a motivational force for compliance to the regime of pelvic floor muscle stimulation/exercise in these children. The other factor for a comparatively lesser degree of response in those with shorter duration of rectal prolapse could be a lesser role of pelvic floor weakness in these subjects.

It is possible that this therapy alone could have resulted in total cure in all our patients had they all of them remained compliant to the therapy or the therapy would have continued for a fairly long period. This is because additional subjects with lesser than 'complete relief' affirmed to have achieved complete response after 12 weeks of follow up enquiry, as has been given under 'results' section. Still, we believe that 12 week is an optimal period after which most parents were found to lose patience and begged for the alternate treatment. Besides, we also found that the pattern of response of the patient at 2 week of faradic treatment (i.e. 6 -8 sittings) have great bearing in prediction of success of this treatment. Those showing >50% response at this stage just need to be compliant and relief generally follows over next few weeks. However, all the poor responders at this stage in our study did not reach their final goal within our period of observation. The reason for this response may be related to the variable role of the weakness of the pelvic floor muscles in the genesis of rectal prolapse in different children.

This modality can be criticized for the prolonged duration of treatment and the need for the frequent hospital visits causing inconvenience to the patient and the parents. However, when compared with the inconvenience of undergoing general anaesthesia for the surgical procedure (even repeated general anaesthesia as are required for the injection therapy, i.e. upto 2-3 times) and likelihood of their complications (e.g. infection, abcesses for injection treatment) [1, 3,11], this inconvenience may appear quite trivial. Above all, this modality of treatment is not marred by any side effect; except that in few cases the total cure may not follow early or at all. However, the trend of response can be predicted fairly early during the course of the treatment. We are presently considering the nonresponders of our study for Thiersch wiring, out of many procedures advocated for rectal prolapse, as it is easy to perform and has yielded optimum results in our hands in the past.

Recently, extracorporeal magnetic innervation (Ex MI) therapy has been tried to improve results in patients with incontinence. ExMI technology works by producing a highly focused, time-varying magnetic field that penetrates deep into the perineum, innervating the pelvic floor muscles by activating motor neurons. Pulses of steep gradient magnetic flux are produced by the therapy head. These fields penetrate the patient's perineum and initiate nerve impulses. Yokoyama et al have used this treatment modality in patients with urinary incontinence and found good results in those with stress incontinence [12]. In future this therapy may also be tried to increase the strength of the pelvic floor muscles in patients with rectal prolapse.

Conclusion

Rectal prolapse is benign condition that causes great concern to the patient and the parents. An adequate trial of conservative management, along with faradic stimulation of pelvic floor can totally cure most of these children, if proper heed is paid to the compliance for the treatment regime and to the pelvic floor and abdominal muscles exercises. Our study does point to the fact that pelvic floor weakness does significantly contribute in the pathogenesis/pathology of rectal prolapse in children as well.

Table 1 :
1The relation between the duration of rectal prolapse and the extent of reliefDuration of symp
<= 50% relief (n = 10)
Upto 75%relief
(n = 9)

100% relief
(n = 28)

<= 4 wk(n = 12)
2
3
7

>4 to <= 12 wk (n = 10)
2
1
7

>12 wk to <= 1 yr (n = 15)
5
3
7

>1 yr (n = 10)
1
2
7

(page number not for citation purposes)
Publish with Bio Med Central and every scientist can read your work free of charge"BioMed Central will be the most significant development for disseminating the results of biomedical researc h in our lifetime." Sir Paul Nurse, Cancer Research UK Your research papers will be: available free of charge to the entire biomedical community peer reviewed and published immediately upon acceptance cited in PubMed and archived on PubMed Central yours -you keep the copyright Submit your manuscript here: http://www.biomedcentral.com/info/publishing_adv.aspBioMedcentral
Competing interestsThe authors declare that they have no competing interests. Authors' contributions SKR conceived of the study and participated in its design and co-ordination, monitored patients' response and follow ups. KNR supervised study. AJ collected patient data. YPM and PJ gave faradic stimulation therapy and monitored the response. DK performed statistical analysis. All authors read and approved the final manuscriptPre-publication historyThe pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2431/9/44/prepub
. BMC Pediatrics. 9BMC Pediatrics 2009, 9:44 http://www.biomedcentral.com/1471-2431

Rectal prolapse in infancy: Conservative versus operative treatment. N Qvist, L Rasmussen, K-E Klaaborg, L P Hansen, S A Pedersen, J Pediatr Surg. 21Qvist N, Rasmussen L, Klaaborg K-E, Hansen LP, Pedersen SA: Rec- tal prolapse in infancy: Conservative versus operative treat- ment. J Pediatr Surg 1986, 21:887-888.

Rectal prolapse in children. M L Corman, Dis Col Rectum. 28Corman ML: Rectal prolapse in children. Dis Col Rectum 1985, 28:535-539.

The injection treatment of rectal prolapse. G G Wyllie, J Pediatr Surg. 14Wyllie GG: The injection treatment of rectal prolapse. J Pediatr Surg 1979, 14:62-64.

Management of rectal prolapse in children. B Antao, V Bradley, J P Robets, R Shawis, Dis Col Rectum. 48Antao B, Bradley V, Robets JP, Shawis R: Management of rectal prolapse in children. Dis Col Rectum 2005, 48:1620-1625.

Observations upon the aetiology and the treatment of complete rectal prolapse. P Ryan, Aust NZ Surg. 502Ryan P: Observations upon the aetiology and the treatment of complete rectal prolapse. Aust NZ Surg 1980, 50(2):109-115.

Outcome and cost analysis of sacral nerve stimulation for fecal incontinence. F H Hetzer, A Bieler, D Hahnloser, Br J Surg. 9311Hetzer FH, Bieler A, Hahnloser D, et al.: Outcome and cost anal- ysis of sacral nerve stimulation for fecal incontinence. Br J Surg 2006, 93(11):1411-7.

Electrical stimulation and biofeedback exercise of pelvic floor muscles in children with fecal incontinence after surgery for ano-rectal malformation. M W Leung, B P Wong, A K Leung, J S Cho, E T Leung, N S Chao, K W Chung, W K Kwok, K K Liu, Pediatr Surg Int. 2212Leung MW, Wong BP, Leung AK, Cho JS, Leung ET, Chao NS, Chung KW, Kwok WK, Liu KK: Electrical stimulation and biofeedback exercise of pelvic floor muscles in children with fecal incon- tinence after surgery for ano-rectal malformation. Pediatr Surg Int 2006, 22(12):975-8.

Rectal prolapse following posterior sagittal anorectoplasty. A Belizon, M A Levitt, G Shoshany, G Rodriguez, A Pena, J Pediatr Surg. 40Belizon A, Levitt MA, Shoshany G, Rodriguez G, Pena A: Rectal pro- lapse following posterior sagittal anorectoplasty. J Pediatr Surg 2005, 40:192-196.

Electrical stimulation of nerve and muscle. J Low, A Reed, Electrotherapy explained, Principles and Practice. Oxford3rd editionLow J, Reed A: Electrical stimulation of nerve and muscle. In Electrotherapy explained, Principles and Practice 3rd edition. Butter- worth Heinemann, Oxford; 2001:46-69.

Prolapse of the rectum. R Fowler, Aust Pediatr J. 3Fowler R: Prolapse of the rectum. Aust Pediatr J 1967, 3:90-98.

Treatment of prolapse rectum in children with injections of sclerosing agents. B N Dutta, A K Das, J Indian Med Assoc. 6912Dutta BN, Das AK: Treatment of prolapse rectum in children with injections of sclerosing agents. J Indian Med Assoc 1977, 69(12):275-276.

Extracorporeal magnetic innervation treatment for patients with urinary incontinence. T Yokoyama, O Fujita, J Nishiguchi, K Nozaki, H Nose, M Inoue, H Ozawa, H Kumon, Int J Urol. 118Yokoyama T, Fujita O, Nishiguchi J, Nozaki K, Nose H, Inoue M, Ozawa H, Kumon H: Extracorporeal magnetic innervation treatment for patients with urinary incontinence. Int J Urol 2004, 11(8):602-6.