Joint action goals reduce visuomotor interference effects from a partnerâs incongruent actions

Philosophy

Joint actions often require agents to track othersâ actions while planning and executing physically incongruent actions of their own. Previous research has indicated that this can lead to visuomotor interference effects when it occurs outside of joint action. How is this avoided or overcome in joint actions? We hypothesized that when joint action partners represent their
          actions as interrelated components of a plan to bring about a joint action goal, each partnerâs
          movements need not be represented in relation to distinct, incongruent proximal goals. Instead they can be represented in relation to a single proximal goal â especially if the movements are, or appear to be, mechanically linked to a more distal joint action goal. To test this, we implemented a paradigm in which participants produced finger movements that were either congruent or incongruent with those of a virtual partner, and either with or without a joint action goal (the joint flipping of a switch, which turned on two light bulbs). Our findings provide partial support for the hypothesis that visuomotor interference effects can be reduced when two physically incongruent actions are represented as mechanically interdependent contributions to a joint action goal.

Scientific RepoRtS | (2019) 9:15414 | https://doi.org/10.1038/s41598-019-52124-6 www.nature.com/scientificreports Joint action goals reduce visuomotor interference effects from a partner's incongruent actions Sam clarke1*, Luke Mcellin2, Anna francová2, Marcell Székely2, Stephen A. Butterfill3 & John Michael2,3 Joint actions often require agents to track others' actions while planning and executing physically incongruent actions of their own. previous research has indicated that this can lead to visuomotor interference effects when it occurs outside of joint action. How is this avoided or overcome in joint actions? We hypothesized that when joint action partners represent their actions as interrelated components of a plan to bring about a joint action goal, each partner's movements need not be represented in relation to distinct, incongruent proximal goals. instead they can be represented in relation to a single proximal goal – especially if the movements are, or appear to be, mechanically linked to a more distal joint action goal. To test this, we implemented a paradigm in which participants produced finger movements that were either congruent or incongruent with those of a virtual partner, and either with or without a joint action goal (the joint flipping of a switch, which turned on two light bulbs). Our findings provide partial support for the hypothesis that visuomotor interference effects can be reduced when two physically incongruent actions are represented as mechanically interdependent contributions to a joint action goal. From handshakes to music-making, dance and team sports, social interactions often require an efficient means of tracking others' actions while simultaneously planning and executing actions of one's own1. A basketball player, for example, must monitor and anticipate her teammate's movements in order to successfully contribute to a pick and roll play. Given the broad range of social interactions in which it is important to anticipate, monitor and respond to others' actions, it is no surprise that a considerable amount of research has been devoted to investigating how we achieve this2–5. An influential idea that has emerged is that the representation of others' actions is often supported by one's own motor system, implying that representations of others' actions are often functionally equivalent to the representations involved in action production2–4,6,7. As a result, the observation of others' actions can result in action representations that do not clearly distinguish self from other8–10. An upshot is that the observation of others' actions can give rise to representations that interfere with one's own task performance. In a striking illustration of this, Brass et al.2 found that participants who were instructed to produce finger movements in response to symbolic cues responded more quickly when simultaneously observing irrelevant finger movements that were physically congruent to the ones they were instructed to produce, and more slowly when simultaneously observing irrelevant finger movements that were physically incongruent to these. These findings – and others that build on them11–13 – are taken to indicate that, when observing others' actions, we automatically represent those actions using motor representations of the same type as those subserving action production. This neatly explains why the observation of congruent actions facilitates task performance, while the observation of incongruent actions leads to visuomotor interference effects. However, it also raises a challenge. This is because many joint actions require individuals to produce physically incongruent yet complementary actions14. 1Department of Philosophy, York University, S900 Ross Building, 4700 Keele Street, Toronto, Ontario, M3J 1P3, Canada. 2Department of Cognitive Science, Central European University, Október 6. u. 7, Budapest, 1051, Hungary. 3Department of Philosophy, The University of Warwick, University of Warwick, Coventry, CV4 7AL, UK. *email: firstname.lastname@example.com open 2Scientific RepoRtS | (2019) 9:15414 | https://doi.org/10.1038/s41598-019-52124-6 www.nature.com/scientificreportswww.nature.com/scientificreports/ A proficient basketball player, for example, may need to coordinate her movement towards the basket with her teammate's passing of the ball. But if tracking her teammate's action elicits motor representations that compete with those underpinning the action she herself must perform, then they may interfere with her own action preparation. In more general terms: where the tracking of others' actions involves motor representations that are functionally equivalent to the representations underpinning action production, this could give rise to interference effects and prove counter-productive in many cases of joint action. This problem can, however, be overcome. In a recent paper by Sacheli, Arcangeli, & Paulesu15 participants played learned melodies with, or merely alongside, a virtual partner. In both cases, this required them to sequentially produce actions that were either physically congruent (e.g. point-point) or physically incongruent to those that had just been produced by the partner (e.g. point-grasp). When participants and their partners performed these actions alongside one another (i.e. in a Non-Interactive Condition) performance was affected by the physical (in)congruence of the movements, as expected. But, when these actions were directed towards a joint action goal (i.e. the joint production of a single melody in a Joint Action Condition), physical congruence became irrelevant: task performance was affected by a reversal in movement-note associations, but not by the congruence or incongruence of the two agents' movements. This raises the question: why would doing something in the context of a joint action eliminate interference from the perception of incongruent movements but create interference from the perception of anomalous sounds? Sacheli et al.'s proposed answer is that the representation of a joint action goal enables joint action partners to integrate representations of their own and their partner's actions within a single dyadic (multi-person) motor plan15. As they put it, this dyadic motor plan enables agent's to select appropriate responses to their partner's actions on the basis of their predicted outcomes (e.g. the production of a musical note). This explains why anomalous movement-note associations would have generated interference in their study. However, it does not appear to explain why the joint action frame would have reduced interference from physically incongruent movements. In principle, integrating representations of incongruent movements within a larger motor plan could have increased interference effects instead16. One possibility, left open by the aforementioned study, is that a joint action frame may lead participants to represent their partner's actions in relation to a more distal joint action goal (i.e. a string of musical notes) instead of the more proximal goals that bring this about (i.e. grasping or pointing). In cases where the physical incongruence of the actions only obtains at the level of these more proximal goals, this might allow agents to bypass the representation of their partners' physically incongruent movements altogether, reducing or eliminating visuomotor interference effects (See Fig. 1). The trouble is: there seem to be cases of joint action where it is not sufficient to bypass the representation of a partner's proximal goal altogether and to merely consider the more distal outcome of the joint action goal. Rather, as illustrated by the basketball players mentioned above, it is often necessary to represent the more proximal goals of a partner's action in order to select actions that would complement these Figure 1. Two physically incongruent actions become part of a larger Joint Action plan. If there is no need to represent the other partner's incongruent action (i.e. if the agent can produce their contribution to the joint action without taking their partner's behaviour into account), then this may allow agents to bypass the representation of a partner's actions altogether, allowing interference effects to be reduced or avoided. However, when one agent has to select an action based on which action their partner performs, their individual action cannot be represented only in terms of the more distal joint action or its goal. 3Scientific RepoRtS | (2019) 9:15414 | https://doi.org/10.1038/s41598-019-52124-6 www.nature.com/scientificreportswww.nature.com/scientificreports/ with respect to the more distal joint action goal. Indeed, this can be true of even the most basic motor movements involved. Thus, basic questions remain. Specifically: can the introduction of a distal joint action goal reduce visuomotor interference effects in cases where incongruent proximal goals are contingently related to one another, and attention to these is required for the selection of appropriate motor movements? And, if so, how might this be achieved? In addressing the latter question, a natural starting point is the observation that action production typically involves the simultaneous representation of multiple, instrumentally related actions at multiple, instrumentally related levels of abstraction17–20. For example, we represent the action of turning the steering wheel not only at the level of the comparably distal goal (turned steering wheel) but also at the level of comparatively proximal goals, designed to bring this about (e.g. raised left arm; lowered right arm). Importantly, this hierarchical structure must capture instrumental relations between these different goals. Plainly, proximal goals must function to bring about comparatively distal goals. But, in addition to this, the comparatively proximal goals must (themselves) be sensitive to each other such that a modification to one will lead others to change appropriately. For instance, one need not bother moving one's arms if one is no longer grasping the wheel; and even when one is grasping the wheel, it may be no use raising one's left arm if one does not simultaneously lower one's right arm. Here, the individual agent must simultaneously produce physically incongruent movements (arm lifting and arm lowering). But, in this case, it is not possible to avoid motoric interference by simply considering each arm's movement independently of the other, or by simply considering the more distal goal outcome to which these are both directed (a turned wheel). This is because all of these goals are interrelated. Thus, the introduction of the more distal goal must change the way in which the more proximal goals are represented. Specifically, it must lead to their representation as interrelated, and not simply independent contributions to a larger action. This raises the possibility that the actions of our joint action partners can be represented in relation to the same action hierarchy (See Fig. 2). Here, the introduction of a comparably distal joint action goal might enable the physically incongruent movements of self and other to be represented as interrelated components of a plan to bring about the joint action goal. If this is possible, then it might reduce or even eliminate interference from the observation of a partner's physically incongruent movements, even when success in joint action requires one's selective response to these. Thus, we hypothesise that where agents represent their actions as interrelated components of a plan to bring about a joint action goal, each partner's movements need not always be represented in relation to distinct, incongruent proximal goals. Instead, they might be represented as interrelated contributions to a single goal. If true, the joint action frame could potentially reduce or even eliminate visuomotor interference effects arising from the observation of what an outsider might take to be a physically incongruent action. To test this, we adapted Brass and colleagues'12 paradigm to incorporate a joint action goal, namely turning on two light bulbs by jointly flicking a switch. Here, participants were required to perform one of two finger-lifting movements depending on which numerical cue was presented on a screen, in between a virtual partner's index and middle fingers (See Fig. 3). These movements could be physically congruent or physically incongruent with a movement performed by the virtual partner. In a Joint Action Goal Condition, lightbulbs were turned on when the participant and the partner simultaneously performed physically incongruent actions, but not when they performed physically congruent actions (something about which our hypothesis makes no predictions). In the Individual Goal Condition, the lights were never turned on (i.e. there was no joint action goal). We reasoned that if participants are able to utilize the joint action goal (turning on the lightbulbs) to represent a planning structure in which their partner's movement forms a complementary and mutually interrelated contribution, then the physical incongruence of their own and the partner's movement should be less relevant. This generates the Figure 2. Where one agent has to select an action based on which action the other performs, interference effects may be reduced if the agent can represent both actions as interrelated components of a single goal and not only in terms of the more distal goal (e.g. passing the ball, in a pick and roll play). 4Scientific RepoRtS | (2019) 9:15414 | https://doi.org/10.1038/s41598-019-52124-6 www.nature.com/scientificreportswww.nature.com/scientificreports/ prediction that we should observe reduced visuomotor interference effects in the Joint Action Goal Condition compared to the Individual Goal Condition. In other words, the difference in response times between Congruent trials (wherein the participant and the partner lift the same fingers) and Incongruent trials (wherein the participant and the partner lift different fingers) should be smaller in the Joint Action Goal Conditions than in the Individual Goal Condition. The predictions, sample size, methods, and planned analyses were all pre-registered before data collection and can be accessed at: http://aspredicted.org/blind.php?x=cr4cg2. Unless otherwise noted, we implemented all steps as pre-registered. Results To control for speed-accuracy tradeoffs, reaction time (RT) for correct responses and hit rates (HR) were merged into inverse efficiency scores (IES), a combined measure which homogenizes different patterns of speed-accuracy trade-offs (IES)21, by dividing RTs by accuracy for each condition in each group (lower scores mean more efficient responses). We also analyzed the participants' RT's. For the IES, we conducted a 2 × 2 × 2 mixed ANOVA with Jointness (Joint vs Individual Action Goal) and Congruence (Congruent vs Incongruent) as within participants factors, and Group (Joint First, Joint Last) as a between participants factor. The ANOVA revealed a significant main effect of Congruence F(1,70) = 44.41, p < 0.001, ηp2 = 0.39, with lower IES in the Congruent condition, (M = 1168.67, SD = 211.75) than in the Incongruent condition (M = 1243.39, SD = 230.94); but no significant main effect of Jointness, F(1,70) = 0.49, p = 0.48, ηp2 = 0.007, and no significant main effect of Group F(1,70) = 1.72, p = 0.19, ηp2 = 0.02. There was no significant interaction between Jointness and Congruence, F (1,70) = 0.05 p = 0.82, ηp2 = 0.001, no significant interaction between Congruence and Group, F(1,70) = 3.68, p = 0.06, ηp2 = 0.05 (although this was close to significance, we cannot make conclusions on the basis of this statistic), but a significant interaction between Jointness and Group, F(1,70) = 9.61, p = 0.003, ηp2 = 0.12. There was also a three way interaction between Jointness, Group and Congruence, F(1,70) = 14.49, p < 0.001, ηp2 = 0.17 (see Fig. 4). Post-hoc t-tests for the Joint First Group revealed IES did not differ between congruent and incongruent trials for the Individual Goal condition, t(35) = −1.99, p = 0.054, d = −0.33 but IES were significantly lower for congruent trials than incongruent trials for the Joint Goal condition, t(35) = −4.89, p < 0.001, d = −0.82. Congruent trials' IES in the Individual Goal condition were not significantly different from congruent trials' IES in the Joint Goal condition t(35) = −0.55, p = 0.59, d = −0.09 and incongruent trials' IES in the Individual Goal condition were not significantly different from incongruent trials' IES in the Joint Goal condition, t(35) = −1.93, p = 0.06, d = −0.32. Post-hoc t-tests for the Joint Last Group revealed that congruent IES were significantly lower than incongruent trials' IES for the Individual Goal condition, t(35) = −5.25, p < 0.001, d = −0.88 and congruent trials' IES were significantly lower than incongruent trials' IES for the Joint Goal condition, t(35) = −3.43, p = 0.002, d = −0.57. Congruent trials' IES in the Joint Goal condition were not different from congruent trials' IES in the Individual Goal condition, t(35) = 2.05, p = 0.05, d = 0.34, however incongruent trials' IES in the Joint Goal condition were significantly lower than incongruent trials' IES in the Individual Goal condition, t(35) = 3.98, p < 0.001, d = 0.66. Figure 3. Illustration of the task. Participants were instructed to lift the same finger as the hand in the video when a '1' is displayed (Congruent Condition) and to lift the other finger when a '2' is displayed (Incongruent Condition). The left side illustrates the Individual Goal Condition, in which the lights never turn on. The right side illustrates the Joint Action Goal Condition, in which the lights are turned on when two conditions are fulfilled: the number cue ('2') indicates that the participant should perform the 'incongruent' action, and the participant correctly does so. 5Scientific RepoRtS | (2019) 9:15414 | https://doi.org/10.1038/s41598-019-52124-6 www.nature.com/scientificreportswww.nature.com/scientificreports/ We believe that the three-way interaction was likely the result of the incongruent trials causing comparatively little visuomotor interference in the first block of trials when there was a Joint Action Goal, compared to when there was not a Joint Action Goal. To further investigate the three-way interaction, we subtracted the IES of congruent trials from the IES of incongruent trials (IES difference) for each participant, for each condition, giving us an index of how much the incongruent trials interfered with participants' responses in each of the conditions. We conducted a 2 × 2 mixed ANOVA, with Jointness as a within participants factor, and Group as a between participants factor, which revealed no main effect of Jointness, F(1,70) = 0.05, p = 0.82, ηp2 = 0.001, and no main effect of Group, F(1,70) = 3.69 p = 0.06, ηp2 = 0.05 (although this is close to significance, it does not permit us to draw any conclusions). However, there was an interaction between Jointness and Group, F(1,70) = 14.49, p < 0.001, ηp2 = 0.17 (see Fig. 5). Bonferroni corrected post-hoc t-tests revealed that Joint Action Goal IES differences were significantly larger than Individual Goal IES differences for the Joint First group, t(35) = −2.64, p = 0.012, d = −0.44, and Joint Action Goal IES differences were significantly smaller than Individual Goal IES differences for the Joint Last Group, t(35) = 2.74, p = 0.01, d = 0.46. There was no significant difference between the Joint First group and Joint Last group for Joint Action Goal Trials, t(70) = 0.43, p = 0.67, d = 0.1, but Joint First IES differences were significantly smaller than Joint Last IES differences for Individual Goal trials, t(70) = −3.51, p < 0.001, d = 0.83. Figure 4. 2 × 2 × 2 ANOVA. Mean Inverse Efficiency Scores (IES) are plotted separately for Congruent and Incongruent trials in the Joint Action Goal and the Individual Goal Conditions, for each group. Error bars represent the within-subject confidence intervals29,30. Figure 5. 2 × 2 ANOVA. Mean difference between Congruent and Incongruent Inverse Efficiency Scores (IES) for Joint Action Goal and the Individual Goal Conditions, for each group. Error bars represent the withinsubject confidence intervals29,30. 6Scientific RepoRtS | (2019) 9:15414 | https://doi.org/10.1038/s41598-019-52124-6 www.nature.com/scientificreportswww.nature.com/scientificreports/ Discussion Our results revealed a three-way interaction of Congruence, Jointness and Group. In the group which performed the Joint Action Goal Condition last, the difference between congruent and incongruent trials was significantly smaller when there was a joint action goal than when there was none. In the group which performed the Joint Action Goal Condition first, in contrast, the difference between congruent and incongruent trials was significantly greater when there was a joint action goal than when there was none. The results from the group which performed the Joint Action Goal Condition last indicate that a joint action goal representation may, as we predicted, reduce visuomotor interference effects arising from the observation of a physically incongruent action. These results build upon earlier research indicating that the visuomotor interference effects arising from the perception of a virtual partner's physically incongruent movement can be reduced when two physically incongruent movements are represented as complementary contributions to a joint action goal15,22. Our results extend this research by suggesting that a joint action goal representation can reduce visuomotor interference effects even when agents must detect a joint action partner's physically incongruent movement while simultaneously producing actions that are contingently related to these. This supports the hypothesis that where agents represent their actions as interrelated components of a plan to bring about a joint action goal, the movements of each partner's effectors need not always be represented in relation to distinct, incongruent proximal goals. Instead they can be represented as interrelated contributions to a single goal – especially if the movements are, or appear to be, mechanically linked to a more distal joint action goal. The results from the group which performed the Joint Action Goal Condition first, in contrast, are not consistent with our prediction. For this group, the joint action framing did not lead to a reduction of visuomotor interference effects. Indeed, the difference in performance between congruent and incongruent trials was significantly larger in the Joint Action Goal condition than in the Individual Goal condition for this group. This pattern may be partially be attributed to a carryover effect: having completed the Joint Action Goal Condition first, participants in this group may have continued representing incongruent trials as contributing towards a joint action goal, even once this goal (turning on lightbulbs) had been removed. This conjecture would explain why, for participants in this group, performance on incongruent trials did not worsen from the first block (Joint Action Goal Condition) to the second block (Individual Goal Condition). However, it does not explain why performance on congruent trials did worsen slightly from the first block (Joint Action Goal Condition) to the second block (Individual Goal Condition). We might speculate that this was due to fatigue: it may have been difficult to maintain the high level of performance that we observed in this group on congruent trials in the first block (Joint Action Goal Condition). The mixed pattern of results between our two groups underscores the broader point that we should not expect the introduction of a joint action goal representation to always eliminate or reduce visuomotor interference effects arising from the concurrent performance and perception of physically incongruent actions. And indeed, there is evidence that it does not always do so. For instance, it has been shown that under certain circumstances joint action goal representations can lead to an increase in this form of interference. In one study16, participants were instructed to draw either circles or straight vertical lines on tablet screens resting on a table in front of them while a second participant, sitting diagonally across from them, performed the other (i.e. incongruent) action. On a screen directly in front of each participant, they could see the outcome of their partner's action (i.e. a straight line or a circle appearing on the screen). One group of participants (the Joint Action Condition) were informed that, together, their own and their partner's drawings constituted complementary components of a single drawing. A second group of participants (the Parallel Condition) were informed that the other agent's action was irrelevant to their own task. Within this setup, the joint action goal (i.e. in the Joint Action condition) led to an increase in interference effects. If our hypothesis is correct, this finding is not surprising: since participants did not have to identify their partner's movement in order to select their own movement, and since they were unable to see the outcome of their combined efforts, they did not perceive their own and their partner's actions as mechanically linked to a more distal joint action goal. Thus, participants simply represented their own and their partner's movements in relation to distinct and incongruent proximal goals. This would be consistent with research showing the facilitatory effects of Lissajous plots on bimanual coordination23,24. In this research, participants are instructed to perform two separate rhythmic actions, one with each hand. Lissajous plots are used to display the location of one limb on the x-axis, and the location of the other limb on the y-axis, as well as the location of a dot integrating the locations of the two hands. This visual feedback enables participants to represent the two hand movements as mechanically linked to a single, combined outcome. As a result, they are able to maintain otherwise unstable phase relations25,26. It would be important for future research to investigate other contexts in which individuals must efficiently represent and respond to others' physically incongruent actions. In particular, it would be valuable to probe competitive scenarios in which there are no joint action goals but in which the outcomes of two agents' are interdependent and to investigate the extent to which the representation of a partner's actions can be modulated by the degree of coordination required by a joint action goal. Future research should also investigate the neural mechanisms that underpin the integration of physically incongruent actions into unified motor plans. One important starting point in this respect is provided Sacheli, Tieri, Aglioti, & Candidi's22 study demonstrating that virtual lesions (created using continuous theta-burst stimulation) in the left anterior intraparietal sulcus led to an increase in visuomotor interference effects in a scenario in which participants observed a partner's action and were required to select a physically incongruent action to perform synchronously. Methods participants. Using G∗Power 3.127 we determined that a sample size of thirty would provide 80% statistical power for detecting a small-to-medium-sized effect of the interaction of the two main factors, Jointness (Joint Action Goal vs Individual Goal) and Congruence (Congruent vs Incongruent), assuming a two-way repeated measures ANOVA and an alpha level of 0.05. We therefore recruited 30 participants. Because of the high exclusion 7Scientific RepoRtS | (2019) 9:15414 | https://doi.org/10.1038/s41598-019-52124-6 www.nature.com/scientificreportswww.nature.com/scientificreports/ rate in this initial data collection, we had to recruit more participants to replace those who had been excluded, and overestimated the number of participants needed to compensate for exclusions. This resulted in a total of 36 participants. Due to experimenter error, all of these 36 participants were administered the Joint Last condition, so we then collected 36 participants for the Joint First group to counterbalance. Thus, our sample included 72 participants (13 females; age range: 21–46, M = 27.1, SD = 4.77). All participants were recruited from student organizations in the Budapest and Warwick areas, were naïve to the purpose of the study, and reported normal or corrected to normal vision. All participants signed informed consent prior to the experiment, and received gift vouchers or money for their participation. The experiment was conducted in accordance with the Declaration of Helsinki and was approved by the (EPKEB) United Ethical Review Board for Research in Psychology. Apparatus and stimuli. The experiment was displayed on a 13-inch computer screen (resolution: 2560 × 1600 pixels, refresh rate: 60 Hz). The program for the experiment was written in OpenSesame Python28, with a frame rate of 17 frames per second. Figure 3 illustrates the task environment. procedure. After giving their informed written consent, participants were seated alone at a desk in a lab room and provided with further instructions, after which they had the opportunity to ask clarificatory questions to the experimenter. They then performed two test blocks (Joint Action Goal Condition and Individual Action Goal Condition) consisting of 80 trials each (40 Congruent, 40 Incongruent; 20 of each required index finger movements and 20 middle finger movements). Each test block was preceded by 4 practice trials (i.e. 8 in total). The order of test blocks was counterbalanced across participants; Incongruent and Congruent trials were evenly distributed across blocks and randomly mixed within each block. At the beginning of each trial, participants were instructed to hold down the left and right buttons of the mouse with the index and middle fingers of their right hand. The stimuli were short video sequences of simple finger movements (lifting), as illustrated in Fig. 3. Then a picture of the hand was presented for 2000 ms with a number displayed between the index and middle finger, indicating the participant's instruction for that trial: the participant was instructed to lift the same finger as her/his virtual partner on trials in which a '1' was displayed below the picture of the hand (Congruent trials), and to lift the other finger on trials in which the number '2' was displayed (Incongruent trials). Participants were instructed to respond as quickly and accurately as possible as soon as the virtual partner's finger began to move. Next, a still frame of the virtual partner's hand was displayed with the index or the middle finger having been lifted. The onset of this image was the participant's go-signal to lift the appropriate finger. In the Individual Action Goal Condition, the two light bulbs were displayed on the two sides of the screen (See Fig. 3) during all trials, but remained switched off at all times. In the Joint Action Goal Condition, the instructions were the same as in the Individual Action Goal Condition except that participants were informed there would sometimes be a 'bonus' effect: when the participant and her/ his partner correctly lifted different fingers (i.e. this was only possible on incongruent trials), they would jointly flip the switch, causing the lights to be switched on (See Fig. 3). Participants were explicitly informed that this could happen only on the trials in which the correct response was to perform the incongruent action. At the end of each trial, the scene was displayed for 2000 ms. When the switch was flipped and the lightbulbs turned on, the scene was displayed with the switch having been flipped and the lightbulbs turned on. On trials when participants performed the incorrect action, the background turned red and the scene was displayed, otherwise unchanged, for 2000 ms. On trials when participants performed the correct action but the switch was not flipped and the lightbulbs not turned on, the scene was displayed just as in the previous frame. Data processing and analysis. For the analysis, we had several exclusion criteria. Firstly, we excluded four participants (three in the Joint First Group and one in the Joint Last Group) from all of our analysis as they had an unusually high rate of premature responses (all > 90%), meaning that it is likely that they did not understand the instructions and that their data cannot be relied on. Secondly, we excluded any participants with an overall accuracy more than 2.5 SD below the group mean (either Joint First group or Joint Last group) from all our analyses, as their data is likely unreliable. This resulted in the exclusion of 480 trials (7.6%) or 3 participants from the Joint First group, and 480 trials (7.6%) or 3 participants from the Joint Last group. Secondly, we excluded 72 (1.1%) premature responses (responses before the stimulus onset) from the Joint First group, and 58 (0.9%) premature responses from the Joint Last group, from all of our analysis. Thirdly, 147 (2.3%) trials with RTs more than 2.5 standard deviations (SDs) removed from the mean (calculated for each participant for each condition) were excluded from the Joint First group, and 128 trials (2.1%) were excluded from the Joint Last group. Finally, 240 trials (3.9%) incorrect responses for the Joint First group, and 297 trials (4.7%) from the Joint Last group were excluded from the RTs. Although these criteria were not pre-registered, we determined to apply them prior to analysing any data. Our rationale was that the hypothesis being tested pertained to the processes engaged when people perform actions while perceiving a physically incongruent action from a joint action partner; on trials on which participants committed errors, we could not be confident that these processes were actually engaged. For each participant, we calculated the mean RT's and accuracy (proportion correct), for congruent and incongruent trials for each condition (see Supplementary File for means per condition). We divided the RTs by the accuracy in order to compute Inverse Efficiency Scores (IES)21 as an index of efficiency, appropriately weighting speed and accuracy. 8Scientific RepoRtS | (2019) 9:15414 | https://doi.org/10.1038/s41598-019-52124-6 www.nature.com/scientificreportswww.nature.com/scientificreports/ Received: 21 December 2018; Accepted: 4 October 2019; Published: xx xx xxxx References 1. Hassin, R. R., Aarts, H. & Ferguson, M. Automatic goal inferences. Journal of Experimental Social Psychology. 41(2), 129–40 (2005). 2. Brass, M., Bekkering, H., Wohlschläger, A. & Prinz, W. Compatibility between Observed and Executed Finger Movements: Comparing Symbolic, Spatial, and Imitative Cues. Brain and Cognition. 44, 124–143 (2000). 3. Kilner, J. M., Paulignan, Y. & Blakemore, S. J. An interference effect of observed biological movement on action. Current Biology. 13(6), 522–5 (2003). 4. Sebanz, N., Knoblich, G. & Prinz, W. Representing others' actions: just like one's own? Cognition. 88, B11–B21 (2003). 5. Ramsey, R. What are reaction time indices of automatic imitation measuring? Consciousness & Cognition. 65, 240–54 (2018). 6. Craighero, L., Fadiga, L., Umiltà, C. A. & Rizzolatti, G. Evidence for visuomotor priming effect. Neuroreport. 8, 347–349 (1996). 7. Craighero, L., Fadiga, L., Rizzolatti, G. & Umiltà, C. A. Visuomotor priming. Visual Cognition. 5, 347–349 (1998). 8. Jeannerod, M. & Pacherie, E. Agency, Simulation and Self-identification. Mind & Language. 19(2), 113–46 (2004). 9. Prinz, W. Perception and Action Planning. European Journal of Cognitive Psychology. 9, 129–54 (1997). 10. Rizzolatti, G. & Sinigaglia, C. The functional role of the parieto-frontal mirror circuit: interpretations and misinterpretations. Nature Reviews Neuroscience. 11(4), 264–74 (2010). 11. Stürmer, B., Aschersleben, G. & Prinz, W. Corresponce effects with manual gestures and postures: A study of imitation. Journal of Experimental Psychology: Human Perception and Performance. 26(6), 1746–59 (2000). 12. Brass, M., Bekkering, H. & Prinz, W. Movement observation affects movement execution in a simple response task. Acta Psychologica. 106(1–2), 3–22 (2001). 13. Wang, Y., Ramsey, R. & Hamilton, A. F. The control of mimicry by eye contact is mediated by medial prefrontal cortex. Journal of Neuroscience. 31(33), 12001–10 (2011). 14. Sartori, L. & Betti, S. Complementary actions. Frontiers in Psychology. 6(557) (2015). 15. Sacheli, L. M., Arcangeli, E. & Paulesu, E. Evidence for a dyadic motor plan in joint action. Scientific Reports. 8, 5027, https://doi. org/10.1038/s41598-018-23275-9 (2018). 16. della Gatta, F. et al. Drawn together: When motor representations ground joint actions. Cognition. 165, 53–60 (2017). 17. Candidi, M., Sacheli, L. M. & Aglioti, S. M. From muscles synergies and individual goals to interpersonal synergies and shared goals: mirror neurons and interpersonal action hierarchies: comment on "Grasping synergies: a motor-control approach to the mirror neuron mechanism" by D'Ausilio et al. Phys. Life Rev 12, 126–128 (2015). 18. Chersi, F. Neural mechanisms and models underlying joint action. Experimental brain research 211(3–4), 643–653 (2011). 19. Kilner, J. M. More than one pathway to action understanding. Trends in Cognitive Sciences 15(8), 352–7 (2011). 20. Grafton, S. T. & Hamilton, A. F. C. Evidence for a distributed hierarchy of action representation in the brain. Human Movement Science 26(4), 590–616 (2007). 21. Bruyer, R. & Brysbaert, M. Combining speed and accuracy in cognitive psychology: Is the inverse efficiency score (IES) a better dependent variable than the mean reaction time (RT) and the percentage of errors (PE)? Psychologica Belgica 51(1), 5–13 (2011). 22. Sacheli, L. M., Tieri, G., Aglioti, S. M. & Candidi, M. Transitory Inhibition of the Left Anterior Intraparietal Sulcus Impairs Joint Actions: A Continuous Theta-Burst Stimulation Study. Journal of Cognitive Neuroscience. 30(5), 737–51 (2018). 23. Kovacs, A. J., Buchanan, J. J. & Shea, C. H. Bimanual 1: 1 with 90° continuous relative phase: difficult or easy! Experimental Brain Research. 193(1), 129–136 (2009). 24. Kennedy, D. M., Boyle, J. B. & Shea, C. H. The role of auditory and visual models in the production of bimanual tapping patterns. Experimental brain research. 224(4), 507–518 (2013). 25. Kovacs, A. J., Buchanan, J. J. & Shea, C. H. Impossible is nothing: 5: 3 and 4: 3 multi-frequency bimanual coordination. Experimental brain research. 201(2), 249–259 (2010). 26. Kovacs, A. J. & Shea, C. H. The learning of 90 continuous relative phase with and without Lissajous feedback: external and internally generated bimanual coordination. Acta psychologica. 136(3), 311–320 (2011). 27. Faul, F., Erdfelder, E., Buchner, A. & Lang, A. G. Statistical power analyses using G*Power 3:1 tests for correlation and regression analyses. Behavioral Research Methods. 41(4), 1149–60 (2009). 28. Mathôt, S., Schreij, D. & Theeuwes, J. OpenSesame: An open-source, graphical experiment builder for the social sciences. Behavioral Research Methods. 44(2), 314–24 (2012). 29. Cousineau, D. Confidence intervals in within-subject designs: A simpler solution to Loftus and Masson's method. Tutorials in Quantitative Methods for Psychology. 1(1), 42–45 (2005). 30. Loftus, G. & Masson, M. Using confidence intervals in within-subject designs. Psychonomic Bulletin & Review. 1(4), 476–490 (1994). Author contributions All authors contributed to the study design. A.F. and M.S. programmed the script and collected the data. L.M., A.F. and J.M. performed the analysis. S.C., L.M., S.B., A.F. and J.M. drafted the manuscript. All authors reviewed and approved the manuscript. This work was supported by a Starting Grant from the European Research Council (nr 679092, SENSE OF COMMITMENT). competing interests The authors declare no competing interests. Additional information Supplementary information is available for this paper at https://doi.org/10.1038/s41598-019-52124-6. Correspondence and requests for materials should be addressed to S.C. Reprints and permissions information is available at www.nature.com/reprints. Publisher's note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. 9Scientific RepoRtS | (2019) 9:15414 | https://doi.org/10.1038/s41598-019-52124-6 www.nature.com/scientificreportswww.nature.com/scientificreports/ Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/. © The Author(s)


Worldlets, 3D Thumbnails for 3D Browsing

Philosophy

Dramatic advances in 3D Web technologies have recently led to widespread development of virtual world Web browsers and 3D content. A natural question is whether 3D thumbnails can be used to find oneâs way about such 3D content the way that text and 2D thumbnail images are used to navigate 2D Web content. We have conducted an empirical experiment that shows interactive 3D thumbnails, which we call worldlets, improve travelersâ landmark knowledge and expedite wayfinding in virtual environments.

Elvins, T, Nadeau, D., Schul, R., Kirsh, D. Worldlets: 3D Thumbnails for 3D Browsing. Proceedings of the Computer Human Interaction Society ACM Press, Los Angeles, CA. pp 163-170 1998. PAPERS CHI 98 018-23 APRIL 1998 wayfinding cues. Such cues enable a traveler to build a cognitive map of a region, and thereafter navigate using that map [9] [lo]. A cognitive map is built from survey, procedural, and/or landmark knowledge. Survey knowledge provides a map-like, bird's eye view of a region and contains spatial information including locations, orientations, and sizes of regional features. Procedural knowledge characterizes a region by memorized sequences of actions that construct routes to desired destinations. Landmark knowledge records the visual features of landmarks, including their 3D shape, size, texture, etc. Ul[71. Although social and engineering scientists have studied human wayfinding in the real world for many years [9] [l] [7] [4] [lo] [ 1 l] [ 121, only recently have computer scientists investigated this subject. Satalich [13] led human subjects on a tour of a virtual building and then tested the subjects' ability to find their way to places seen during the tour. Darken [3] applied Lynch's city structure theories [9] to large sparsely featured virtual environments and then compared subjects' ability to find their way to target destinations with and without the added city structure. Witmer [ 171 studied route learning and found that subjects who rehearsed a route in a virtual building learned the route better than subjects who rehearsed the route verbally. Although each of these studies recognized the value of landmark knowledge for wayfinding, none studied the value of landmarks in familiarizing subjects with an environment before entering it. Virtual cities and landmarks 3D models of large cities are becoming available for exploration via the Web [S] [2] [14]. The studies just described showed that travelers visiting such large virtual environments for the first time are easily disoriented, may have difficulty identifying a place upon arrival, and may not be able to find their way back to a place just visited. We hypothesized that by enabling a traveler to become familiar with landmarks before visiting an environment, the traveler will be better prepared to: (1) find their way to destinations of interest, (2) recognize destinations upon arrival, and (3) find their way back to previously visited places. More generally, we believe that directions are more effective if presented using landmarks, and specifically using 3D landmarks. This is in keeping with Wickens' claim that virtua1 environment search tasks are difficult because "often the object of the search is specified in a format different from and more abstract than its rendering in the virtual environment." [16] Worldlets address this problem by representing the object of the search in the same 3D format as the virtual environment, thereby giving searchers first-person experience with virtual world landmarks. WHAT ARE WORLDLETS In two previous papers we described our preliminary worldlet design, implementation, and pilot study [5] [6]. Like a 3D photograph, a worldlet is associated with a viewing position and orientation, and captures a snapshot of the 3D shapes (usually landmarks) falling within the viewpoint's viewing volume. Worldlets can be explored 164 and manipulated in the same way that virtual worlds can be explored and manipulated. Incorporated into a VRML browser, a guidebook or list of worldlets enables a world traveler to view landmarks and landmark context from multiple vantage points. Seeing the landmarks presented in worldlets may help a traveler select destinations of interest and navigate unfamiliar worlds. Figure 3 depicts the process of constructing a spherical worldlet from a fragment of a virtual city surrounding a central viewpoint position. Once captured, worldlets can be browsed, manipulated, and explored in a number of ways. Figure 3. Construction of a worldlet. A gas station landmark in a virtual city (a) viewed from a vantage point in the street. (b) Viewed from above, the gas station is upper left and a region around the previous vantage point is highlighted. (c) The gas station is captured within a spherical worldlet, and (d) is viewed from street level within the worldlet. METHODS General description The experiment was designed to test whether guidebooks of worldlets are better for wayfinding than guidebooks of text and images, and if so, why are they better. A standard within-subject randomized design was used. Training began by reading subjects a series of instructions and teaching them how to operate the user interface in a practice three block by three block virtual city. Next, subjects were allowed to spend unlimited time in this city becoming familiar with the controls and the on-line guidebooks. Each subject was then asked to find their way to a brightly colored goal kiosk, via one landmark, in three practice cities, each five blocks by five blocks. In one city, the guidebook represented each landmark with a paragraph of text, in another city the landmarks were represented as 2D thumbnail images, and in a third city the landmarks were represented in a spherical worldlet. The guidebooks only contained landmark information; subjects were given no procedural or survey information. Although the thumbnail and worldlet guidebook had the same appearance, as shown in Figure 4, the worldlets in the worldlet guidebook could be explored and manipulated. Figure 4. The image and worldlet guidebook looked like this, however, subjects were able to interactively explore the worldlets in the worldlet guidebook I%rameter Constaut value Comments City dimensions 10x10 blocks 100 total blocks Building lots per 20 on perimeter of building footprint block block varies Distance from start 1200 meters past 100 12 meterto goal wide lots Landmarks per 9, including goal ordered in city kiosk guidebooks Distance between 15 blocks +/0.25 block j slow driving speed 1 of each kind in each city left and right turns p$M&iewpoint 1 nt of buildings, :nt in each city, city design and nt to allow for noved at a fixed Table 1. Although the arrangeml landmarks, and routes were differ their basic structure in terms of landmark layout was kept const controlled comparision. Subjects speed in all cities. After completing the practice tasks, the experiment began and subjects were asked to find their way to the goal kiosk via eight ordered landmarks in three ten block by ten block virtual cities. Performance in these three cities was timed and the subject's location, orientation, and actions were recorded at one second intervals. Subjects visited the test cities in a random order to counterbalance possible learning effects. Each of the cities and landmarks were different and unique. Before the computer portion of the experiment, subjects filled out a brief questionnaire on their prior experience with computers. After the experiment subjects answered written questions about the strategies they used while finding their way, and how diicult they found the task using each of the guidebooks. Subjects also completed spatial and verbal neuropsychological tests during a separate session. Constants The major independent variable in this experiment was the type of guidebook used (text, image, worldlet). Parameters concerned with city design and landmark layout, as described in Table 1, were held constant. City design Our primary goal in designing the virtual cities was to preserve as many elements of a real city as possible. Toward this goal we developed a pseudo-random city generator that produced cities based on a regular street grid with pavement roads and sidewalks between the blocks. Each block contained up to twenty buildings, side-by-side around the block perimeter. Using a cache of 250 building designs, buildings were randomly selected and placed on city blocks. Many non-landmark buildings were repeated between and within the cities. Rendering requirements were reduced by programming buildings to make themselves invisible when farther than three blocks from the current viewpoint. To prevent subjects from seeing buildings turning on and off in the distance, we added fog to the environment. Cities contained parks, parking lots, and other civic features but did not contain cars or street signs. The landmarks were placed so that a subject could not shortcut the route and would never have to make an exhaustive search; i.e. as long as they carefully looked down each possible path for familiar landmarks and landmark context, they could always determine in which direction to proceed. No landmarks were located within one block of the perimeter wall that surrounded each city. Radius-of-information Radius-of-information was the most difficult parameter to make constant across the three guidebooks. Given that text, 2D images, and worldlets are different representations, they do not convey an equivalent quantity and quality of landmark information. We determined, while building the guidebooks for the pilot study, that using a radius of twenty meters produced the most nearly equivalent set of guidebooks. For the text guidebook this meant describing in words the landmark, the building or structure directly across the street, the building on each side, and whether the landmark was on a corner or not. Thus our textual description of a landmark with a twenty meter radius read as follows: Citgo Gas Station. Citgo's red gas pumps are at a comer location. Next door is the Cafe 360 in a beige colored building, and across the street is the Lumbini Restaurant in a red brick building. 165 PAPERS CHI 98 . 18-23 APRIL 1998 For the image guidebook, a twenty meter radius meant positioning the camera across the street from the landmark so that the captured thumbnail image would contain the buildings or structures on the left and right of the landmark. For the worldlet guidebook, setting the information radius simply meant setting the yon clip plane to cut away 3D shapes further away than twenty meters while capturing a spherical worldlet. Image and worldlet camera locations were selected so as not to suggest a direction of approach to the landmark -.-.---~.-.-.__-. Figure 5. The main city window. Hardware and software design The experiment was conducted using a standard 19 inch Silicon Graphics monitor. Stereo glasses were not used A main city window displayed the city in a VRhE browser user interface modified for the experiment. Subjects used keyboard arrow keys to move forward and backward a fixed distance on each key press, and to turn left and right by a fixed angle on each key press. In the worldlet guidebook, subjects used the arrow keys in the same manner to move around in the worldlet. Collision detection prevented subjects from passing through buildings and other objects in the city. Subjects pushed a Start button before starting, a Landmark button each time a landmark was reached, and a Stop button upon reaching the goal kiosk Subjects pushed a Guidebook button, located near the Start button, to display the guidebook Subjects practiced operating all of these buttons in practice cities prior to beginning the experiment. The windows were designed so that when the guidebook (Figure 4) was open, the main city window (Figure 5) was completely occluded and no movement could be made. Lessons learned from the pilot study Our pilot study design and results are described in [5]. From the pilot study we learned that landmarks cannot be more than two blocks apart or subjects frequently resort to an exhaustive search of the neighborhood. We also learned that some subjects will take much Ionger than other subjects to complete the task and that the task difficulty and Iength must be designed while keeping the slower wayfinders in mind. Subject instructions The following is a subset of the instructions read to the subjects. Instructions 3 and 4 were repeated before beginning each city. 1. 2. 3. 4. The guidebook contains a series of landmarks that you probably want to follow, since traveling past these landmarks will lead you to the goal kiosk. When you are at a landmark, you may have to walk as far as one or two blocks before you will be able to see the next landmark. Your task is to locate the goal kiosk as quickly as possible. Go as fast as you can, but be careful to not get lost Remember, you can look at the guidebook as often as you like Subjects Subjects were primarily University of California, San Diego students from a number of major departments. Ages ranged from 18 to 36, with a mean age of 24. Fourteen males and twelve females completed the experiment. Results from four other subjects had to be excluded from our data because the subjects became lost in one or more of the cities. All subjects were computer literate, but had varying degrees of experience with virtual environments. Neuropsychological tests To provide an independent measure of a subset of the subjects' cognitive abilities, subjects were tested in a separate session, using two standardized subtests from the Wechsler Adult Intelligent Scale Revised (WAIS-R) [ 151. The Vocabulary subtest, comprising part of the WAIS-R Verbal subtests, was used to assess the subject's verbal ability. Performance was scored according to standardized procedures based on the subject's degree of familiarity with a list of up to 35 words and their elegance of expression 1151. The second test was the Block Design subtest comprising part of the WAIS-R Performance subtests. The test was used to assess the subjects' constructional-spatial ability, and was administered in the standardized manner [ 151. Analysis A one-way repeated measures analysis of variance (ANOVA) was performed for each of the dependent variables and the overall times and distances. The withinsubjects variable was the landmark description type with three levels: text, image, and worldlet. Post-hoc analyses were done using the Scheffe test for differences in sample means. Correlation analysis was performed across the scores of the neuropsychological tests and the scores of the wayfinding tests. We adopted a significance level of 0.05 on all analyses unless otherwise noted. RESULTS Our primary result was that people spend more time consulting the worldlet guidebook than the other guidebooks, but require less time to reach the goal when using the worldlet guidebook. The following specific findings tell us more about the component activities that benefitted from worldlet use. Time and distance measures Figures 6 through 9 show time and distance measures where the independent variable (type of guidebook) is displayed as white, grey, and black bars along with the standard error of 166 CHl 98 l 18-23 APRIL 1998 PAPERS ' the mean @EM). Figure 6 shows the mean overall time and istance required to reach the goal kiosk. Figure 7 breaks the overall time into its components. Figure 8 gives the forward and bat&racking distance components. Figure 9 shows the mean number of times that each type of guidebook was opened. Figure 10 diagrams the subjects' scores on de Wechsler neuropsychological tests. Analysis of variance followed by Scheffe post hoc tests were performed for the different measures. Table 2 summarizes these results. (a) @I Figure 6. (a) hgean total time. The overall time spent traveling to the goal was significantly less using the worldlet guidebook than when using the text (F&24)=9.89, ~~0.01) or image (F&24)=4.63, pcO.01) guidebooks. (b) Mean total distance. The overall distance traveled using the worldlet guidebook was significantly less than the distance traveled using the text (F&24)=11.41, ~~0.01) or image (F&24)=8.93, p~O.01) guidebooks. Figure 7. Tmning: looking for landmark is time spent standing in one location. Turning: changing heading is time spent changing forward direction. Consulting guidebook is time spent with the guidebook on screen. Moving forward is time spent traveling over new ground. Backtracking is time spent retreating. Note that time spent consulting the worldlet guidebook is significantly greater than time spent consuIting the text or image guidebook. Also note that time backtracking is near zero when using the worldlet guidebook Figure 8. The mean distance moving forward while 8 using the worldlet guidebook is significantly less than the forward distance covered while using the text or image guidebook. Backtracking distance is almost non-existent for the worldlet guidebook task. Mean number of times guidebook opened Figure 9. Subjects opened the text and image guidebooks on average 15-16 times while seeking the eight landmarks enroute to the goal kiosk. The worldlet guidebook was opened 11 times on average. Spatial and verbal scores Subject Figure 10. Z transformed spatial and verbal scores from the neuropsychological tests. Before applying the 2 transform, the mean spatial test score was 45, and mean verbal test score was 55. 167 PAPERS CHI 98 . 18-23 APRIL 1998 cc> Figure 1 1_ Street plots. In these plots line thickness and brightness indicates the number of times that a street was traversed. In (a) and (b) the 26 subjects traversed some streets 47 times indicating backtracking. (a) and (b) also exhibit circuitous trails. In (c) very little backtracking occurred, and circuitous trails are minimal. Street plots Figure 11 shows plots of the streets traversed by the 26 subjects in the three test cities. The thickness and brightness of each street line indicate the total number of times that the street was traversed. For example, wider brighter streets indicate heavily used avenues. Backtracking can increase the number of street traversals beyond the expected minimum of 26. The starting point and goal kiosk are indicated on each plot, and landmarks enroute to the goal are marked with a star (a). tracking pcO.0003 overall 10.34 * p<0.0002 Distance moving11 5.33 * pco.01 pco.01 0.98 9.89 * 4.63 * pco.01 p<O.Ol 1 0.05 (12.23 * 110.72 * 1 forward JKo.0001 w back 10.19 * trackin p<O.O002 overall 13.65 * pco.01 pco.01 0.23 8.84 * 6.21 * pco.01 pco.01 0.15 11.41 * 8.93 * I Ilpc0.0001 Guidebook I Jpco.01 lpco.01 1 open 15.27 * 1.04 7.59 * 14.26 * p<0.0001 pco.01 pCo.001 Table 2. Our ANOVA indicated that there were significant differences between all the sample means and post hoc analysis was warranted. Post hoc (Scheffe) analysis indicated that, for all dependent variables, there were significant differences in the sample means of the text versus worldlet performance, and thumbnail versus worldlet performance. Performance using the text versus image guidebooks was not significantly different except that time spent consulting the text guidebook was significantly greater than time spent consulting the image guidebook. In each table cell, the upper number is the F value, and the lower number is the p value. Asterisk indicates significance ~~0.05. Time and distance measures are explained in the captions for Figures 6-9. Subjective measures As part of the written questionnaire administered after the experiment, subjects were asked how difficult the search task was using each of the different types of guidebooks. 168 CHl 98 . 18-23 APRIL 1998 PAPERS Responses were scattered, however, a clear trend in favor of the worldlet guidebook was evident. Very =Y Text image worldlet 0 1 2 I 11 1 Median Doable Doable Easy Table 3. Subjects were asked to rate the difficulty of the search task using the three different types of guidebooks. Correlation analysis One surprising finding was a high inverse correlation (-0.77, p=O.Ol) between spatial skills and the time spent consulting the text guidebook. Subjects with good spatial abilities seemed able to form a mental picture of a landmark on the basis of textual description more quickly than subjects with lower spatial abilities. This is not the case for image and worldlet guidebooks. We expected to find a negative correlation between spatial skills and overall travel time in the thumbnail and worldlet guidebook cities. The data did not support this, nor did they support another of our hypotheses that there would be a positive correlation between verbal skills and the performance in the text guidebook city. Verbal skills made no difference in overall wayfinding time. Aclditional observations Many subjects said that the tasks were harder than they had imagined and afterwards several said that they felt tired. No subjects complained of motion sickness. Most subjects required seventy minutes or more to finish all aspects of the computer-based part of the experiment. We observed subjects to use three general types of strategies while deciding what direction to go from a street intersection: 1. Look down each street from many viewpoints within the intersection, until a landmark or landmark context is seen. 2. Look down each street and if nothing looks familiar, search exhaustively in each direction until the next landmark is found 3. Look down each street and if nothing looks familiar, try going one half block in each direction and look again. Subjects using strategy rr"l seemed to find their way more quickly and with less backtracking than subjects who used strategies #2 and #3. Another strategy, described by some subjects on their written questionnaire, involved viewing a landmark in the worldlet guidebook from many distant vantage points in tbe hopes that one of the vantage points would be similar to the direction from which they would actually approach the landmark in the city. We observed that some subjects often forgot what it was they were looking for, and so had to open the guidebook momentarily to remind themselves. Not surprisingly, landmarks with signs such as "Grocery Store" and landmarks painted with more highly saturated colors seemed easier to find than other types of landmarks. Also, after passing all the landmarks, several subjects exclaimed that they had lost all sense of direction within the world. A final unexpected result is that there were many similarities between the text and image guidebooks. The text and image guidebooks were only significantly different in the time subjects spent consulting them. Considering that text and images represent landmarks in very different ways, we expected differences in time and distance measures. Major findings As we predicted, the total time spent traveling to the goal was significantly less using the worldlet guidebook than when using the text or image guidebooks. Subjects spent significantly more time studying the worldlet guidebook than the text or image guidebook. Increased worldlet study time was more than compensated for by a reduction in the time required to reach the goal. Using the worldlet guidebook, subjects traveled on average 55% faster than when using the text guidebook, and 38% faster than when using the image guidebook. Increased worldlet study time also allowed subjects to move more directly to the goal and reduced backtracking time to almost zero. This is shown diagramatically in Figure 11 where Figures 1 la and lib show brighter wider streets than Figure llc. Figure 1 lc also exhibits fewer detours and less wandering than Figures lla and lib. We postulate that compared to the landmark knowledge gained from text and image guidebooks, the improved landmark knowledge built while consulting the worldlet guidebook enabled subjects to refer less frequently to the guidebook. This postulation is further supported by examining worldlet guidebook results: l Subjects spent significantly more time studying the worldlet guidebook than the other types of guidebooks l When using the worldlet guidebook, subjects traveled faster, more directly, and with more skill, to the goal kiosk l Subjects' median rating for the difficulty of the search task was 'Easy' when using the worldlet guidebook. CONCLUSION We know of no studies that have considered the comparative advantages of different types of guidebooks for wayfinding. Intuitively, one would predict that 3D thumbnails make better guidebooks for 3D worlds than do text and 2D thumbnail images, and indeed, this is what we found. We wanted, however, to identify the component activities in using guidebooks that most benefited from 3D elements. Perhaps of greatest interest was our finding that 169 , PAPERS CHI 98 l 18-23 APRIL 1998 worldlets enabled subjects to find their way in complex environments with almost no backtracking. By spending more time studying the worldlets in their guidebooks, they spent less time looking around, re-orienting themselves, conducting exhaustive searches, and performing similar time-wasting activities. We must be cautious, however, in what we conclude since worldlets contain more information than text or 2D thumbnail images. To control for this difference in information content, more experiments are necessary. Future work We would like to investigate the comparative advantages of worldlet guidebooks for wayfinding tasks in corresponding virtual and real world environments (both urban and rural). We are also considering an experiment to evaluate tandem use of traditional 2D maps and landmark guidebooks. ACKNOWLEDGMENTS Our appreciation goes to Planet 9 Studios (http://www.planet9.com) for the use of their city building textures. Thanks to John Moreland and Mike Bailey for technical assistance on this project, and thanks to Suzi Feeney for collaborating in the pilot study design. UCSD librarian Debbie Cox and her staff were invaluable in locating references. Thanks also to John Helly and Reagan Moore for their support San Diego Supercomputer Center is funded by NSF, grant ASC 8902825, the University of California, the State of California, and industrial partners. REFERENCES 1. Appleyard, D.A., "Why buildings are known," Environment and Behavior, 1969,1,131-156. 2. Art+-Corn, "The cybercity Berlin project", http://www.artcom.de/projects 3. Darken, R. P., and Sibert, J. L., 'Wayfmding strategies and behaviors in large virtual worlds," in Proceedings of the ACM CHI 96 Conference, Vancouver, BC., April, 1996,142-149. 4. Downs, R. J., and Stea, D., "Cognitive maps and spatial behavior," in Image and Environment, Chicago: Aldine Publishing Company, 1973, S-26. 5. Elvins, T-T., Nadeau, D.R., Kirsh, D., 'Worldlets 3D thumbnails for wayfinding in virtual environments,"in Proceedings of the ACM User Interface Sojiware and Technology Symposium, Banff, Canada, October, 1997, 21-30. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. Elvins, T.T., 'Wayfinding in layered information worlds," Short paper in Proceedings of the CODATA Euro-American Workshop on Data and Information Visualization, Paris, France, June, 1997,6-7. Goldin, S.E., Thomdyke, P.W., "Simulating navigation for spatial knowledge acquisition," Human Factors, 1982,24 (4), 457-471. Liggett, R., Friedman, S. and Jepson, W., "Interactive design/decision making in a virtual urban world: visual simulation and GIS," in Proceedings of the 1995 ESRI User Conference, Environmental Systems Research, July, 1995 http://www.esri.com/base/common luserconfiproc95 Lynch, K., The Image of the City, M.I.T. Press, 1960. Passini, R., Wayfinding in Architecture, Van Nostrand Reinhold, NY, second edition, 1992. Peponis, J., Zimring, C., and Choi, Y.K., "Finding the Building in Wayfinding," Environment and Behavior, 1990,22 (5), 555-590. Presson, CC., Hazehigg, M.D., "Building Spatial Representations Through Primary and Secondary Learning," Journal of Experimental Psychology: Learning, Memory and Cognition, 1984,10,716-222. Satalich, G. A., Navigation and Wayfinding in Virtual Reality: Finding the Proper Tools and Cues to Enhance Navigational Awareness., Masters Thesis, Department of Computer Science, University of Washington, 1995. http://www.hitl.washington.edulpublications~ University College London Center for Advanced Spatial Analysis, "Virtual London: a proposal," http://www.geog.ucl.ac.uk/casa/vl.html Wechsler, D., WARS-R Manual, The Psychological Corporation, Harcourt Brace Jovanovich, Inc., 1981. Wickens, C., and Baker, P., "Cognitive Issues in Virtual Reality," Chapter 13 in Mental Models and Human-computer Interaction, Editors Woodrow Barfield and Thomas Fumess III, Oxford University Press, 1995,514541. Witmer, G.G., Bailey, J.H., Knerr, B.W., Parsons, K.C., "Virtual spaces and real world places: transfer of route knowledge," In?. J. Human-computer Studies, 45, 1996,413-428.