
2004; Spence et al., 2004) . These cross-modal spatial links support humans in constructing representations of space by integrating collocated information from several different sensory channels (Driver & Spence, 1998a; Ernst & Bülthoff, 2004) . Within a characteristic range of cue-target stimulus-onset asynchronies (SOAs), they also lead to improved performance on target detection tasks when a preceding cue originates from the same or a similar location as a subsequent target in a different modality (valid/ipsilateral cuing). The facilitation for valid/ipsilateral cuing is most pronounced when cue and task stimuli originate from the exact same spatial location (e.g., Driver & Spence, 1998b; Frassinetti, Bolognini, & Làdavas, 2002; Schmitt, Postma, & de Haan, 2001 ). Performance suffers when cues and targets are spatially disparate.
Cross-modal spatial links have been demonstrated for both endogenous (or top-down) and exogenous (or bottom-up) attention orienting (for an overview, see . Studies examining cross-modal links in exogenous attention allocation -the focus of the present studyhave documented significant cuing effects for all cue-target pairings involving vision, audition, and touch (see Table 1 for a sample of these studies: Kennett, Spence, & Driver, 2002; Spence & Driver, 1997a; Spence, Nicholls, Gillespie, & Driver, 1998; Ward, McDonald & Lin, 2000) .
However, individual studies have reported various forms of cross-modal asymmetry. For example, Ward (1994) and Ward et al. (2000) found that spatially nonpredictive visual cues (cues that are equally likely to originate from the same or opposite side) orient auditory attention but that spatially nonpredictive auditory cues do not affect visual attention. Conversely, Spence and Driver (1997a) demonstrated that auditory cues influence responses to visual targets but not vice versa. Attempts have been made to explain these apparently contradictory results with the particular research paradigms and with the complexity of the stimulus conditions (i.e., the set size of cue-target conditions in any given trial; Ward et al., 2000) . The findings from the studies summarized in Table 1 will be compared with the results of the current study.
The present experiment, like the four studies in Table 1 , used spatially nonpredictive cues and focused exclusively on exogenous cross-modal cuing effects. The goal was to examine the risk of unintentional and inappropriate reorienting of attention if the timing and location of signals is not carefully controlled through interface design or cannot be controlled in domains that are highly dynamic and event driven.
Compared with uncued presentations, cued targets are subject to three additive effects. First, the cue's general alerting effect tends to benefit target detection and response. Second, the performance effects of exogenous orientation (or reorientation) of attention depend on the spatial proximity of cue and target, with benefits or costs being observed for ipsilateral (same-side) or contralateral (opposite-side) cuing, respectively. Given the focus of this study, we investigated the net contributions of these first two effects and will emphasize the second, which we refer to as the spatial cuing effect. The use of spatially nonpredictive cues minimizes the third effect, in which perceived reliability of cues endogenously influences the control of attention.
The Present Study
The main goals of the present study were (a) to determine whether the performance effects of cross-modal spatial links in exogenous attention that were observed in earlier laboratory studies scale to more complex environments and therefore need to be considered in multimodal interface design; and (b) to examine the unresolved issue of cross-modal cuing asymmetries among vision, audition, and touch in such environments.
The following specific hypotheses were tested: (a) For each target modality (visual, auditory, and tactile), cued targets, regardless of the spatial relationship between cue and target, result in faster responses than uncued targets because of a general alerting effect; (b) for all cross-modal cue-target pairings, ipsilateral cuing results in faster response times than contralateral cuing; and (c) the percentage cuing effect is at least as large in more complex environments as has been found in earlier studies conducted almost exclusively in highly controlled laboratory settings.
The earlier controlled laboratory studies largely involved simple tasks, generic cues, targets that were presented at fairly regular intervals, and/or an artificial central fixation of visual attention before the onset of each trial. In contrast, the present study examined cross-modal spatial links in the context of a microworld simulation of future battlefield operations. In addition to a fairly complex cue and target environment, this simulation involved a concurrent task set in order to encourage a more realistic divided attention baseline. The resulting higher attentional demands were expected to result in response times that were longer and more variable than those in earlier research.
METHODS

Participants
The participants in this study were 12 cadets and 2 recent graduates from the University of Michigan (UM) Army Reserve Officers'Training Corps (ROTC) program and 1 civilian UM graduate student employed by the U.S. Army (7 women and 8 men). Their average age was 21.6 years (SD = 1.18), and their average military experience was 3.53 years (SD = 0.99). All UM-ROTC cadets are required to have normal or corrected-to-normal vision and hearing (verified through examination), thus fulfilling the requirements for this study. Participation was voluntary, and a compensation of $30 was paid for completion of the experiment.
Participant Role and Experimental Stimuli
Each participant played the role of a vehicle commander (VC) for the first of a convoy of vehicles in a simulated nighttime rendezvous mission. Consistent with this role, participants were responsible for command-level tasks, while a simulated driver piloted the vehicle. Throughout the mission, participants were presented with visual, auditory, and tactile stimuli, of which some were designated as targets (requiring a speeded response) and others served as cues or distracting events (see Table 2 ). The cue and target events were designed to originate from similar horizontal eccentricities in order to elicit the strongest possible cross-modal link effects.
Visual Display
The participant sat in front of a 19.75-× 12-in (50.2-× 30.5-cm) monitor displaying information and controls similar to those in a VC station for a Stryker Infantry Carrier Vehicle (see Figure1). Centered on this screen, a top-down map of the surrounding area was shown with icons indicating the location of convoy vehicles along the planned route, enemy vehicles, and areas of suspected weapons caches.
In the top left and right corners of the screen, small rectangular displays showed the terrain in front of the vehicle as if viewed through topmounted periscopes. At times, animations of distant explosions or traveling vehicle headlights could be seen on these displays. Immediately below each periscope display, a large red rectangle appeared on a thermal detection system display whenever a heat source was detected nearby. Below each thermal detection system display, the temporary loss of satellite communications was indicated by a change in the backlighting of a radio display from faint green to bright yellow. The visual events presented on the lateralized displays (i.e., periscope, thermal detection, and satellite radio displays) appeared at approximately 20°of visual angle from the center of the map display.
Auditory Display
Scenario-related sounds included gunfire and explosions, radio communications, and "chirp" noises that represented radio short-circuit problems. These sounds were composed of primarily midrange frequencies and played from speakers located immediately to the left and right of the monitor at approximately 25°from the center of the visual display. The 50°separation of speakers in this study is considerably larger than the lateral eccentricity required for 50% of humans to recognize separate sound origins, which is less than 10°f or static midrange sounds (Blauert, 1997; Gröhn, Lokki, & Takala, 2002) . Earlier research in this area has used speaker separations that ranged from as little as 22° (Schmitt et al., 2001 ) to 44° (Ward et al., 2000) to as much as 96° (Spence & Driver, 1997a) .
Tactile Display
Vibrating devices, called tactors, were attached to wristbands worn on both arms. These tactors are 1-× 3 ⁄4-× 1 ⁄ 2-in (2.54-× 1.90-× 1.27-cm) piezoelectric devices that presented vibrations to the skin at a frequency of 100 Hz and a maximum amplitude 50 dB above sensory threshold (comparable to a personal pager) for 500-ms pulses. Pilot testing was used to subjectively match the salience of the vibrations to corresponding visual and auditory stimuli. Vibrations on the left or right side served to alert the VC to potential danger approaching from the corresponding vehicle side. In addition, four tactors were secured to the outside of each thigh in a randomized pattern to represent incidental vibrations similar to those experienced when riding over terrain. Participants were instructed to rest their hands on task controllers that were placed such that vibration stimuli were presented to the wrists and thighs at horizontal eccentricities similar to those of the visual and auditory stimuli.
Procedure
Each 2-hr session consisted of a 20-min introduction and mission briefing, a 25-min training session, a 60-min mission scenario, and a 15-min debriefing. The mission briefing was delivered according to U.S. Army protocol and emphasized the hierarchy of mission objectives. The training session served to familiarize the participants with each of three tasks (described next) and allowed them to practice responding to target events. In the final stage of training, all participants reached an acceptable level of proficiency (no more than 3 errors in 24 cue/target presentations) while concurrently performing the tasks.
Tasks
Participants were responsible for three tasks. The highest priority task was to respond as quickly as possible to the visual, auditory, or tactile stimuli designated as targets (see Table 2 ) by pressing a button. The target stimuli represented events that posed potential danger to the Stryker convoy, and the button press served to alert the other convoy vehicles. Targets were similar in terms of their salience and importance so as to avoid any response biases. Forty-eight targets were presented either in isolation (uncued trials, n = 24) or preceded by an ipsilateral (n = 12) or contralateral (n = 12) peripheral cue (see Table 2 ) in a different modality. Given the focus of this study on cross-modal links in attention, no intramodal cue-target pairings were presented. Note that the timings for target presentations were designed to be less predictable than in earlier studies, which precluded the use of directly related attention-directing cues. Instead, a number of potentially distracting environmental events coincidentally preceded targets within the scenario. These distracting events are referred to as "cues" to maintain consistency with the terminology used in previous studies. They should not differ from cues in earlier studies in terms of their impact on strictly exogenous attention orientation or reorientation.
In order to minimize response priming, we presented each cue event in isolation an additional 12 times throughout the mission. Participants were instructed to not respond to cue events because to do so would alarm the convoy unnecessarily. In those instances when targets were preceded by a cue, the SOA between cue and target was 120 ms. This choice was made based on earlier studies of cross-modal spatial links with exogenous cuing mechanisms (e.g., Spence & Driver,1997a; Ward et al., 2000) .
The participants' secondary task was to monitor and respond to mission-related radio communications. For this purpose, a push-to-talk radio button was secured to the hand that was not used for the primary task. The task with the lowest priority was to manually control (using a joystick) an unmanned aerial vehicle to search for and mark (by pulling the joystick trigger) weapons caches in designated areas on the map. This task encouraged, in a natural way, a centralized fixation of visual attention. Additionally, a remote eye tracker (Applied Science Laboratories Model 504) was used to delay cue presentation (or target presentation, in uncued trials) until gaze direction was near the center of the screen.
Because cues and targets were lateralized, the hand used to depress the response button and the one used to push the radio button and control the joystick were alternated between participants to control for the Simon effect (the tendency to respond faster to stimuli presented on the same side as the response mechanism; Simon, 1990; Simon & Rudell, 1967) .
Experimental Design
This study employed an unbalanced nested design. The two main factors were target modality (visual, auditory, tactile), and peripheral cuing (uncued, cued). Within cued presentations, two factors were varied: cue modality (visual, auditory, tactile) and spatial relationship of the cue-target pairings (ipsilateral or contralateral).
Dependent Measures
Target detection rates and response times (measured from target onset to button press) were recorded. The nature of the target events in this study required that they be presented for extended periods of time. With the exception of tactor wrist vibrations (500 ms), all of the targets were displayed for at least 2,500 ms. As a result, target detection rates were extremely high; therefore the primary dependent measure was response time.
RESULTS
The data were analyzed using a mixed linear model (Proc Mixed function in SAS 9.1). Levene's (1960) test for variance homogeneity was used to identify groups with unequal variances, and Satterthwaite's (1946) correction was applied when comparing such groups. Given the unbalanced experimental design and the need to remove some trials from the data set, replicates for each treatment condition were averaged for each participant, and those averaged values were analyzed. Trials had to be removed because of software errors in target presentation or button presses that were not forceful enough to register a response, as verified through video analysis. Out of a total of 720 trials, 17 responses were removed.
Because the potentially uneven loading of sensory channels imposed by the task set likely affected differences in response times between modalities, the data analysis focused primarily on within-modality comparisons. As a consequence of the higher task and environmental complexity, response variability was expected to be high. Therefore, a slightly relaxed decision rule was used to include p values less than .1 as significant.
Within each target modality, responses to uncued and cued (ipsilateral and contralateral combined) targets did not differ significantly. However, two significant and opposite response time differences were found between uncued targets and ipsilaterally cued targets (see Table 3 ). Ipsilateral cuing resulted in faster response times for auditorily cued visual targets (818 ms) than for uncued visual targets (919 ms), F(1, 14) = 5.71, p = .0315, but slower response times for visually cued tactile targets (911 ms) than for uncued tactile targets (810 ms), F(1, 14) = 8.48, p = .0114. Additionally, auditory targets were responded to faster when preceded by a contralateral tactile cue (843 ms) than when uncued (890 ms), F(1, 14) = 3.62, p = .0777.
The most interesting results were found within cued target presentations (see Table 4 ). A significant benefit was found for ipsilateral (over contralateral) auditory cuing of visual targets (818 vs. 923 ms), F(1, 21.8) = 4.51, p = .0453, with Satterthwaite's (1946) correction, but cuing laterality did not result in a significant difference for visually cued auditory targets. The remaining significant cuing effects favored contralateral presentations, which resulted in faster responses than ipsilateral pairings for tactile cuing of auditory targets (contralateral: 843 ms; ipsilateral: 912 ms), F(1, 14) = 12.40, p = .0034, tactile cuing of visual targets (contralateral: 875 ms; ipsilateral: 976 ms), Table 1 confirms, as expected, that response times were longer and sizes of the significant cuing effects were larger in this study than in the earlier research.
DISCUSSION
This study sought to examine the performance effects of cross-modal spatial links among vision, audition, and touch. In particular, its goals were to revisit the unresolved issue of cross-modal cuing asymmetries and determine to what extent the effects of cross-modal spatial links that were observed in earlier laboratory studies scale to a more complex environment and, thus, can inform the design of multimodal interfaces for domains such as military operations, aviation, and process control.
Overall, the findings from this research confirm that cross-modal spatial links do affect performance in more complex settings and that these performance effects differ for the various modality pairs. In particular, ipsilateral cross-modal cuing was beneficial only for auditory cuing of visual targets but not vice versa, which mirrors the asymmetry found in studies by Spence and Driver (e.g., 1997a ). Significantly faster responses were found for contralateral tactile cuing of auditory targets, but not vice versa. Each visual-tactile cue-target combination showed faster responses for contralateral presentations.
As expected, in this experiment response times were longer (and varied to a higher degree) and cuing effects were larger than in earlier research. These findings can be explained, in part, by the competing attentional demands that were imposed by mission-related tasks in the present experiment. The longer response times and larger cuing effects could also reflect the low predictability of the timing of target events in this experiment, as compared with most previous studies, which presented trials at regular intervals or used a precue to prime participants prior to trials.
No significant difference was observed between uncued and cued trials within each modality, contradicting our expectation of performance benefits attributable to a general alerting effect of cuing. However, for each target modality, a significant difference existed between uncued targets and either ipsilateral or contralateral cued targets.
Differential Effects of Cue-Target Laterality
To date, significant benefits for ipsilateral cuing have been reported for all cue-target combinations involving vision, audition, and touch . The surprising finding from the present study is that tactile-cued auditory targets, tactilecued visual targets, and visually cued tactile targets show the opposite effect: faster response times for contralateral presentations. In fact, a general trend suggests that visually and tactually cued targets result in slower responses when cued ipsilaterally, as compared with contralaterally cued or uncued targets.
One possible explanation for these findings is cross-modal inhibition of return (IOR), in which attention is inhibited, for a particular period of time, from returning to a previously attended location to avoid wasting attentional resources (Klein, 2000) . Cross-modal IOR has been demonstrated for visual, auditory, and tactile cues and targets (Reuter-Lorenz, Jha, & Rosenquist, 1996; Spence & Driver,1998; Spence, Lloyd, McGlone, Nicholls, & Driver, 2000; Tassinari & Campara, 1996) .
That the IOR effect was not found for auditorily cued targets in the cuing paradigm used in this study is consistent with work by , who found that auditory cues do not elicit IOR for visual targets unless saccades to the cue origin and back to a central fixation point are additionally required (however, see . The IOR effect is usually seen at larger SOA ranges (between 350 ms and a few seconds) than the 120 ms used in this study, which may reflect the particular cuing paradigm and/or the environmental and task complexity.
The longer responses to ipsilateral targets could also be attributed to the fact that participants were trained to not respond to cue events. This response inhibition may have extended into the presentation of subsequent target stimuli. It may be more difficult to dissociate target stimuli from the cue stimuli when the stimuli originate from the same area of space, and thus the identification of a separate target event may be delayed, lengthening response times to ipsilateral presentations.
Cross-Modal Asymmetries
Ipsilateral (compared with contralateral) auditory cuing resulted in significantly faster responses to visual targets. However, laterality did not significantly influence response times to visually cued auditory targets. This modality asymmetry is similar to the one observed by Spence and Driver (1997a) but opposite to the findings by Ward (1994) and Ward et al. (2000) , who found that visual cues orient auditory attention but not vice versa. Ward et al. (2000) suggested that differences in the complexity of the cue and target environments in each experiment may explain these apparently contradictory results. In particular, they argued that in the complex cue but simple target environment employed in their study (i.e., an environment with several potential cue modalities but only a single target modality), participants did not necessarily process the spatial location of spatially nonpredictive auditory cues but, instead, simply used their appearance as alerts to upcoming visual targets; hence auditory cues did not spatially orient visual attention. Visual cues did facilitate responses to collocated targets because when visual cues are processed, the two aspects -alerting and location information -cannot be separated.
In contrast, Spence and Driver (1997a) used a complex target (but simple cue) environment and did find a significant spatial orienting effect for auditory cues. The results of the present study confirm that participants do process the spatial aspects of an auditory cue when the target environment is complex, even if the cue environment is also complex.
One possible explanation for these discrepant findings involves the particular cuing paradigm used in experiments in which auditory and visual stimuli originated from the same hemifield but not from identical spatial locations (see Spence, 2001; Spence et al., 2004; Ward et al., 2000) . The precision of spatial orienting for a visual cue may cause responses to a subsequent auditory target from a proximal but separate location to be facilitated to a very limited extent. In contrast, spatial orienting for an auditory cue would tend to be more broadly distributed because of the poorer spatial resolution of the auditory modality (Mondor & Zatorre, 1995) , facilitating responses to targets occurring within a larger spatial area surrounding the cue origin.
Spatial resolution may also help explain another cross-modal asymmetry that was observed in this study: Tactile cues significantly inhibited responses to ipsilateral auditory targets, but not vice versa. Whereas the auditory cues and targets originated from the same location, the tactile stimuli did not; tactile cues were presented from multiple tactors on the left or right thigh, and targets were single-point vibrations presented to the wrists. Because participants were seated, cue vibrations could propagate throughout the leg and up into the torso within the left or right hemifield, possibly creating an IOR effect for a broader area of space that included the origin of the auditory targets. In contrast, the higher spatial resolution of the individual devices on the wrists may have resulted in the perception of tactile targets being more obviously spatially segregated from auditory cues, thus neutralizing the effect.
CONCLUSIONS
The findings from this study confirm that crossmodal spatial links in attention can significantly affect detection performance in complex environments. However, the strength and direction of these effects appear to be different from what prior laboratory studies have suggested. Most notably, a facilitative effect was observed for displaying visual information from the same location as a preceding auditory cue. This particular cuing combination may be the most robust way to exploit the benefits of cross-modal spatial links for multimodal interface design, especially for high-tempo, critical operations in which alarms and events require rapid response.
Also, although recent work has shown that spatial tactile stimuli can be used to support spatial awareness tasks in complex environments (e.g., Raj, Kass, & Perry, 2000; van Erp, van Veen, Jansen, & Dobbins, 2005) , this benefit may be offset by the risk of inhibitive effects such as IOR in cases of cross-modal cuing involving the tactile modality. For other cross-modal cue-target combinations, target detection and/or response may not be appreciably affected; rather, cuing effects were apparently masked by the variability introduced by high attentional task demands.
The observed significant cross-modal cuing effects suggest a need for developing adaptive multimodal interfaces in which the location, modality, and timing of information presentation is varied as a function of surrounding stimuli and task context. The next step toward achieving this goal is to investigate in more detail the performance effects of cross-modal spatial links in endogenous attention allocation and to achieve a better understanding of the interplay between cross-modal top-down and bottom-up attention management in complex, event-driven domains.

{'paper_id': '2116258', 'title': 'Cross-modal links among vision, audition, and touch in complex environments', 's2_url': 'https://api.semanticscholar.org/CorpusID:2116258', 'source': 'S2ORC'}