Abstract This article describes part of a research programme whose goal is to develop a better understanding of how operators monitor complex, dynamic systems under normal operations. In a previous phase, "eld observations were made at two older nuclear power plant control rooms (CRs) consisting primarily of analogue, hard-wired instrumentation. In this phase, additional "eld observations were conducted in a newer computer-based CR to determine the impact of CR technology on operator monitoring. Eleven di!erent operators were observed in situ for a total of approximately 88 h. The "ndings indicate that there are many similarities in the monitoring strategies adopted by operators in the two types of CRs. However, in most cases, these same strategies are performed using di!erent behaviours, thereby showing the shaping e!ect of the CR technology. A new way of conceptualizing the di!erence between traditional analogue CRs and modern computer-based CRs is proposed. Introduction For some time now, the technology that is being used to design interfaces for complex, dynamic work domains (e.g. aviation, manufacturing, medicine, nuclear, petrochemical) has been changing. Traditionally, indications were often displayed using analogue indicators that were hard-wired, meaning that their appearance and location were "xed at the time of design. In some industries (e.g. nuclear power), the majority of existing control rooms in operation still rely primarily on this older technology, albeit to di!ering degrees. As newer designs are implemented and upgrades are made, however, more information in complex, dynamic work domains is being managed and displayed using digital, computer-based indicators that provide important forms of #exibility in data management. Speci"cally, data can be brought up on multiple CRTs as a function of the operators' information needs, and work domain variables can be viewed in the context of other variables to facilitate interpretation (Woods, 1991) . Newer interface designs rely much more heavily on this advanced technology, although in some industries, a subset of information is still displayed using the traditional analogue technology. Introduction The power and #exibility of computer-based technology introduces myriad new design possibilities. Nevertheless, new does not necessarily mean better. What impact does the introduction of computer-based technology have on operator performance in complex, dynamic work domains? We have been pursuing this question in the context of nuclear power plants. As far as we know, no comparative "eld studies have ever been conducted to investigate the impact of control room technology in this industry. The work described in this article is part of a research programme that addresses this gap in the literature. In particular, we have focused on the impact of control room (CR) technology on operator monitoring under normal operations. Introduction In earlier work (Vicente and Burns, 1996; Mumaw, Roth, Vicente & Burns, 2000) , we conducted "eld observations in two older plants whose CRs consisted primarily of analogue, hard-wired technology. A total of 27 operators were observed on day and night shifts at these plants over approximately 177 h. Those observations led us to a characterization of operator activities that facilitated monitoring*speci"cally, to monitor e!ectively in spite of the limitations of the analogue, hard-wired CR technology. In the work reported here, we conducted similar observations at a newer plant whose CR consisted primarily of computer-based displays (a detailed description is provided later). The nuclear reactor at this newer plant was of the same type (i.e. CANDU) as those that we had observed previously. Furthermore, the newer plant was owned and operated by the same electrical utility as the two older plants. Therefore, although there were di!erences in the level of automation in the three plants we studied, we were fortunate to be able to control for some factors (i.e. reactor type and utility) that might a!ect our results. As a result, this sequence of "eld studies can be viewed as a &&naturalistic experiment'' (cf. Vicente, 2000) , allowing us to study the impact of CR technology on operator monitoring during normal operations. Introduction The article is organized as follows. We begin by providing a brief summary of the "ndings from our previous work. Then, we describe the methodology that was adopted for this phase of research. Next, we describe the design of the CR that we observed in this phase. The results of the "eld study are presented, organized according to the following factors: the sources of information that are available for monitoring, the factors that make monitoring di$cult and the strategies that operators adopt to make monitoring easier. A discussion of these results is presented, with the primary focus on how the CR technology impacts operator monitoring. Summary of previous findings To provide a baseline for interpreting the "ndings presented in this article, we "rst brie#y summarize the "ndings that were obtained in the earlier "eld studies of monitoring that we conducted [see Mumaw et al. (2000) for more details]. Those studies were conducted in two older CRs that will be referred to as the older CR and the oldest CR since one was slightly older than the other. Both were based predominantly on analogue, hard-wired Mumaw et al. (2000) for details] System complexity and reliability Alarm system design Displays and controls design Design of automation technology. Table 1 summarizes the sources of information that operators had to monitor the status of the older and oldest plants. We were surprised by the number and diversity of these sources. Operators do not just rely on the CR panels and alarms, but use other media and other people as sources of information. Table 2 summarizes the factors that make monitoring di$cult for operators. Some of these could be improved through more attention to human factors principles in CR design (e.g. alarm system, displays and controls, automation). However, there is a host of factors pertinent to system complexity and reliability that seem to be unavoidable. Since each reactor unit is composed of so many parts, it is inevitable that, on any given day, at least a few components or indicators will not be functioning, will be functioning imperfectly, or will be in the process of being maintained, repaired or tested. It is important for operators to keep track of these activities and states because they provide the background context for monitoring. That is, what indications are considered &&nor-mal'' depends on which components or indicators are functioning properly, being maintained, repaired or tested. The same set of indications can be normal on one day, but an indication of a problem on the next. This job of tracking the background context to generate accurate expectations for monitoring is complicated by two factors. First, there is a great deal of information to track. Second, the status of these components or indicators can change from day to day. Thus, the operators' understanding of the background context must be continually updated if it is to be accurate. This need for dynamic and comprehensive updating of a complex background context cannot be avoided and makes monitoring a very challenging cognitive activity, even under &&nor-mal'' (i.e. non-fault) operations. Summary of previous findings Despite the complexity just noted, operators are able to monitor the plant e!ectively because they have developed a set of strategies to facilitate monitoring. For example, operators seem to build and maintain a situation model that they use as a basis for monitoring. In addition, operators appear to frequently engage in top-down, knowledge-driven monitoring rather than wait for events to occur before interpreting them. Perhaps most interesting, we also found that operators adopted a number of ingenious strategies that compensate for limitations in the existing CR interface and thereby reduce OPERATOR MONITORING the cognitive demands associated with monitoring. As shown in Table 3 , we placed these strategies into three classes: (1) strategies designed to enhance information extraction by increasing the salience of important indicators and reducing the background &&noise''; (2) strategies designed to create new information; and (3) strategies designed to o!-load some of the cognitive processing onto the interface (e.g. creating external aids and reminders for monitoring). These strategies are largely passed from operator to operator informally (instead of through formal training). Summary of previous findings How many of the "ndings summarized in Tables 1}3 change when computer-based technology is introduced into the CR? As far as we know, this question had yet to be addressed in the literature because no comparative "eld study of operator monitoring had been conducted in the nuclear industry. Consequently, we were in an excellent position to make a unique contribution to the research literature. Since information technology is being introduced into many complex, dynamic work domains, the lessons we extract from this research program on operator monitoring should be of great interest to many cognitive engineering researchers. Field study methodology The following questions drove our research: Field study methodology E How do operators routinely monitor the plant? E What factors contribute to monitoring di$culty and vulnerability to missing key indicators of a developing problem? E What cognitive strategies have operators developed to facilitate early detection of a problem? E How did the di!erences in user interface technology from the previous CR we observed impact monitoring demands and monitoring behaviour? Field study methodology These questions could best be addressed through "eld observations of operator performance in the actual work context. Field study methodology Field observations support a discovery process. They serve to draw attention to signi"cant phenomena and suggest new ideas, the validity and generality of which can then be evaluated through additional studies. The "eld studies we performed served this role. They helped us to discover and characterize the cognitive complexities that operators confront during normal operations and the active, problem-solving activities that play a fundamental role in monitoring a complex, dynamic work domain, such as an NPP. The "eld study methodology that we adopted is similar in approach to ethnographic methods (e.g. Jordan and Henderson, 1995; Nardi, 1997) and the European "eld study tradition (De Keyser, 1990 ) in that we placed ourselves in the actual work context to observe and interview operators as opportunities arose. Field study methodology The computer-based CR observed in this study will be referred to as the New CR because it is the most recent design operated by this utility. The study's objectives were to con"rm the classes identi"ed in the observations at the two older control rooms, identify additional examples of these classes and look for new classes that had not been observed before. Field study methodology In general, one observer sat in most of a work shift to experience the various activities that occur over the course of the day (or night). Interviews were used to obtain general information about operator practice, to follow up on the monitoring actions that we observed and to test the representativeness of monitoring strategies that we observed or were told about. Particular attention was focused on documenting illustrative incidents that provided concrete examples for the kinds of complexities that can arise in the environment and the kinds of monitoring strategies and facilitating activities that operators use to handle these situations. Field study methodology Initially, our observations in the New CR were made using an open-ended methodology. Three di!erent operators were observed over 3 day shifts for a total of approximately 24 h of in situ observation. The goal of this initial phase of data collection was to get an overall understanding of the CR design and a preliminary assessment of the similarities and di!erences with the two plants that had been observed before. Additional observations were made at the New CR using the more structured, directed methodology used in the previous phase of this work (see Mumaw et al., 2000) . Four di!erent operators were observed on four di!erent day shifts for about 8 h at a time. The goal of this second phase of data collection was to get a more detailed understanding of operator monitoring activities at the New CR, with particular focus on understanding how the CR technology a!ected monitoring activities. Field study methodology During this second phase two observers were in the control room at the same time, but worked independently, each at a di!erent unit. The observers used a checklist of issues to guide observations: (1) plant or crew policy and procedures for guiding monitoring, (2) unit and control room sta$ng, (3) overview of the control room interface and information resources for the operator, (4) overview of the alarm system, (5) problems with control room technology or operation, (6) use of automation in the interface, (7) how monitoring can become di$cult, (8) the motivations for monitoring, (9) operator interactions with the interface to support monitoring, and (10) monitoring strategies. Field study methodology In summary, we were able to observe 11 of the 28 operators at the New plant for a total period of approximately 88 h, thereby providing a fairly representative sample of "eld observations. OPERATOR MONITORING There are a number of advantages to the methodology we adopted. The "eld observations a!orded the opportunity to gain a realistic view of the full complexity of the work environment, which is not possible with other approaches. We were able to observe the broad range of operator duties, the distractions that arise, the need to time-share tasks, other demands on operator cognition or types of complexity and the interactions with other people both inside and outside the control room. The "eld observations also allowed us to observe the distributed cognitive system (Hutchins, 1995) . We were able to examine how operators adapt the work environment in order to o!-load cognitive demands and facilitate performance. Finally, "eld observations provided the opportunity to observe and document actual incidents that reveal the cognitive demands operating in the environment and strategies that have been developed by operators in response to these demands. We were able to collect a corpus of concrete incidents that illustrate the complexities in the environment and the monitoring and facilitating strategies that operators have developed. Control room design Since the primary purpose of this research was to determine the impact of CR technology on operator monitoring, it is important to describe the new plant and its CR in some detail. There are substantial di!erences in both plant design and CR design between the new and the older or oldest plants. The implications of these di!erences for operator monitoring will be discussed in subsequent sections. PLANT DESIGN Although all three plants observed in this research are of the same reactor type, the structure of the new plant is quite di!erent from that of the oldest plant. Among the di!erences that were noted was that many dedicated analogue controllers have been replaced by digital automation. It should also be noted, however, that the design of the new plant is quite similar to that of the older plant. However, the CR designs of these two plants are quite di!erent from each other (see later). Consequently, more weight should be given to the di!erences between the results obtained at the new plant and those obtained at the older plant rather than those obtained at the oldest plant, since the former set of di!erences is likely to be a result of the CR technology, not the design of the plant itself. CONTROL ROOM LAYOUT One obvious di!erence between the new CR design and both the older and oldest CR designs is the layout of the panels for the four units. The older and oldest CRs were similar to each other, with the panels for each unit facing each other in opposite corners of a relatively square-shaped room. As shown in Figure 1 , the new CR has a linear layout with the panels for each unit along the same side of the room in a serial fashion. In this case, the unit panels do not face each other and the distance between units is generally longer than that with the older and oldest CR designs. Also, note that the thick lines represent control panels, whereas the thin lines indicate short workspace dividers that were added after the CR was built and the units had become operational. The implications of all of these characteristics for operator monitoring will be discussed in the next section. Figure 1 shows the layout of the control panels for the di!erent units at the new CR. Figures 2 and 3 illustrate the layout of instruments within any one of the four units. More speci"cally, Figure 2 provides an overhead view of the workspace for one unit. Aside from the panels themselves, there is one CRT to the far right of the panels, a PC on the extreme left of the operator's desk, a number of phones and wireless communication devices on the centre of the desk and four CRTs on the right of the desk. Figure 3 illustrates the relative layout of the CR panels for one unit at the new plant. As is standard in the nuclear industry, the top row contains the alarms, the middle row contains the displays and the bottom row contains the controls. As indicated by the labels along the top, the panels are roughly grouped according to systems. Since we are primarily interested in the impact of computer-based CR technology on operator monitoring, the primary CRTs that are embedded in the panels are also identi"ed in Figure 3 (see below for a more complete inventory). Along the top row, there are four CRT screens that are dedicated to presenting alarm information. Information has been allocated to these four alarms CRTs roughly according to systems. For example, the CRT that is the farthest to the right in the alarm row provides information about heat transport alarms. This top row also contains more traditional hard-wired alarm windows or tiles, although these are not shown in Figure 3 . Referring now to the middle row containing displays, much of this space is allocated to analogue hard-wired instrumentation (not shown in the "gure). However, as illustrated in Figure 3 , there are also four OPERATOR MONITORING FIGURE 2. An approximate bird's eye view of the panels and workspace for one of the four reactor units in the new CR. Not to scale. UNIT WORKSPACE AND PANEL LAYOUT CRTs that are interspersed along the panels. These CRTs are #exible in that they can be used to display any of the numerous computer-based displays that are available to operators (see more detailed description below). Thus, unlike the alarm CRTs in the top row, these CRTs in the middle row are not dedicated. Finally, the bottom row contains the controls, including those that allow operators to select which screens to display in the four #exible CRTs in the middle row. Although this is not shown in Figure 3 , the new CR design has signi"cantly fewer hard-wired instruments on the panels than the older or oldest CRs. In part, this is because some of the functions (e.g. refueling) have been moved to a separate area. There is, however, another reason why there are fewer hard-wired instruments on the panels at the new CR, namely that there are many more CRTs that can be used to bring up computer-based displays than at the oldest CR or even the older CR. In total, we counted 28 CRTs per unit. UNIT WORKSPACE AND PANEL LAYOUT E 1 PC on the LAN, shown on the extreme left of the desk in Figure 2 . E 1 CRT on the operator's desk that is used to display shut down system (SDS) information, shown at the bottom of Figure 2 . E 3 on the operator's desk that are completely #exible (the top 3 on the desk in Figure 2 ). E 1 near the SDS panels to the far right in Figure 2 . This CRT is slaved to the top desk CRT in Figure 2 , and is used to monitor alarms during testing. E 4 that are dedicated to displaying alarms (for the generator, electrical, nuclear, and heattransport systems), shown in the top row of Figure 3 . E 4 completely #exible CRTs embedded in the panels near the meters, shown in the middle row of Figure 3 . E 12 green CRTs embedded in the SDS panels. E 2 small CRTs embedded in the panels. These are used primarily for testing purposes. UNIT WORKSPACE AND PANEL LAYOUT The #exible CRTs are controlled by light pen or dedicated push buttons. In summary, the new CR design seems to have traded o! hard-wired instruments for CRT-based displays. CRT DISPLAY SYSTEM Due to the more prominent role of CRT-based displays at the new CR, it is worthwhile describing them in more detail. The status displays have a hierarchical structure that is menu-driven. There are setpoint displays that can be brought up for each of the control systems for that unit. These displays provide information about the automation (i.e. the instrumentation and control system). There are also displays showing the status of the process. At the highest level in the hierarchy, there are approximately 45 system status displays. For each of these, a variety of sub-displays can be called up, including: more detailed equipment status displays; point data displays for speci"c variables; dedicated trend displays; bar chart displays; and bar trend displays (a hybrid of bar charts and trends). In addition, there is also a &&dummy'' graphical trend capability that allows operators to create and then view, customized trend plots consisting of combinations of operator-speci"ed variables. These customized trend plots can be used to observe data that may not be available on the designer-speci"ed trend displays. Up to 32 variables can be viewed in this way (4 pages;4 variables;2 digital computers). However, it is important to note that only some variables are stored in history. For these variables, OPERATOR MONITORING 839 previous values can be viewed when a dummy graphical trend is brought up. There is an option to put a limited number of variables for which history has not been stored into a historical mode. However, for the remainder of the variables, operators can only view their state beginning from the time that a dummy graphical trend is called up on the screen. When that dummy trend is no longer displayed on the CRT, the history for those variables is unrecorded and cannot be recovered. SUMMARY The computer-based display set at the new CR is both relatively comprehensive and #exible. It consists of many di!erent display screens, many more than can possibly be viewed at any one time, even with the large number of CRTs available (see above). Also, the same display can be brought up on many di!erent CRTs, and the same variable can be viewed in di!erent ways. Furthermore, there is a provision for tailoring the presentation of trend displays, although there are constraints on the number of variables that can be viewed in this way at any one time and on which variables have stored history information. As we will see below, this comprehensiveness and #exibility have interesting implications for operator monitoring. Sources of information for monitoring In previous phases of this research programme, we identi"ed a diverse set of sources of information that operators can consult to monitor the status of a unit in a traditional CR based primarily on analogue, hard-wired technology (Mumaw et al., 2000) . These are summarized in Table 1 . Do operators at the new plant rely on the same set of information sources? Sources of information for monitoring Our observations suggest that the answer is yes. We did not notice some major di!erences in the sources of information used by operators. One di!erence observed, more quantitative than qualitative, is that operators make much more use of computerbased displays at the new CR, which is expected of course, given the di!erences in CR technology described earlier. We also found that operators at the new plant do not seem to walk through the plant as frequently nor did they report relying on unmediated indications (cf. Vicente and Burns, 1996) . Thus, the use of this type of information seems to di!er from that at the oldest plant in particular. The reason for this di!erence is unknown, although it is possible that it takes years of experience to build up the episodic memory required to notice and e!ectively utilize informal information sources. Sources of information for monitoring One pronounced similarity is that operators at the new plant rely heavily on the alarm CRTs to direct their attention to problems. This reinforces the pattern observed both at the older and oldest CRs. What makes monitoring difficult? In previous work, we identi"ed a number of factors that make monitoring di$cult in a traditional CR consisting primarily of analogue hard-wired technology. These factors are summarized in Table 2 . Since the new plant was designed more recently, we would 840 expect that at least some of these di$culties would be ameliorated because of improved design practices and additional operating experience. In the remainder of this section, we discuss the factors that make monitoring di$cult from three complementary perspectives: improvements, new di$culties and remaining di$culties. What makes monitoring difficult? 6.1. IMPROVEMENTS 6.1.1. System reliability. Nuisance alarms appeared with noticeably less frequency than at the older or oldest CRs. Also, operators seemed to have more trust in the indications they received from the panels. Thus, monitoring at the new CR seems to be slightly easier because there are fewer contingencies to take into account in creating and maintaining an accurate situation model (Mumaw et al., 2000) . Alarm system design. Although the design of the alarm system is very similar to those at the older and oldest CRs, it does seem to be improved in at least two ways. First, the new CR apparently has approximately 3000 alarms which is about 3 times more than at the oldest CR (we do not know how many alarms there are at the older CR). As a result, operators are provided with considerably more information that they can use to monitor the unit. Thus, the new CR alarm system design is more diagnostic or sensitive than that at the oldest ssimmons@example.org. Alarm system design. Second, as illustrated in Figure 3 , there are four di!erent alarm CRTs at the new CR, each dedicated to displaying a particular category of alarms (e.g. generator, electrical, nuclear and heat transport systems). In contrast, at the older and oldest CRs, there were only two alarm CRTs for each unit, so the CRT alarm messages were not segregated physically in any way. All alarms were presented in a serial fashion on the same CRTs. Since the alarms are separated by type at the new CR, it is easier for operators to determine where the problem lies (e.g. in the heat transport system rather than the electrical system). They do not have to mentally categorize the alarms because the designers have done this job by allocating di!erent alarms to di!erent CRTs. Also, because each alarm CRT is located near the displays and controls that are pertinent to the system for which it displays alarms, the job of integrating information across alarms and displays is facilitated. For these reasons, we believe that the alarm system design at the new CR makes monitoring slightly easier than at the older and oldest CRs. However, as we discuss below, this does not mean that the alarm system does not have any limitations. Displays and controls design. It was clear that the displays and controls in the new CR are better designed from a human factors point of view. Both the visibility and the layout seem to be improved over both the older and oldest CRs. These improvements facilitate monitoring because they make it easier for operators to extract information from the instruments. For example, each individual light bulb on the panel can be tested easily by pressing down on the bulb. This provides an enormous improvement over the older and oldest designs. Since operators can easily test the validity of these indications, they have a much better idea of which light bulbs are burnt out. As a result, they are much less likely to be misled by false instrument indications. OPERATOR MONITORING 6.1.4. Radio communications. The new plant was constructed so that it is possible to use wireless radios to communicate between the CR and the "eld. This represents a distinct change, at least compared to the oldest plant. From what we could tell, this added information channel can facilitate coordination of activities across individuals distributed over the CR and the "eld. Thus, when monitoring requires coordination with personnel in the "eld, it can be made easier by these #exible and convenient communication devices. Noise and distractions. Although we did not measure them, it seemed that there was less noise and fewer distractions at the new CR than at the other two CRs we had observed. This di!erence may be attributable to the CR layout (see below). Alternatively, it may be due to the fact that access to the CR is better regulated and thus there may be fewer people in the CR. Noise and distractions. 6.1.6. CR layout. As described earlier (see Figure 1) , the new CR has a very di!erent layout from either the older or the oldest designs. This seems to lead to some bene"ts, as well as some disadvantages (see below). The primary bene"t is that sound is less likely to travel from one unit to another. As a result, there does not seem to be a problem with operators not being able to determine if an alarm is coming from their unit or an adjacent unit, as there sometimes was at the oldest CR for instance. Furthermore, the overall noise level seems lower than at the older or oldest CRs. NEW DIFFICULTIES Keyhole ewect. There is another previously observed factor that makes monitoring di$cult, but that is more pronounced at the new CR, so we have treated it as a new category. We are referring to the keyhole e!ect frequently associated with computerbased displays (Woods, 1984; Woods and Watts, 1997) . In this case, the keyhole e!ect arises because of an interaction between several factors: (1) there are fewer analogue instruments on the panels at the new CR (e.g. there is only one strip chart recorder), so operators have to rely more on computer-based displays for monitoring than operators at the older and oldest CRs; (2) each CRT can only present a limited amount of information at a time; and (3) there are many more displays available than there are CRTs that can display them. In contrast, one of the advantages of a traditional hard-wired instrument CR design is that it presents all of the available data in parallel. Thus, operators merely have to move their eyes, or perhaps walk a little bit, to search for new information. Keyhole ewect. The computer-based displays at the new CR create a very di!erent situation. Due to the factors listed above, operators either have to access information by serially bringing up di!erent displays in sequence, or they have to be content to view only a subset of the available data at any one time in parallel (this subset being limited by the number of CRTs available for #exible data presentation). Keyhole ewect. An example can be o!ered to illustrate why the keyhole e!ect can make monitoring more di$cult. As we discuss in the next section, operators dedicate the #exible panel CRTs to displaying valuable plant information to get an overview of the status of the unit. As a result, there are not enough CRTs to comprehensively monitor all of the control systems status displays. In fact, operators usually choose not to display the control systems status screens unless there is a speci"c reason to call them up. Operators therefore rely heavily on the alarm system to detect problems with the control systems (see below). Alternatively, if operators bring up a new display on a particular CRT, they lose the information that was represented on the display that was previously presented on that CRT. Of course, they could bring up that display on a di!erent CRT, but then this would over-write a third display. This example illustrates that the keyhole e!ect makes it more likely that something is happening in the unit that the operator cannot see, given the displays that are currently up on the limited number of CRTs available. Keyhole ewect. Some operators we spoke to said that the keyhole e!ect does not impact their ability to monitor the unit e!ectively. They said that they only need a few displays to maintain an accurate overview of the unit, so even if several CRTs are being used for other purposes, they say that they can still keep an eye on &&the big picture''. Some operators also stated that, if at any time they need to monitor more displays than can be displayed on the number of CRTs available, then they will periodically cycle several displays through the same CRT. Further investigation would be required to determine how big of an impediment the keyhole e!ect is to monitoring at the new CR. REMAINING DIFFICULTIES 6.3.1. System complexity. Although it seems that the new plant is in a better state of repair than the older or oldest plants, it is still the case that there are many components that contribute to the complexity of the plant. Thus, it is still not uncommon for equipment to have to be repaired, maintained or tested. As we noted in our previous article (Mumaw et al., 2000) , these activities temporarily cause the plant to be in a di!erent state or even structure. As a result, the symptoms that are considered normal change accordingly. This change makes monitoring quite di$cult because there is no single, "xed de"nition of what constitutes an expected set of indications. Operators must be able to update their situation model to be consistent with the current state of the plant, and then derive the appropriate referents for that particular state. Given the complexity of the unit (i.e. number of components and possible interactions), these cognitive activities can pose a signi"cant demand on operators. These demands are particularly imposing for operators who are less experienced (see below). Alarm system design. Although the alarm system design at the new CR does seem to represent an improvement over previous designs (see above), it also exhibits some of the same problems that we had observed at the older and oldest CRs. These limitations stem from the fact that the alarm system is not very context-sensitive. As a result, alarms that come in are frequently false positives in the sense that they are highlighting states that are expected given the current situation (e.g. operating mode, testing, maintenance, etc.). This situation can make monitoring challenging because it is up to operators to discriminate alarms that are normal for the current situation from those that are abnormal for the current situation. That is, it may be di$cult for operators to detect OPERATOR MONITORING a signal because of the large degree of noise generated by the alarm system. This is a particularly di$cult task for less experienced operators (see below). Displays and controls design. Although the displays and controls at the new CR are better designed than those at the older and oldest CRs, they still do not always conform to basic human factors design principles. Interestingly, problems exist with both the panel instruments and the CRT displays. Several examples are discussed by Vicente, Mumaw and Roth (1997) . Thus, although it is a more modern design and represents an improvement over the older and oldest CRs, the new CR still has basic design limitations that make monitoring more di$cult than it needs to be. Automation design. Another remaining problem is the feedback o!ered by the automation (i.e. the control systems). With the replacement of many analogue controllers by digital controllers, the information available to monitor the state of the automation is increasingly provided in the form of computer-based displays. Aside from the rudimentary problems identi"ed earlier, these displays also make monitoring di$cult because they do not provide comprehensive information about the various factors that are needed to determine if a control system is operating correctly (i.e. as designed). More speci"cally, to uniquely distinguish automation failures from component failures, operators need to know the state of: the error signal being fed into the controller, the demand signal being sent from the controller to the component being controlled, and the actual state of that component [see Mumaw et al. (2000) for a more detailed discussion]. Some of this information is not provided on some of the control systems status displays. For example, some control systems displays show the demand signal but not the current state, whereas others show the current state but not the demand signal. Thus, it would be di$cult for operators to detect an automation problem unambiguously by directly monitoring these displays. Automation design. Of course, eventually automation failures will lead to one or more alarms if the problem is serious enough. However, these indications can be indirect and can therefore occur at a di!erent system or location in the unit, causing operators to have to trace back to "nd the root of the problem. Also, such indirect indications may happen only after a delay, which puts operators in a position where they are required to react to a problem rather than anticipate it. Automation design. Furthermore, the information that is available on control system status displays is presented in alphanumeric form, making it more di$cult for operators to compare di!erent values. This is an important point because, generally, automation faults can only be unambiguously detected by noticing that certain relationships between variables no longer hold, rather than by merely examining the value of individual variables in isolation. For these reasons, it is di$cult to monitor the state of the automation directly. Automation design. Operators do not seem to be too concerned about this situation for two reasons. First, the automated systems seem to be quite reliable, so they rarely fail. Second, a failure in an automated system should generate one or more alarms. Thus, rather than regularly devoting a great deal of resources for monitoring the status of automated systems via the computer-based displays, operators instead rely extensively on the alarm system to notify them of an automation failure. CR layout. As already mentioned, the new CR layout is di!erent from those at the older and oldest CRs (see Figure 1) . The linear sequencing of the unit panels makes it more di$cult for operators on di!erent units to communicate with each other, especially those who are not on adjacent units. For example, it is more di$cult to determine if another operator is in need of help. Similarly, it is very di$cult for operators to have an awareness of the state of, or the activities going on at, other units. For example, if one operator has to come over to help another during an abnormal event, it will be more di$cult for him to keep an eye on his own unit. Both of these factors can make monitoring more di$cult, although perhaps not substantially so. Thus, di!erences in CR layout seem to lead to a trade-o! between noise level (see above) and ability to monitor other units. CONCLUSIONS As one would expect, the new CR is better designed in several respects than the previous generation designs at the older and oldest CRs, thereby facilitating some monitoring activities. Nevertheless, there are still design de"ciencies, especially in the analogue and computer-based displays. These de"ciencies are important because they make monitoring more di$cult, not because the job itself is more di$cult, but because the job has to be done with an interface that does not support monitoring activities as well as it could. CONCLUSIONS There is a similar &&good news*bad news'' story with respect to the knowledge required to develop and maintain an accurate situation model. On the positive side, the new plant and instrumentation seem to be more reliable and trustworthy than at either the older or oldest plants. Also, failures seem to occur less frequently. Consequently, the demands associated with monitoring are somewhat reduced. On the negative side, however, the plant is still very complex and the focus of training is more on the severe but infrequent events rather than on mundane but frequently encountered tasks (e.g. prioritizing work orders). As a result, the demands associated with maintaining an accurate situation model are quite substantial. That is, despite the noted improvements, monitoring is still a di$cult task. CONCLUSIONS There is one new factor related to the change in CR technology that seems to add to the di$culty associated with monitoring, namely the keyhole e!ect. Although operators state that this is not a problem, the reduced visibility brought about by the keyhole e!ect is a potential threat to e!ective monitoring. CONCLUSIONS In summary, most of the factors that made monitoring di$cult at the older and oldest CRs can also be found at the new CR. Some of these di$culties have been slightly ameliorated, but most of them remain. Monitoring is still a challenging task that requires extensive skill and experience to perform e$ciently and reliably. Does the change in CR technology improve or inhibit operators' abilities to deal with these challenging demands? Strategies that operators use for monitoring In the previous work, we outlined a number of clever strategies that operators have adopted to monitor the plant more e!ectively (Vicente & Burns, 1996; Mumaw et al., 2000) . Do operators at the new CR use the same or a di!erent, set of monitoring strategies? OPERATOR MONITORING BUILDING AND MAINTAINING A SITUATION MODEL At the older and oldest CRs, we noted that some of the operators' activities explicitly serve the purpose of building and maintaining an accurate situation model, which in turn directs their attention and sets their expectations during monitoring activities. These "ndings were replicated at the new CR. For example, the shift turnover serves as an important mechanism for operators coming on shift to update their situation model. Similarly, all operators take printouts at the beginning of a shift and then review these printouts to obtain a more detailed understanding of the current state of the unit. If operators are on an unfamiliar unit, or if they have been away for an extended period, then they will devote extra e!ort to these activities. For example, we saw an operator who had not been on shift for 10 days read all of the logs, take more detailed printouts, and conduct a more thorough turnover. It seems that the e!ort that operators devote to updating their situation model is a function of the time since they last worked on that unit. BUILDING AND MAINTAINING A SITUATION MODEL While our observations in this category largely con"rm our previous "ndings, there was one interesting change that can be attributed to a di!erence in CR technology. At the older and oldest CRs, we had observed operators walk down the control panels at the beginning of a shift to brie#y familiarize themselves with the current state of the unit. In contrast, at the new CR we observed that some operators would instead cycle through a number of di!erent computer-based displays at the start of the shift, essentially for the very same purpose. This example is simple but interesting because it shows that the task goals can be the same in CRs with di!erent technology, but that the behavioural manifestations of these goals are shaped by the CR technology. This theme appeared several times in our observations. KNOWLEDGE-DRIVEN MONITORING Just as at the older and oldest CRs, we found that operators at the new CR frequently engage in knowledge-driven monitoring of the unit. Rather than merely reacting to stimuli in their environment in a data-driven fashion, operators actively seek out speci"c information, as a function of the current situation. Several examples of this type of activity were observed. KNOWLEDGE-DRIVEN MONITORING (1) During refuelling, status alarms are actively sought out to con"rm that everything is going as planned. If the expected pattern of alarms is not obtained, then this is a sign that there is some sort of anomaly. (2) After refuelling, we saw one operator compare the new thermal power value to the old value on a printout to con"rm that the refuelling operation was successful. (3) One operator regularly checks the primary heat transfer pump seals at the beginning of a shift because these seals are known to fail on his unit. Many other similar examples of knowledge-driven monitoring were observed. KNOWLEDGE-DRIVEN MONITORING Again, we observed the shaping e!ect of the CR technology. For instance, if there is a reason to monitor a certain variable particularly closely, operators at the older and oldest CRs might open a strip chart recorder door or create an external reminder to monitor the variable. These strategies o%oad the operators' memory. At the new CR, 846 operators might achieve the identical goal by temporarily bringing up a computer-based display on one of their desk CRTs. A frequently encountered example is the use of context-speci"c, dummy graphical trends during refuelling. Operators will customize a set of trends that are the most relevant to the channels being refueled so that they can closely monitor these variables during refuelling. Note, however, that by temporarily dedicating a CRT to knowledge-driven monitoring, operators have to remove another display from that CRT. Thus, they will not have access to this information (see the earlier discussion of the keyhole e!ect). KNOWLEDGE-DRIVEN MONITORING This contrast in the activities required to achieve the same objective with di!erent CR designs is instructive because it reveals a lesson that may be generalizable. Analog hard-wired instruments are "xed in their location and grouping, so no degrees of freedom are intentionally provided in the design for tailoring the information to a particular local context. As a result, operators have to go outside of the means explicitly provided by the designer to tailor the displays to the ssimmons@example.org. Usually, this involves using ad hoc methods (e.g. opening the door on strip chart recorders, creating external reminders). In contrast, computer-based displays o!er a wealth of designed-in #exibility in terms of information content, location, grouping and form. Operators can exploit these built-in degrees of freedom to tailor the information presentation to the task at hand (e.g. creating a dummy graphical trend display to support monitoring during refuelling). Although these methods of tailoring were not anticipated by designers, they rely on interface functionality that was explicitly built into the CR. Whereas, in hard-wired CR designs, operators have to go outside of the means explicitly provided by designers, in computer-based CR designs, operators can work within the means provided by designers (if the designer had the foresight to provide the relevant degrees of freedom). This "nding will be revisited below. FACILITATING ACTIVITIES In previous phases of this research (Mumaw et al., 2000) , we observed that operators at the older and oldest CRs have developed an ingenious set of strategies that reduce the cognitive demands imposed by monitoring tasks (see Table 3 ). As this subsection shows, the same strategies can be used to capture the facilitating activities of the new CR's operators, although the manifestations of these strategies are again shaped by the CR technology in many cases. Reduce noise. Operators at the new CR also exhibited many di!erent strategies whose common goal is to facilitate monitoring by reducing the noise created by the CR interface. Reduce noise. Some of the strategies that operators used to reduce noise are identical to those exhibited by operators at the older and oldest CRs. For example, operators frequently cursor o! nuisance alarms from the alarm CRT screens. This reduces the clutter on these screens by minimizing the number of uninformative alarm messages. Reduce noise. The shaping in#uence of the CR technology can also be seen in these strategies. At the older and oldest CRs, operators sometimes changed or jumpered alarm setpoints in cases where many annoying nuisance alarms were being generated by a particular variable. Many times, the change of alarm setpoints could be accomplished by taking advantage OPERATOR MONITORING of the adjustable alarm setpoints on some of the analogue meters (a feature that was not explicitly designed for this purpose). At the new CR, there is only one analogue meter of this type on each unit, so operators change and jumper alarm setpoints from the CRTs instead. This provides another example showing how hard-wired instruments force operators to develop ad hoc facilitating activities, whereas #exible computer-based displays can provide built-in means that operators can rely on to facilitate monitoring. Enhance signal. Several cases were observed where operators enhanced the signal available to them in order to facilitate monitoring. For example, when operators want to monitor a particular variable very closely (e.g. annulus gas system), they can change the range of the y-axis on a computer-based trend display. Reducing the range of the variable magni"es small changes in state, and thereby improves the saliency of the signal. This strategy shows how operators can exploit the #exibility provided by designers of computer-based displays to tailor information presentation to a local context. Document baseline or trend. As in our previous studies, we found that operators at the new CR frequently document baselines or trends by writing down values or by printing out screen dumps. These activities facilitate monitoring by o%oading operators' memory. If they
{'paper_id': '8081522', 'title': 'How do operators monitor a complex, dynamic work domain? The impact of control room technology', 's2_url': 'https://api.semanticscholar.org/CorpusID:8081522', 'source': 'S2ORC'}