What are shortcomings of query bases tools for measuring Situation Awareness? A review compared many different ways for assessing Situation Awareness (SA), such as query based tools, external observers and self-report tools.

The paper argues that query based tools, such as the SAGAT (Endsley, 1988) and the (SPAM; Durso et al., 2004), are the most reliable way of assessing SA, compared to the other methods. What are the shortcomings of these query based methods, however?



References

Durso, F. T., Dattel, A. R., Banbury, S., &amp; Tremblay, S. (2004). SPAM: The real-time assessment of SA. A cognitive approach to situation awareness: Theory and application, 1, 137-154.  

Endsley, M. R. (1988, May). Situation awareness global assessment technique (SAGAT). In Aerospace and Electronics Conference, 1988. NAECON 1988., Proceedings of the IEEE 1988 National (pp. 789-795). IEEE.

Endsley, M. R. (2013). Situation awareness-oriented design. The Oxford handbook of cognitive engineering. Oxford University Press, New York, 272-285.
Short Answer

The SAGAT is incredibly intrusive and relies on the subjects' memory, which is fallible. The SPAM is less intrusive and is less dependent on the subjects' memory. However, it is difficult to match the SPAM questions to the decision making process of an operator. More complete explanations of the methods can be found in papers of Pierce (2012) and the review of Endsley (2013).

Long Answer

SAGAT
The most well known query based tools are the Situation Awareness Global Assessment Tool (SAGAT; Endsley, 1988). The SAGAT is used by periodically asking a set of questions about the situation at hand, thereby covering each of the three levels of SA (See Table 2 for examples in the Train Traffic Controller domain). The amount of correctly answered questions is the measure for the quality of SA. 


The main drawback of the SAGAT (or any query based method) is that situation specific questions need to be formulated. The researcher must thus know what will happen in the simulator and how the subjects will try to gain SA and tackle the problems.
Secondly, the SAGAT requires that the task is frozen for a while. The task is thus incredibly intrusive and is not well suited for field studies, because you cannot pause the task in real life. In simulators, the freezes may greatly impact the flow of the task and, subsequently, how the task is performed. 
Beside the freeze, the screen is also made blank, to prevent the subject from looking up the answers. The subject is thus entirely reliable on his memory. Memory is fallible, however, and it may very well be possible that the subject was aware of a cue a couple of minutes ago, but at the moment the SAGAT questions are shown not anymore. On top of that, it isn't even necessary to remember every piece of information; the operator perceives the information, acts on it, and can then forget about it an focus on other parts. 
Finally, with the SAGAT you only get a general idea of one's SA, because of two reasons: (1) without asking hundreds of questions you cannot capture the entire SA of the operator and (2) you only get one score of SA (the sum of correct answers). In other words, you will never understand how SA varied throughout the task. 



SPAM
In an attempt to simplify SA measures, Durso et al (2004) developed the Situation Present Assessment Method (SPAM; see also Pierce et al, 2012). The SPAM uses the same questions as the SAGAT but presents them while the task is ongoing. Instead of a large list of questions, only one or two questions are presented at a time. The questions are in general presented every two to three minutes. The correctness of the answers plus the response times (RTs) are measures of SA. The idea is that when RTs are short knows (where to find) the answer and is therefore aware. Conversely, when RTs are long, the operator did not know the answer or where to find them, and is therefore considered to be less aware. 


The first and foremost shortcoming of the SPAM is that it is extremely difficult to formulate the questions and plan them at the right moments. Decision processes may vary between subjects. Some questions may thus be shown too early for some subjects (they had not attended that information yet), whereas it would be too late for others (it may have been forgotten already). 
Moreover, some subjects may find the content of the question irrelevant; One's SA may thus differ from another, which is not necessarily better or worse. One or two question can also impossibly capture the entire SA. To minimize these issues, it is again extremely important to understand what happens in the scenario and how the subjects will tackle the issues. 
The temporal resolution is also still lacking. Only one or two questions can be asked every two-to-three minutes. Although this is much better than the SAGAT, it still leaves room for a lot of interpretation. 
Finally, because the questions are presented while the task is ongoing, it can greatly increase the amount of workload. Moreover, it may fundamentally change the dynamics of the task; the questions may probe the subjects to focus on particular aspects, thereby losing the natural way of performing the task.