Skills training for mass casualty incidents (MCI) shows a lack of reliable and valid assessment criteria; therefore, feedback about incident management is based either on expert judgment or self-reflection by participants. A questionnaire was developed as a means to support the tactical appraisal of MCI training and to close the methodological gap. In this study four groups of emergency medical response personnel (N = 86) carried out three MCI scenarios and rated the incident management by the use of Likert scale items. Scale construction relied on exploratory factor analysis (EFA). Multiple item (Likert) scales were tested with respect to reliability by internal consistency measures. The diagnostic potential of the scales was analyzed by exploratory mean (scale values) comparisons. The EFA revealed a model with three factors labelled resource allocation, goal orientation and stress management. The multiple item (Likert) scales showed satisfactory reliability for group comparisons based on scale values. Subsequent exploratory analysis of differences between scenarios and tactical interventions demonstrated the diagnostic potential of the scales. The new questionnaire can be used in the context of training debriefing and research. The scales have convincing psychometric properties; however, research into the effectiveness of the tactical dimensions is still required.