Title: Creating a Cybersecurity Concept Inventory: A Status Report on the CATS Project,
Abstract: We report on the status of our Cybersecurity Assessment Tools (CATS) project
that is creating and validating a concept inventory for cybersecurity, which
assesses the quality of instruction of any first course in cybersecurity. In
fall 2014, we carried out a Delphi process that identified core concepts of
cybersecurity. In spring 2016, we interviewed twenty-six students to uncover
their understandings and misconceptions about these concepts. In fall 2016, we
generated our first assessment tool--a draft Cybersecurity Concept Inventory
(CCI), comprising approximately thirty multiple-choice questions. Each question
targets a concept; incorrect answers are based on observed misconceptions from
the interviews. This year we are validating the draft CCI using cognitive
interviews, expert reviews, and psychometric testing. In this paper, we
highlight our progress to date in developing the CCI.
The CATS project provides infrastructure for a rigorous evidence-based
improvement of cybersecurity education. The CCI permits comparisons of
different instructional methods by assessing how well students learned the core
concepts of the field (especially adversarial thinking), where instructional
methods refer to how material is taught (e.g., lab-based, case-studies,
collaborative, competitions, gaming). Specifically, the CCI is a tool that will
enable researchers to scientifically quantify and measure the effect of their
approaches to, and interventions in, cybersecurity education.