Shootout-89: A Comparative Evaluation of Knowledge-based Systems that
  Forecast Severe Weather
  During the summer of 1989, the Forecast Systems Laboratory of the National
Oceanic and Atmospheric Administration sponsored an evaluation of artificial
intelligence-based systems that forecast severe convective storms. The
evaluation experiment, called Shootout-89, took place in Boulder, and focussed
on storms over the northeastern Colorado foothills and plains (Moninger, et
al., 1990). Six systems participated in Shootout-89. These included traditional
expert systems, an analogy-based system, and a system developed using methods
from the cognitive science/judgment analysis tradition. Each day of the
exercise, the systems generated 2 to 9 hour forecasts of the probabilities of
occurrence of: non significant weather, significant weather, and severe
weather, in each of four regions in northeastern Colorado. A verification
coordinator working at the Denver Weather Service Forecast Office gathered
ground-truth data from a network of observers. Systems were evaluated on the
basis of several measures of forecast skill, and on other metrics such as
timeliness, ease of learning, and ease of use. Systems were generally easy to
operate, however the various systems required substantially different levels of
meteorological expertise on the part of their users--reflecting the various
operational environments for which the systems had been designed. Systems
varied in their statistical behavior, but on this difficult forecast problem,
the systems generally showed a skill approximately equal to that of persistence
forecasts and climatological (historical frequency) forecasts. The two systems
that appeared best able to discriminate significant from non significant
weather events were traditional expert systems. Both of these systems required
the operator to make relatively sophisticated meteorological judgments. We are
unable, based on only one summer's worth of data, to determine the extent to
which the greater skill of the two systems was due to the content of their
knowledge bases, or to the subjective judgments of the operator. A follow-on
experiment, Shootout-91, is currently being planned. Interested potential
participants are encouraged to contact the author at the address above.
