During the summer of 1989, the Forecast Systems Laboratory of the National Oceanic and Atmospheric Administration sponsored an evaluation of artificial-intelligence-based systems that forecast severe convective storms. The evaluation experiment, called Shootout-89, took place in Boulder, Colorado, and focused on storms over the northeastern Colorado foothills and plains.
Six systems participated in Shootout-89: three traditional expert systems, a hybrid system including a linear model augmented by a small expert system, an analogue-based system, and a system developed using methods from the cognitive science/judgment analysis tradition.
Each day of the exercise, the systems generated 2–9-h forecasts of the probabilities of occurrence of nonsignificant weather, significant weather, and severe weather in each of four regions in northeastern Colorado. A verification coordinator working at the Denver Weather Service Forecast Office gathered ground-truth data from a network of observers.
The systems were evaluated on several measures of forecast skill, on timeliness, on ease of learning, and on ease of use. They were generally easy to operate; however, they required substantially different levels of meteorological expertise on the part of their users, reflecting the various operational environments for which they had been designed. The systems varied in their statistical behavior, but on this difficult forecast problem, they generally showed a skill approximately equal to that of persistence forecasts and climatological forecasts.
*National Oceanographic Atmospheric Association (NOAA), Forecast Systems Laboratory, 325 Broadway, Boulder, CO 80303
**Atmospheric Environmental Service (AES), Arctic Weather Centre, Twin Atria Bldg., 4999 98th Ave., Edmonton, Alberta T6B 2X3, Canada
***Atmospheric Environmental Service, Forecast Research Division, 4905 Dufferin Street, Downsview, Ontario M3H 5T4, Canada
+Cooperative Institute for Research in Environmental Sciences, University of Colorado, Campus Box 216, Boulder, CO 80309-0216
++University of Nevada at Las Vegas, Environmental Research Center, 4505 S. Maryland Parkway, Las Vegas, NV 89154
°Consultant's Choice, Inc., 8800 Rowell Road, Suite 130, Atlanta, GA 30350
°°NOAA/NESDIS/RAM Branch, Colorado State University, Foothills Campus, CIRA Bldg., Fort Collins, CO 80523
#Micro Forecasts, Inc., 319 SW Washington, Suite 909, Portland, OR 97204
##Center for Policy Research, Milne 300, State University of New York–Albany, 135 Western Avenue, Albany, NY 12222
@NOAA/National Weather Service, Office of Meteorology, 8060 13th Street, Silver Springs, MD 20910
