1. Introduction

For many years, transport and dispersion models have been used to forecast areas of airborne ash following volcanic eruptions because of safety and economic issues associated with airborne ash. Volcanic ash can cause abrasion to forward-facing aircraft surfaces, adversely impact aircraft instrumentation, and cause jet engines to fail (Guffanti, et al. 2005). Diverting aircraft or canceling flights increases airline costs (Vanier and Hellroth 2005). Clearly, there is a need for reliable forecasts of volcanic ash transport and dispersion.

In the late 1990s nine Volcanic Ash Advisory Centers (VAACs) were established under the auspices of the International Civil Aviation Organization (ICAO) to monitor, via satellite, the detection of volcanic ash; to run models for the transport and dispersion of volcanic ash; and to issue advisory information on the extent of the observed and forecast movement of the ash (ICAO 2004). The advisory information has been used by meteorologists who issue warnings (SIGMETs, for significant meteorological information) and by personnel at air traffic control facilities and airline operation centers.

Traditionally, transport and dispersion models have been evaluated near ground level using controlled tracer releases by comparing field-measured concentration areas with model-calculated areas using archived analysis meteorology (e.g., van Dop et al. 1998; D’Amours 1998). A compilation of such tracer release information is available on the Internet (Air Resources Laboratory 2006a). However, models simulating the transport and dispersion of volcanic ash are typically evaluated using satellite analyses (e.g., Heffter 1996; Turner and Hurst 2001). Servranckx and Chen (2004) discuss the main factors that influence the accuracy and uncertainties of volcanic ash dispersion modeling: the source, meteorology, and transport and dispersion.

Little, if any, comprehensive information is available pertaining to the reliability of the actual volcanic ash forecast areas themselves, which is of concern for aviation decision making. The goal of this study was to provide an estimate of the reliability of volcanic ash forecast areas under a wide range of meteorological conditions for several volcanoes. In this study, a volcanic ash forecast area is an area depicted on a map that is the location of the ash cloud at some time after the eruption; it is calculated by a transport and dispersion model using a meteorological model forecast. Forecast area reliability was based on comparing ash forecast areas with those ash areas calculated using meteorological model analyses. In effect, the meteorological forecast was evaluated through the application of a transport and dispersion model. The dispersion model itself was not evaluated, and no comparisons to observed volcanic ash were made.

2. Description of the volcanoes, meteorology, and the model

Seven volcanoes representing different meteorological regions were selected for this study (Fig. 1, Table 1) because of recent eruptions or because their airborne ash could impinge on frequently traveled aircraft flight paths. Five of these volcanoes erupted in 2006: Popocatepetl in Mexico, Soufriere Hills on the island of Montserrat, Tungurahua in Ecuador (Washington VAAC 2006), Augustine in Alaska (Anchorage VAAC 2006), and Sheveluch (Shiveluch) in the Kamchatka region of Russia (Tokyo VAAC 2006). Hekla in Iceland last erupted in 2000 (Smithsonian Institution 2006a), but Icelandic volcanoes are of major concern to air traffic. Rainier, in Washington State, was chosen because of its proximity to major northwest U.S. air flight centers; its last eruption may have been in the early 1800s (Smithsonian Institution 2006b).

National Oceanic and Atmospheric Administration–National Centers for Environmental Prediction (NOAA–NCEP) Global Forecast System (GFS; Kanamitsu et al. 1991) and GFS ensemble forecasts (Toth and Kalnay 1997) were used in this study. The GFS output forecast fields were available at 3-h intervals every 6 h on a 1° latitude–longitude grid. The vertical resolution of the output, as processed by NOAA/Air Resources Laboratory (Air Resources Laboratory 2006b), was every 25 hPa from 1000 to 900 hPa, every 50 hPa up to 50 hPa, and at 20 hPa. The ensemble forecast fields were available at 6-h intervals on a 100 km × 100 km grid extracted from the NCEP ensemble output. The vertical resolution was the same as for the GFS. The analysis data consisted of a series of GFS initializations and 3-h forecasts.

The NOAA/Air Resources Laboratory Hybrid Single-Particle Lagrangian Integrated Trajectory (HYSPLIT) transport and dispersion model (Draxler and Hess 1998) was used in this study. The model was run for hypothetical daily eruptions at 0000 UTC from August 2004 through December 2005 to create an archive of 515 days. Due to data transmission and computer problems, forecasts and/or analyses were not archived on 177 days (34%, scattered throughout the entire period), leaving 338 days (66%) for use in this study.

Each 18-h simulation per volcano assumed a 1-h eruption. Model output was a 1-h-average ash area valid 17–18 h after the start of the eruption. Each eruption was simulated as a vertical line source from the volcano summit up to 12 km above mean sea level. Ash particle sizes were 0.3–30 μm with a distribution as given in Heffter and Stunder (1993). Total eruption mass was one unit. Following the practice of the VAAC, wet deposition was not simulated.

HYSPLIT was run with analysis data and three forecast data configurations: GFS, offset, and ensemble. The GFS forecast used the operational GFS forecast meteorology. The offset forecast was run with the HYSPLIT system’s ensemble configuration (Draxler 2003) using the GFS forecast meteorology. This run was composed of nine separate member runs with the meteorological grid shifted one grid point (1°) north, northeast, east, southeast, south, southwest, west, and northwest, and one member with the original data grid. The output concentrations from the nine runs were averaged to give one offset forecast. The ensemble forecast was an average of the HYSPLIT runs using each of the 10 GFS ensemble members’ forecast meteorology. The averaging was a simple method of combining the individual offset or ensemble members and gives a representation of the spatial complexity of the meteorology (see section 4).

3. Comparisons of forecast and analysis modeling

One measure of the degree of similarity between a forecast area and the corresponding analysis area is the threat score (TS) or critical success index (CSI), which is defined as

View Expanded

where F and AF form the forecast area, A and AF form the corresponding analysis area, and AF is the intersect area (Fig. 2). Threat score is commonly used in meteorological verification and is described in numerous texts (e.g., Jolliffe and Stephenson 2003; Wilks 2006) and other works (e.g., Doswell et al. 1990; Marzban 1998; Schaefer 1990). In meteorological verification terminology, AF is called hits, A is misses, and F is false alarms. Some air quality verification work (e.g., Mosca et al. 1998; Chang and Hanna 2004) uses a statistic called the figure of merit in space (FMS), which is identical to the TS, though the FMS is given as a percent instead of a fraction. The TS ranges from zero (no intersect area) to one (perfect match).

The TS statistic has some qualifications. The TS is sensitive to the size of the intersection (overlap; AF in Fig. 2) of the forecast and analysis area and the dimensions of the forecast and analysis areas. Low TS values can result from small overlap areas and/or large analysis or forecast areas. A low TS can also result from similar analysis and forecast areas, but displaced from one another due to a low quality wind direction forecast. The current analysis did not relate the forecast or analysis area sizes to the TS, nor did it investigate forecast wind direction effects, both of which were beyond the scope of the present study. The TS also does not account for areas with no ash in the analysis that are correctly forecast, areas that are not of interest here (i.e., area outside both the A + AF and F + AF areas in Fig. 2).

Given the TS of all volcanoes, the median was 0.45, and ranged from 0 to 0.8. Many forecast–analysis pairs in this study were visually examined to estimate whether, by comparison with the analysis, the forecast would have been acceptable or unacceptable for aircraft flight operations. Virtually all forecasts with a TS smaller than 0.15 were unacceptable and those with a TS greater than 0.35 were acceptable. To identify a TS threshold for acceptable forecasts, forecast–analysis pairs with a TS between 0.15 and 0.35 for all volcanoes were visually examined. For illustration, Fig. 3 shows an acceptable forecast and Fig. 4 shows an unacceptable forecast for the Rainier volcano. Figure 5 shows the percent of acceptable and unacceptable forecasts as a function of TS for TS values between 0.15 and 0.35. Forecasts that could not be classified as either acceptable or unacceptable were not included. A TS value of 0.25 has been selected to separate predominantly acceptable from unacceptable forecasts because it occurs just after the large change in performance between TS 0.21 and 0.24.

4. Forecast classification

It was hypothesized that forecast reliability would vary depending on the meteorological situation. Hence, forecasts were classified by quadrant with respect to each volcano (northeast, NE; southeast, SE; southwest, SW; and northwest, NW) and by the offset forecast excess area. Figure 6 shows an idealized example of a forecast area (F, white oval) compared to its corresponding offset forecast area (O + F), and the offset forecast excess area (O, hatching only). Here, O is the portion of the offset forecast area outside the forecast area and is considered in this study to be related to the complexity of the forecast meteorology acting on the volcanic ash. When there was minimal meteorological variability, the offset forecast ash area usually was small compared to the forecast area. When there was meteorological complexity (e.g., weather fronts or deep low pressure weather systems), the offset forecast ash area was relatively large compared to the forecast area. An offset excess percent (OX) was calculated as follows:

View Expanded

where O and F are the sizes of the offset excess and forecast areas, respectively.

The 25th percentile, median, and 75th percentile of the offset excess distribution were identified so that the meteorological complexity for each forecast could be classified by the offset excess quartile. Hence, two parameters were used to classify each forecast: the forecast area quadrant with respect to the volcano and the offset excess quartile.

An alternative measure of the meteorological complexity was to similarly define an ensemble excess percent quartile. The ensemble excess percent (EX) was calculated for each forecast in the same manner as for the offset excess percent:

View Expanded

where E is the size of the ash area calculated from the NCEP global ensembles. The ensemble provided an “envelope” of dispersion possibilities, since each ensemble member was based on the uncertainty in the GFS analyses. The excess ash area from an ensemble forecast run compared to the forecast ash area can be viewed similarly to the offset excess area. An example ensemble forecast is shown in Stunder and Heffter (2004).

Note that the ensemble and offset excess areas are analogous to the spread parameter used in meteorological ensemble forecasting. In general, some positive correlation between meteorological ensemble spread and forecast skill has been found (e.g., Scherrer et al. 2004; Whitaker and Loughe 1998).

5. Volcanic ash forecast area reliability (VAFAR) results

For each volcano, forecast area quadrant, and offset (or ensemble) excess quartile, the volcanic ash forecast area reliability (VAFAR) was calculated based on the number of forecast–analysis pairs with TS greater than the critical value of 0.25 (N_TS>0.25) divided by the total number of forecast–analysis pairs (N_total):

View Expanded

As an illustration, out of 22 forecast–analysis pairs for Popocatepetl when the forecast area was in the southwest quadrant and for the highest offset excess quartile (4), there were 13 cases of forecast–analysis pairs with a TS greater than the critical value of 0.25. Hence, the VAFAR = 100 (13/22) = 59%.

VAFAR calculations were made for each volcano, in each forecast area quadrant, and within each offset excess [Eq. (2)] and ensemble excess [Eq. (3)] quartile in that quadrant. Table 2 shows the VAFAR values for all volcanoes, by forecast area quadrant and offset excess quartile, and shows the offset excess quartile boundaries. (Note that the example VAFAR value of 59 for Popocatepetl in the southwest quadrant reflects the calculation illustrated above.) Similarly, Table 3 shows the VAFAR values with respect to the ensemble excess.

For comparison purposes, the VAFAR values in Tables 2 and 3 are shown graphically in Fig. 7. The VAFAR values, for the most part, are quite impressive, with most above 70 and some approaching 100. There are, however, several exceptions—most notably the lower VAFAR values for the Popocatepetl volcano in the SE, SW, and NW forecast area quadrants. It is unknown why these forecasts for Popocatepetl have lower VAFAR values compared to the volcanoes in other regions. Further investigation of the meteorological forecasts could be helpful but is beyond the scope of this study. Isolated low VAFAR values (e.g., Tungurahua-SE-quartile 2, Rainier-SW-quartile 3, Popocatepetl-NW-quartile 4) may be due to a low frequency of occurrence in the category and may fall into the more general pattern if more data were available. In general, the VAFAR values tend to decrease as offset excess or ensemble excess increases (moves into higher quartiles), reflecting an increase in the complexity of the forecast meteorology.

In an operational setting, when there is a similar eruption (1-h duration, vertical line source from the summit to 12 km above sea level) of one of the volcanoes studied, a VAFAR value may be assigned to that forecast, and the forecast interpreted accordingly. If the forecast area quadrant with respect to the volcano and the offset or ensemble excess is known, a forecaster may assign the appropriate VAFAR value from Tables 2 or 3 to the forecast. When offset excess or ensemble excess quartiles are not available, Table 4 can be used to provide a less robust VAFAR value by simply estimating the forecast area quadrant with respect to the volcano. Table 4 can also be used when VAFAR is relatively independent of the excess quartile (Fig. 7). Alternatively, for any volcanic eruption, offset or ensemble forecast output may suggest a qualitative evaluation of the forecast based on the spread of the member forecast areas.

6. Conclusions

Comparisons of archived volcanic ash forecast and analysis areas were used to estimate the volcanic ash forecast area reliability (VAFAR) for hypothetical large eruptions (1-h duration, vertical line source from the summit to 12 km above sea level) of several volcanoes. The focus was on the meteorological forecast rather than the transport and dispersion model. The ash forecast is important for aircraft operations because of the hazardous nature of volcanic ash.

Results showed generally favorable VAFAR values, greater than 70%, for the short-term, 18-h, forecast. VAFAR values for two of the volcanoes, Popocatepetl and Tungurahua, tended to be lower than for the others, indicating less accuracy of the meteorological forecast in those regions. VAFAR values varied somewhat by the transport direction as indicated by the forecast area quadrant and by the offset or ensemble excess quartile. In operations, the VAFAR values can be assigned to a forecast when one of the studied volcanoes erupts in a similar manner as in the simulations. Although the simulated eruptions were at 0000 UTC, the VAFAR values should apply for other eruption times at all of the volcanoes because of the typically much larger eruption height compared to the typical planetary boundary layer (PBL) height above the volcano summit. Extension at Soufriere Hills during times of high PBL height, because of its low summit, is less certain. Application to nearby volcanoes is reasonable within areas of similar meteorology, but extension to other regions of the earth is much more uncertain.

Use of the GFS ensemble excess and the HYSPLIT offset excess, indicating the complexity of the meteorology acting on the ash, gave generally comparable VAFAR values.