a. An experimental NWP dataset: Pseudo–global ensemble forecast system (GEFS)

An experimental NWP dataset was created and used as input into GTG, hereafter referred to as the “pseudo-GEFS.” The pseudo-GEFS forecasts consist of 21-member ensemble forecasts for 12-, 18-, 24-, 30-, and 36-h forecasts at a 13-km horizontal grid spacing on 50 vertical sigma levels with a model top at 10 hPa. This pseudo-GEFS dataset was generated through dynamical downscaling of the publicly available GEFS forecasts dataset from the NOAA National Centers for Environmental Information (NCEI) (https://www.ncei.noaa.gov/products/weather-climate-models/global-ensemble-forecast) at 1° horizontal resolution on 26 pressure levels from 1000 to 10 hPa, consisting of 21 NWP ensemble members—1 control (unperturbed) and 20 perturbation forecasts. The GEFS ensemble is configured by perturbing initial conditions and employing stochastic physics schemes (Zhou et al. 2017), and the pseudo-GEFS is designed to mimic these characteristics of the GEFS ensemble.

For dynamical downscaling, the Weather Research and Forecasting (WRF) Model version 4.2.1 was configured over the contiguous United States (CONUS) domain and run at a 13-km horizontal grid spacing. The WRF Model and simulation setup are provided in the appendix. The 1° GEFS 0-h forecasts were used to initialize the pseudo-GEFS forecasts, and 1° GEFS forecasts were used to provide lateral boundary condition (LBC) forcings, respectively, as illustrated in Fig. 1. The 12-h 21-member pseudo-GEFS forecasts valid at 1800 UTC were initialized every 0600 UTC over a 1-yr period between 21 May 2019 and 20 May 2020.

View Full Size

Illustration of pseudo-GEFS forecasts valid at T₀ (marked by red circles). The 1° GEFS 0-h forecasts were used to initialize the pseudo- GEFS forecasts, and the 1° GEFS forecasts were used to provide LBC forcings to the pseudo-GEFS forecasts. Gray triangles show when and what GEFS forecasts are used to provide LBC forcing to the pseudo-GEFS forecasts.

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

The 13-km horizontal grid spacing used for the pseudo-GEFS is comparable to those currently employed in operational global deterministic forecasts—e.g., 13 km of NCEP GFS (NCEP 2022), 10 km of UKMO UM (Walters et al. 2019), and 9 km of ECMWF HRES (Haiden et al. 2021), but is somewhat higher resolution than those currently used in operational global ensemble forecasts—e.g., 25 km of NCEP GEFS and 20 km of UKMO MOGREPS and ECMWF EPS. The vertical resolution is also similar to the “raw” GEFS forecasts (∼50–54 hybrid levels below 10 hPa), i.e., before the data were interpolated to constant pressure levels. This downscaled GEFS ensemble output is needed because high-resolution input (close to operational NWP resolution) with all needed input variables into GTG is not available from the publicly available 1° GEFS data. This then provides an experimental ensemble NWP system that mimics the operational NWP ensemble forecasting method. The pseudo-GEFS dataset was created over a 1-yr period between 21 May 2019 and 20 May 2020, for 345 days on which the full 21 GEFS ensemble forecasts are available from the NOAA NCEI archive.

We begin by examining the ensemble spreads for two particular cases of observed widespread clear-air turbulence (CAT). Figure 2 shows the observations—in situ eddy dissipation rate (EDR) and pilot reports (PIREPs)—associated with the two cases. The 3 December 2019 case (Fig. 2a) was studied in detail by Trier et al. (2022) and was shown to be mainly associated with strong jet stream shears (CAT) but also with some embedded MWT over Colorado. The 11 January 2020 case (Fig. 2b) was associated with MWT over the Colorado Rockies, CAT over Illinois, and convectively induced turbulence (CIT) over the Mississippi Valley.

View Full Size

In situ EDR and PIREP observations of moderate (MOD) and severe (SEV) turbulence at upper levels (≥20 000 ft) and within a ±1-h time window with symbols specified in the legend, and lightning area (shaded in orange): (a) 1800 UTC 3 Dec 2019 and (b) 1800 UTC 11 Jan 2020. The cross marks in cyan denote the horizontal location of the Grand Junction station (39.12°N, 108.53°W).

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

The GEFS and the pseudo-GEFS forecasts for different forecast lead times are compared in terms of the ensemble spread of the 250-hPa geopotential height and wind speed for these two cases in Fig. 3. For the 3 December 2019 case (Fig. 3a) the pseudo-GEFS ensemble compares well with the GEFS ensemble in terms of the location and the amount of the ensemble spread and the increase of the spread with the forecast lead time while providing the needed meteorological variables at higher horizontal and vertical resolution into GTG. For the 11 January 2020 case (Fig. 3b) the ensemble spread of the pseudo-GEFS compares well with the GEFS ensemble spread, except for the large spread of wind speed from Mississippi all the way to Michigan and Wisconsin. This large spread in the pseudo-GEFS coexists with squall lines developed at the time indicated by lightning observations (Fig. 2b), which are partially resolved in the 13-km pseudo-GEFS but remain unresolved in the 1° GEFS. This study concentrates on the prediction of CAT and MWT, therefore the similarity between the pseudo-GEFS and GEFS over the non-convective areas supports the use of the pseudo-GEFS as a substitute for the operational GEFS.

View Full Size

Ensemble spreads of 250-hPa geopotential height (dam) (contour lines, 1-dam interval) and wind speed (m s⁻¹) (shaded) derived from GEFS (1°) and pseudo-GEFS (13 km) forecasts valid at (a) 1800 UTC 3 Dec 2019 and (b) 1800 UTC 11 Jan 2020, respectively, for forecast lead times of 12, 24, and 36 h.

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

To show the effects of increased horizontal and vertical resolutions, vertical profiles of gradient Richardson number (Ri = N²/S²) derived from the publicly available 1° GEFS vertically interpolated and the 13-km pseudo-GEFS 12-h forecasts are compared in Fig. 4, together with its components, the vertical wind shear {S ≡ [(∂u/∂z)² + (∂υ/∂z)²]^1/2 where u and υ are zonal and meridional wind components and z is height}, and the square of Brunt–Väisälä frequency [N² = (g/θ)∂θ/∂z where θ is potential temperature of dry air]. The S, N, and Ri terms were calculated at every vertical level of the pseudo-GEFS and GEFS forecasts, using the simple centered differences. These forecasted soundings are compared against high vertical resolution radiosonde data (HVRRD) observations at the Grand Junction station in Colorado (39.12°N, 108.53°W). The location of the Grand Junction station is indicated in Fig. 2, which is close to where a number of moderate and/or severe turbulence reports were recorded. The raw HVRRD soundings at a 1-s sampling frequency are available from the NOAA NCEI (https://www.ncei.noaa.gov/data/us-radiosonde-bufr/), and we used soundings at a 5-m grid spacing that are interpolated from the 1-s raw soundings (as in Ko et al. 2019). Since no sounding observations are available at 1800 UTC for both cases, we used the 1200 UTC sounding observations for the comparison. The HVRRD soundings are displayed every 50 m for better representation; however, no interpolation or smoothing was applied for the calculations of S, N², and Ri.

View Full Size

Vertical profiles of (a),(d) the magnitude of vertical wind shear (s⁻¹); (b),(e) the square of Brunt–Väisälä frequency (N²); and (c),(f) gradient Richardson number (Ri = N²/S²) at the Grand Junction station (39.12°N, 108.53°W) derived from the 1° GEFS (solid black with dots indicating vertical grid spacing) and 13-km pseudo-GEFS (dashed black) 12-h forecasts valid at (top) 1800 UTC 3 Dec 2019 and (bottom) 1800 UTC 11 Jan 2020. The HVRRD observation soundings available at 1200 UTC for each case are presented for comparison (solid gray); the HVRRD soundings are displayed every 50 m for better representation. The horizontal location of the Grand Junction station is indicated in Fig. 2.

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

The 3 December 2019 case (Figs. 4a–c) shows that the pseudo-GEFS forecast has larger shears and consequently smaller Ri at upper levels (e.g., 11–13 km MSL) similar to HVRRD observations, creating more favorable conditions for turbulence generation than the publicly available 1° GEFS. The 11 January 2020 (Figs. 4d–f) case also shows that the pseudo-GEFS forecast predicts larger shears and smaller Ri in the lower stratosphere. The coarse and vertically interpolated publicly available 1° GEFS forecasts (vertical grid spacings are indicated by the dots in the figure) predicts Ri larger than 10 at 11–13 km MSL for the December case and above ∼8 km MSL for the January case. For both cases, the general vertical structure of the computed quantities for the most part follow the HVRRD data, but with much smoother distributions.

b. The GTG system and deterministic forecasts

For these comparative experiments the GTG system (Sharman et al. 2006; Sharman and Pearson 2017) was used to compute various turbulence diagnostics and ultimately probabilities of turbulence using the pseudo-GEFS weather forecasts as input. Both CAT and MWT forecasts are provided. In GTG, multiple turbulence diagnostics in various units and scales are computed from the NWP meteorological variables and remapped to eddy dissipation rate (EDR = ε^1/3; m^2/3 s⁻¹), the International Civil Aviation Organization (ICAO) standard metric for automated turbulence reporting. Then an ensemble mean of the remapped diagnostics is computed for CAT and MWT, separately:

${\text{GTG}}_{\text{CAT}}={\displaystyle \sum }_{i=1}^{N_{\text{CAT}}}{D}_{{\text{CAT}}_i}/N_{\text{CAT}},\hspace{0.17em}$(1a)

${\text{GTG}}_{\text{MWT}}={\displaystyle \sum }_{i=1}^{N_{\text{MWT}}}{D}_{{\text{MWT}}_i}/N_{\text{MWT}},\hspace{0.17em}$(1b)

where $D_{{\text{CAT}}_i}$ and $D_{{\text{MWT}}_i}$ are remapped (to EDR) CAT and MWT diagnostics for the ith diagnostic, respectively; and N_CAT and N_MWT are the number of CAT and MWT diagnostics, respectively. Finally, the maximum of the two is taken to be the final combined GTG deterministic forecast (GTGMAX):

$\text{GTGMAX}=\text{MAX}({\text{GTG}}_{\text{CAT}},{\text{GTG}}_{\text{MWT}})\cdot \hspace{0.17em}$(2)

Various studies have demonstrated that forecasting turbulence based on multiple turbulence diagnostics encompassing several turbulence generation mechanisms is superior to using only a single turbulence diagnostic (e.g., Sharman et al. 2006; Gill and Buchanan 2014; Sharman and Pearson 2017; Kim et al. 2018). Therefore, a total of 62 CAT and 15 MWT diagnostics are available in the version of GTG used in this study, and 11 CAT and 8 MWT diagnostics, a total of 19, among them are used to compute GTGMAX (Table 1). The diagnostics used here are slightly different than those used in Sharman and Pearson (2017) based on reevaluations with newer observations and updates to the NOAA Rapid Refresh (RAP) NWP model. The remapping of turbulence diagnostics to EDR and the selection of the 19 diagnostics used for the ensemble mean are the same as calibrated for the NOAA RAP version 4 forecasts over the CONUS domain, which uses the same horizontal and vertical resolution as the pseudo-GEFS model, following the procedure outlined in Sharman and Pearson (2017).

Table 1

The list of 11 CAT and 8 MWT diagnostics used. The variable d_s (m s⁻¹) is a near-surface MWT diagnostic: d_s = 0, if model terrain height (h) < 500 m or gradient(h) < 6.0 m km⁻¹; otherwise, d_s = vertical wind speed maximum in the lowest 1500 m.

View Table

c. Probabilistic forecasts

Probabilistic turbulence forecasts are generated by calculating percentages of the number of ensemble members that fall within predefined EDR ranges: e.g., “light” (0.15 ≤ EDR < 0.22), “moderate” (0.22 ≤ EDR < 0.34), and “severe or greater (SOG)” (EDR ≥ 0.34) turbulence for midsized aircraft, the same as those used in Sharman and Pearson (2017) based on the guidance provided by Sharman et al. (2014). Herein, we tested three stand-alone methods to produce probabilistic turbulence forecasts—a multi-diagnostic ensemble (MDE) approach, a time-lagged NWP ensemble (TLE) approach, and a forecast-model ensemble (FME) (or an ensemble NWP model) approach, and two combined methods—a time-lagged multi-diagnostic ensemble (TMDE) approach and an ensemble-model multi-diagnostic (FMDE) approach. Table 2 summarizes the five probabilistic forecast methods tested in this study with details as follow.

Table 2

Summary of probabilistic forecast methods used in this study: the number of weather forecasts N_Wx (identical to the number of GTG forecast runs), the number of turbulence diagnostics N_D (=N_CAT + N_MWT), the type of ensemble members (individual diagnostics vs GTGMAX), the number of total ensemble members used to compute probability N_ENS, and approximate computing runtimes using the runtime of MDE-19D as a baseline; N_ENS = N_Wx × N_D if individual turbulence diagnostics are used as ensemble members; N_ENS = N_Wx if the final GTG products, GTGMAX, are used as ensemble members.

View Table

1) Multi-diagnostic ensemble (MDE) approach

The MDE approach represents the uncertainty in diagnosing turbulence for a deterministic weather state (i.e., a single NWP input into GTG). This approach interprets individual remapped turbulence diagnostics calculated in GTG as ensemble members, and the number of ensemble members (N_ENS) is equal to the number of turbulence diagnostics (N_D = N_CAT + N_MWT; Table 2). The MDE approach to compute probabilities showed promise in predicting CAT and MWT based on an earlier version of NCEP’s GFS model (Kim et al. 2018). In the MDE approach, the probability of the remapped diagnostics is computed for CAT and MWT, separately, and then the maximum of the two is taken to be the final combined GTG probabilistic forecast (Prob):

${\text{Prob}}_{\text{CAT}}={\displaystyle \sum }_{i=1}^{N_{\text{CAT}}}{I}_{{\text{CAT}}_i}/N_{\text{CAT}}\times 100\hspace{0.17em}(\%),\hspace{0.17em}$(3a)

${\text{Prob}}_{\text{MWT}}=\hspace{0.17em}{\displaystyle \sum }_{i=1}^{N_{\text{MWT}}}{I}_{{\text{MWT}}_i}/N_{\text{MWT}}\times 100\hspace{0.17em}(\%),$(3b)

$\text{Prob}=\text{MAX}({\text{Prob}}_{\text{CAT}},{\text{Prob}}_{\text{MWT}}),\hspace{0.17em}$(4)

where $I_{{\text{CAT}}_i}$ and $I_{{\text{MWT}}_i}$ are binary outcomes for detecting CAT and MWT for the ith diagnostic (1 if a turbulence event is forecasted and 0 if it is not), respectively. The probabilities are computed over the entire x–y grid at all FL ≥ 20 000 ft (∼6 km).

An example of three representative ensemble members of the MDE method is shown in Figs. 5a–c, which are derived from the control (unperturbed) NWP member of the pseudo-GEFS 12-h forecasts valid at 1800 UTC 3 December 2019. Ellrod3 (Table 1) does capture moderate and severe turbulence events over eastern Colorado, as well as a cluster of light and moderate turbulence events along the Nevada–Utah border, but misses light turbulence events near the Arizona–New Mexico border (Fig. 5a). FTH/Ri (Table 1) predicts the light turbulence events near the Arizona–New Mexico border, but the turbulence events in Florida are not captured (Fig. 5b). VARE/Ri (Table 1) tends to predict larger areas of turbulence; this leads to the prediction of the light turbulence events near the Arizona–New Mexico border and over Florida, but also a large number of false alarms over the southeastern United States (Fig. 5c). These differences among the turbulence diagnostics show the uncertainty that may be associated with diagnosing turbulence.

View Full Size

An example of three ensemble members of the MDE method: (a) Ellrod3, (b) FTH/Ri, and (c) VARE/Ri remapped to EDR scales at FL370, valid at 1800 UTC 3 Dec 2019; an example of ensemble members of the TLE method: (d) 12-h, (e) 24-h, and (f) 36-h GTGMAX forecasts. Cross marks are light or greater (LOG) in situ EDR reports (EDR ≥0.15) within ±1-h time window and ±1000-ft altitude colored by EDR scales (m^2/3 s⁻¹).

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

For this comparative study, two different sets of diagnostics were tested. One uses 19 diagnostics (11 CAT and 8 MWT, as listed in Table 1) and the other uses 77 diagnostics (62 CAT and 15 MWT) that are available in the GTG version used in this study: hereafter, referred to as MDE-19D and MDE-77D, respectively (Table 2). All diagnostics used have been mapped to EDR values using the method outline in Sharman and Pearson (2017).

a. Case studies: Characteristics of different approaches

Ensemble spreads from 12-h turbulence forecasts (EDR) are compared in Fig. 6 for the 3 December 2019 case, together with the corresponding deterministic EDR forecasts derived from the control (unperturbed) NWP member of the pseudo-GEFS weather forecasts. The 3 December 2019 case was dominated by CAT along a jet with the jet core approximately at 300 hPa exceeding 65 m s⁻¹ over southern Colorado (not shown). The deterministic forecasts overlaid with in situ EDR reports show moderate and severe turbulence reports over eastern Colorado, as well as a cluster of light and moderate turbulence reports along the Nevada–Utah border (Fig. 6h). Some MWT reports were also recorded over the Colorado Rockies (Trier et al. 2022).

View Full Size

(a)–(g) Ensemble spreads of EDR (m^2/3 s⁻¹) at FL370 derived from 12-h probabilistic forecasts valid at 1800 UTC 3 Dec 2019, together with (h) 12-h deterministic EDR forecasts derived from the control (unperturbed) NWP member of the pseudo-GEFS. In (a)–(g), in situ EDR reports of light (green cross marks), moderate (yellow cross marks), and SOG (red cross marks) turbulence within ±1-h time window and ±1000-ft altitude are shown; in (h), the same in situ EDR reports are colored by EDR scales, with null-to-light reports in white dots.

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

For the TLE and FME approaches that use GTGMAX to compute probabilities [i.e., Eq. (5)], the ensemble spread is defined as the standard deviation (STD) of GTGMAX with respect to the ensemble mean:

$\text{Spread}=\hspace{0.17em}{\left[{\displaystyle \sum }_{i=1}^{N_{\text{Wx}}}{({\text{GTGMAX}}_i-\overline{\text{GTGMAX}})}^2/N_{\text{Wx}}\right]}^{1/2},$(8a)

$\overline{\text{GTGMAX}}=\hspace{0.17em}{\displaystyle \sum }_{i=1}^{N_{\text{Wx}}}{\text{GTGMAX}}_i/N_{\text{Wx}}\cdot$(8b)

For the MDE, TMDE, and FMDE approaches that compute probabilities of CAT and MWT separately [i.e., Eqs. (3), (4) and (6), (7)], the maximum of the ensemble spreads of CAT and MWT is defined as the final ensemble spread:

${\text{Spread}}_{\text{CAT}}=\hspace{0.17em}{[{\displaystyle \sum }_{j=1}^{N_{\text{Wx}}}{\displaystyle \sum }_{i=1}^{N_{\text{CAT}}}{(D_{{\text{CAT}}_{i,j}}-\overline{{\text{GTG}}_{\text{CAT}}})}^2/(N_{\text{Wx}}\times N_{\text{CAT}})]}^{1/2},$(9a)

$\overline{{\text{GTG}}_{\text{CAT}}}=\hspace{0.17em}{\displaystyle \sum }_{j=1}^{N_{\text{Wx}}}{\displaystyle \sum }_{i=1}^{N_{\text{CAT}}}{D}_{{\text{CAT}}_{i,j}}/(N_{\text{Wx}}\times N_{\text{CAT}}),$(9b)

${\text{Spread}}_{\text{MWT}}=\hspace{0.17em}{\left[{\displaystyle \sum }_{j=1}^{N_{\text{Wx}}}{\displaystyle \sum }_{i=1}^{N_{\text{MWT}}}{(D_{{\text{MWT}}_{i,j}}-\overline{{\text{GTG}}_{\text{MWT}}})}^2/(N_{\text{Wx}}\times N_{\text{MWT}})\right]}^{1/2},$(9c)

$\overline{{\text{GTG}}_{\text{MWT}}}=\hspace{0.17em}{\displaystyle \sum }_{j=1}^{N_{\text{Wx}}}{\displaystyle \sum }_{i=1}^{N_{\text{MWT}}}{D}_{{\text{MWT}}_{i,j}}/(N_{\text{Wx}}\times N_{\text{MWT}}),$(9d)

$\text{Spread}=\text{MAX}({\text{Spread}}_{\text{CAT}},{\text{Spread}}_{\text{MWT}})\cdot$(10)

Among the three stand-alone approaches—MDE, TLE, and FME (Figs. 6a–d)—the MDE method shows the largest ensemble spread by far; and the maximum ensemble spread of MDE exceeding 0.20 m^2/3 s⁻¹ is comparable to the intensity of moderate turbulence (EDR ≥ 0.22 m^2/3 s⁻¹), indicating the large uncertainties in turbulence forecasts. The large dispersion of the MDE method for this selected case mainly results from the spread of CAT diagnostics, except over the Colorado Rockies where some MWT reports were recorded and the Alberta Rockies (cf. Figs. 7a,b). One possible reason for the small ensemble spread of the TLE and FME methods is the limits imposed by using a single model in ensemble weather forecasting (i.e., using the same model formulation, including the parameterization of subgrid-scale processes), which leads to restricted sampling of the forecast space (Slingo and Palmer 2011). Moreover, operational ensemble weather forecasting systems (e.g., GEFS) in general target medium-range forecasts (3–7 days in advance) (e.g., Zhou et al. 2017), and this may limit the ensemble spread of shorter-term weather forecasts that turbulence forecasts rely on. Another contributing factor for the smaller spread is that the TLE and FME methods use GTGMAX to compute probabilities [Eq. (5)], and GTGMAX is based on averages of multiple diagnostics [Eqs. (1) and (2)], which can smooth out outliers that individual diagnostics may produce. However, the impact of using the averages is minimal as can be seen by comparisons using a single diagnostic (not shown). For all approaches, large ensemble spreads are found over the areas of large deterministic EDR values (cf. Fig. 6h).

View Full Size

Ensemble spreads of (left) CAT and (right) MWT diagnostics remapped to EDR (m^2/3 s⁻¹) derived from 12-h probabilistic forecasts based on the MDE-19D approach: (a),(b) 1800 UTC 3 Dec 2019 at FL370 and (c),(d) 1800 UTC 11 Jan 2020 at FL340.

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

Increasing the number of diagnostics in the MDE method further increases the peak values of the ensemble spread, e.g., over Colorado and Montana, and over the Pacific Ocean (cf. Figs. 6a,b). Overall, the spatial distributions and magnitudes of the ensemble spread of the two combined methods—TMDE-5TL×19D and FMDE-21EM×19D (Figs. 6e,f)—are similar to those of MDE-19D alone (Fig. 6a) that use the same number of turbulence diagnostics, except that the areas of the small ensemble spread of approximately 0.04–0.06 m^2/3 s⁻¹ are more widely distributed when the combined methods are used. Increasing the number of diagnostics in the combined TMDE method further smooths the areas of small ensemble spread of 0.04–0.06 m^2/3 s⁻¹, while increasing the peak values of the ensemble spread (cf. Figs. 6e,g).

Ensemble spreads for the 11 January 2020 case are compared in Fig. 8. The 11 January 2020 case has a cluster of moderate and severe turbulence reports associated with MWT over the Colorado Rockies at altitudes around FL340. A number of moderate and severe turbulence reports were also recorded at lower altitudes over the Mississippi Valley associated with strong convection, as well as CAT over Illinois (Fig. 2b). Overall, the characteristic features of different probabilistic approaches that were found from the 3 December 2019 case hold, including the larger spreads of MDE, TMDE, and FMDE (cf. Figs. 6,8). Over the Colorado Rockies, the spread of MWT diagnostics has a relatively larger contribution to the dispersion of the MDE method, while the spread of CAT diagnostics is larger elsewhere (cf. Figs. 7c,d). The ensemble spreads of the TLE and FME methods over the mountainous region are even smaller in this January case than in the 3 December case. Compared to synoptic-scale jet streams that dominate the December case, mountain waves are less resolvable in the 13-km pseudo-GEFS NWP model. This difference in scales of the primary turbulence generation mechanisms between the two cases may contribute to the less dispersive NWP ensembles—i.e., TLE and FME—over the MWT region in the January case.

View Full Size

As in Fig. 6, but for 1800 UTC 11 Jan 2020 at FL340.

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

The results of 12-h probabilistic forecasts of light, moderate, and SOG turbulence at FL370 for the 3 December 2019 case are displayed in Figs. 9–11, together with the corresponding 12-h deterministic forecasts. For the prediction of light turbulence (Fig. 9), MDE-19D shows a relatively wide dispersion of intermediate probabilities ranging between 20% and 80% (Fig. 9a), indicative of a relatively large ensemble spread in comparison to the TLE and FME forecasts (Figs. 9c,d). By increasing the number of diagnostics used in the MDE approach, MDE-77D predicts light turbulence over larger areas at lower probabilities rarely exceeding 60% (Fig. 9b). Such large ensemble spread may result in a high probability of detection for low percentage thresholds (e.g., 10%) but at the same time the number of false alarms may increase if ensemble spread is too large. The light turbulence events over southern Arizona–New Mexico and Florida, which are captured by MDE-77D but missed by MDE-19D are good examples showing the characteristics of a large ensemble spread.

View Full Size

(a)–(g) 12-h probabilistic forecasts (%) of light turbulence (0.15 ≤ EDR < 0.22) at FL370 valid at 1800 UTC 3 Dec 2019, together with (h) 12-h deterministic EDR forecasts (m^2/3 s⁻¹). In situ EDR reports of light turbulence within ±1-h time window and ±1000-ft altitude are marked by green circles in (a)–(g); in (h), in situ EDR reports of light or greater turbulence within the same time and altitude windows are colored by EDR scales, with null-to-light reports in white dots.

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 9, but for moderate turbulence (0.22 ≤ EDR < 0.34) and in situ EDR reports of moderate turbulence marked by yellow circles in (a)–(g).

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 10, but for SOG turbulence (EDR ≥ 0.34) and in situ EDR reports of SOG turbulence marked by red circles in (a)–(g).

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

The TLE- and FME-based forecasts show probabilities exceeding 80% in most regions where light turbulence is predicted (Figs. 9c,d), indicative of a relatively small ensemble spread (i.e., large degree of agreement among ensemble members). Such small spread may reduce the number of false alarms for low probability thresholds but also increases the number of missed events, and generally its performance is highly dependent on the performance of the underlying NWP model. Also, when the TLE and FME methods capture a turbulence event, they tend to capture it at higher probabilities (i.e., with higher confidence). These characteristics of the TLE and FME-based forecasts are better seen in moderate and SOG turbulence forecasts (Figs. 10 and 11). TLE-5TL and FME-21EM predict moderate turbulence over Colorado at higher probabilities than other methods, while missing the moderate events along the Nevada–Utah border and predicting the SOG events over Colorado at probabilities lower than 10% (Figs. 10c,d and 11c,d).

Combining the MDE method with the other two—i.e., TMDE and FMDE—leads to more widely spread lower probabilities compared to TLE and FME as expected from the ensemble spread (Figs. 9e–g, 10e–g, and 11e–g). Similar to the MDE forecasts, the TMDE and FMDE-based forecasts capture the moderate events along the Nevada–Utah border and the SOG events over Colorado, which are missed in the TLE and FME-based forecasts. The differences between the combined approaches and the stand-alone MDE approach are not noticeable from this selected case. The probabilistic forecast results for the 11 January 2020 case are presented in Fig. 12, confirming the characteristics of different approaches: the severe turbulence events over Colorado that are predicted by the MDE approach at low probabilities (10%–20%) but entirely missed by the TLE and FME approaches, and the light and moderate turbulence events over Colorado that are predicted at higher probabilities (with higher confidence) by the TLE and FME approaches.

View Full Size

The 12-h probabilistic forecasts (%) of (a)–(d) light, (e)–(h) moderate, and (i)–(l) SOG turbulence at FL340 valid at 1800 UTC 11 Jan 2020. In situ EDR reports within ±1-h time window and ±1000-ft altitude are marked by circles: light (green), moderate (yellow), and SOG (red) turbulence reports.

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

b. Statistical evaluations

The BSS measures the mean square errors of probability forecasts relative to the errors of reference forecasts (deterministic forecasts in the case of this study). The deterministic (reference) forecasts refer to the GTGMAX fields derived from the control (unperturbed) NWP member of the pseudo-GEFS (e.g., Fig. 6h and used the 19 turbulence diagnostics listed in Table 1, as described in section 2b [Eqs. (1) and (2)]). With the BSS, higher is better with 1 being the best score, and positive/negative score values indicating better/worse performance than the reference forecasts:

$\text{BSS}=1-\text{BS}/{\text{BS}}_{\text{ref}},\hspace{0.17em}$(12a)

$\text{BS}={\displaystyle \sum }_{i=1}^{N_{\text{obs}}}{({\text{Prob}}_i-O_i)}^2/N_{\text{obs}},\hspace{0.17em}$(12b)

${\text{BS}}_{\text{ref}}={\displaystyle \sum }_{i=1}^{N_{\text{obs}}}{({\text{Dete}}_i-O_i)}^2/N_{\text{obs}}\cdot$(12c)

Here BS and BS_ref are the mean square errors of probabilistic and deterministic (reference) forecasts, respectively. The reference deterministic forecasts were computed using 19 turbulence diagnostics listed in Table 1 [Eqs. (1) and (2), section 2b]. The term O_i is the binary outcome of a turbulence event (O_i = 1 if an event is observed and 0 if it is not). Prob_i is the predicted probability and varies between 0 and 1. Dete_i is the binary outcome for predicting turbulence. Dete_i = 1 if a turbulence event is predicted—e.g., GTGMAX ≥ 0.15 for LOG turbulence events—and 0 if it does not. The BSS values derived from different probabilistic forecast approaches are compared in Table 3.

Table 3

The Brier skill score (BSS) of 12-h probabilistic forecasts of LOG and MOG turbulence events, derived from 345 cases over a 1-yr period from 21 May 2019 to 20 May 2020. The BSS is calculated using the 12-h deterministic forecasts as reference predictions. The best scores among the stand-alone methods and among all methods are highlighted in italics and in bold, respectively.

View Table

Among the three stand-alone approaches, the MDE method that uses all 77 diagnostics exhibits the highest BSS for light or greater (LOG) and moderate or greater (MOG) turbulence events. One thing to note is the improvement of the MDE-based probabilistic forecasts attained by increasing the number of diagnostics from 19 to 77. Increasing the number of diagnostics from 19 to 77 leads to improvements of statistical performance by incorporating a larger variety of turbulence generation mechanisms. The BSS is increased from 0.353 to 0.479 for LOG and from 0.157 to 0.359 for MOG. Increasing the number of diagnostics in the MDE framework does not require any additional GTG runs (i.e., N_Wx = 1), therefore this performance gain is achievable with only a small increase of computational cost. A single MDE-77D forecast run time is only 1.2–1.3 times longer than a single MDE-19 forecast, compared to the TLE or FME approaches for which computational times increase linearly with the number of ensembles used (Table 2). Among the three stand-alone approaches, MDE-19D shows the second highest BSS for LOG turbulence, and TLE-5TL for MOG turbulence. Between the two NWP ensemble approaches, the TLE approach outperforms the FME approach. This is at least in part due to the slightly larger ensemble spread of TLE than FME where turbulence events are reported (Figs. 6c,d and 8c,d).

Combining MDE with either TLE or FME also improves the BSS; the skill score is improved from 0.353 to 0.437 (TMDE-5TL×19D) or 0.409 (FMDE-21EM×19D) for LOG turbulence, and from 0.157 to 0.343 (TMDE-5TL×19D) or 0.273 (FMDE-21EM×19D) for MOG turbulence. The performance gain by combining the MDE to TLE is comparable to the performance gain by increasing the number of diagnostics in the MDE approach, but at the expense of additional computational costs proportional to the number of weather forecasts (Table 2). Increasing the number of diagnostics from 19 to 77 in the combined TMDE framework further improves the skill score; in terms of the BSS, the largest improvement over the reference (deterministic) forecast is obtained when using the TMDE approach with 77 diagnostics.

The ROC curve and the area under the ROC curve (AUC) score measure the ability of forecasts to distinguish events from nonevents, based on the 2 × 2 contingency table (Table 4). The ROC curve is created by connecting the pairs of the probability of forecasts to predict observed events (probability of detection (POD); “observations = yes” and “forecasts = yes”) against the rate of forecasts to produce false alarms [false alarm rate (FAR); “observations = no” and “forecasts = yes”] at varying probability thresholds from 0% to 100%. For each probability threshold, POD and FAR are derived from a 2 × 2 contingency table (Table 4):

$\text{POD}=\hspace{0.17em}\frac{\text{hit}}{\text{hit}+\text{miss}},\hspace{0.17em}$(13a)

$\text{FAR}=\hspace{0.17em}\frac{\text{false}\hspace{0.17em}\text{alarm}}{\text{false}\hspace{0.17em}\text{alarm}+\text{correct}\hspace{0.17em}\text{rejection}}.\hspace{0.17em}$(13b)

The ROC curves of LOG and MOG turbulence events are shown in Fig. 13. The MDE, TLE, and FME methods show sharp bends in the ROC curves, particularly in the case of MOG turbulence. Where the curves bend corresponds to the POD-FAR pair of the lowest nonzero percentage threshold of each method: e.g., 9.09% in the case of MDE-19D (1/N_CAT × 100), 20% and 4.76% in the case of TLE-5TL and FME-21EM (1/N_Wx × 100), respectively. These sharp bends are due to a scarcity of higher-intensity turbulence reports. The TLE and FME approaches exhibit sharper bends that are closer to the origin point than other approaches. These two approaches predict the lowest nonzero probabilities less often than other approaches (Fig. 14) and this is related to their small ensemble spread, resulting in more frequent predictions of either 0% or 100% probabilities and less frequent predictions of intermediate probabilities, as shown in the case studies. Therefore, the number of high-intensity turbulence reports at the lowest nonzero probability threshold is even smaller in these two approaches, leading to the sharper bends in the ROC curves and lower AUC scores than in other approaches (Table 5). This issue of ROC curves for rare events was also noted by Casati et al. (2008) (i.e., “a tendency for the points on the ROC to cluster toward the lower left corner of graph”) and where the ROC curve bends (or “how much of the curve is missing”) depends on the lowest nonzero probability (Ben Bouallègue and Richardson 2021), consistent with our analysis results. The ROC curves with the bends inevitably lead to lower AUC scores than those without the bends. Therefore, comparing the slopes of the ROC curves for low FAR provides more useful comparison than comparing the AUC scores (Gill 2016; Storer et al. 2019).

Table 4

The 2 × 2 contingency table.

View Table

View Full Size

ROC curves of (a) LOG and (b) MOG turbulence forecasts derived from MDE-19D (solid blue), MDE-77D (dotted blue), TLE-5TL (solid red), FME-21EM (solid green), TMDE-5TL×19D (solid purple), TMDE-5TL×77D (dotted purple), and FMDE-21EM×19D (solid cyan), together with a random classifier (gray).

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Histograms of predicted probabilities of (a) LOG and (b) MOG turbulence that correspond to in situ EDR reports and PIREPs converted to EDR (1 280 246 samples). The number of bins equals to N_Wx in the case of TLE and FME, and the maximum between N_CAT and N_Wx in the case of MDE, TMDE, and FMDE. Frequency of turbulence (%) shown at the bottom of each panel is the ratio of forecasts that predict turbulence (probability > 0%) to the total number of EDR observations.

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

Table 5

As in Table 3, but for the AUC score.

View Table

The slopes of the ROC curves presented in Fig. 13 are estimated by finding the ratio of the change of POD to the change of FAR centered at FAR ∼0.2 (0.05) where the TLE and FME ROC curves of LOG (MOG) bend. A larger slope indicates a better prediction skill, i.e., a larger increase of POD for the same increase of FAR for practical regions of interest (i.e., low FAR and high POD). Similar reasoning using a partial ROC was used by Gill (2016). For LOG turbulence (MOG turbulence), FME-21EM shows the steepest slopes for FAR < 0.2 (for FAR < 0.05), followed by TLE-5TL and TMDE-5TL×77D. MDE-19D shows the least steep slope. Increasing the number of diagnostics improves the MDE-based probabilistic forecasts (i.e., the slopes of the ROC curves are steeper in MDE-77D), and combining MDE with either TLE or FME also leads to steeper slopes, therefore better prediction skills. This is consistent with the findings based on the BSS scores.

Histograms of predicted probabilities that correspond to in situ EDR reports and PIREPs and reliability diagrams derived from the histograms are presented in Figs. 14 and 15, respectively. As was seen in the case studies, the MDE method predicts intermediate probabilities more frequently than the TLE and FME methods and 0% probability less often. It rarely predicts 100% probability, as expected from the large disagreement among the diagnostics (Figs. 5a–c). The TLE and FME methods, on the other hand, predict both 0% and 100% probabilities more frequently than the other methods, indicative of a small ensemble spread. Combining MDE with either TLE or FME decreases the frequency of 0% probability while increasing the frequency of low probabilities between 0% and 20%. The relative frequency of forecasted events (i.e., the sum of frequencies for probability > 0% in the histograms) is much higher than the relative frequency of observed turbulence events with respect to the total number of observations (2.43% and 0.63% for LOG and MOG turbulence events, respectively). This emphasizes the overall overprediction of events regardless of method, if not calibrated, as presented in Fig. 15. This overprediction of turbulence events by probabilistic forecasts was also found in previous studies as well (e.g., Gill and Buchanan 2014; Storer et al. 2019, 2020; Lee et al. 2020). This is due in part to a suspected bias in the observations toward nonturbulent events created when pilots try to avoid regions of known turbulence (Sharman et al. 2014; Lee et al. 2020). Lee (2021) demonstrated that the reliability curves can be improved by using subsets of in situ EDR reports that were randomly sampled to have the ratios of turbulence events to the total in situ reports similar to those of the forecasts.

View Full Size

Reliability diagrams of (a) LOG and (b) MOG turbulence forecasts derived from MDE-19D (solid blue), MDE-77D (dotted blue), TLE-5TL (solid red), FME-21EM (solid green), TMDE-5TL×19D (solid purple), TMDE-5TL×77D (dotted purple), and FMDE-21EM×19D (solid cyan); reliability diagrams with forecast probability divided by a calibration constant (6.0 for LOG and 12.0 for MOG) are shown in the insets. The diagonal (black solid) and horizontal (black dashed) lines indicate perfect reliability and no resolution (observed climatology), respectively.

Citation: Weather and Forecasting 38, 1; 10.1175/WAF-D-22-0086.1

• Download Figure
• Download figure as PowerPoint slide

To address the overprediction of probabilistic forecast, which is obvious in the reliability curves (Fig. 15), recalibration can be introduced following Storer et al. (2019, 2020). Multiplying the forecast probabilities of LOG and MOG turbulence events by calibration constants of 1/6 and 1/12, respectively, results in the reliability diagrams plotted in the insets of Fig. 15. The calibration constants depend on turbulence thresholds (i.e., LOG or MOG turbulence) and underlying NWP models. For example, Storer et al. (2020) used 1/50 for UKMO MOGREPS and 1/40 for ECWMF EPS. For LOG turbulence (Fig. 15a), the MDE, TMDE, and FMDE show similar slopes that are in parallel with the perfect reliability line (solid diagonal line) for calibrated forecast probability up to ∼3%. For calibrated probabilities above ∼4%, the TMDE-5TL×77D forecasts follow the perfect reliability line but with only slight overprediction. The combined forecasts that use selected diagnostics (TMDE-5TL × 19D and FMDE-5TL×19D) and the MDE forecasts further deviate from the perfect reliability line (i.e., overforecast). The TLE and FME forecasts show relatively flat lines, and it is hard to improve the reliability of these two methods by modulating the calibration constant; decreasing the calibration constant exacerbates the underprediction at low probabilities, while increasing it results in overprediction at all probabilities.

For MOG turbulence (Fig. 15b), MDE-77D and TMDE (TMDE-5TL×19D and TMDE-5TL×77D) show the slope closer to the perfect reliability line than other forecasts, but underforecast for calibrated probabilities above ∼6%–7%. The TLE and FME forecasts show relatively flat lines, as for LOG turbulence. The improvements of statistical performance achieved by combining the NWP-based TLE or FME methods with the MDE method is mainly because of their different characteristics of ensemble spreads: i.e., the larger spread of the MDE approach that leads to a wider dispersion of intermediate probabilities and the smaller spread of TLE and FME that results in probabilities at extremes.