1. Introduction
The prediction of the genesis of tropical cyclones (TCs) has become increasingly accurate in recent years, due to advancements in numerical modeling and data assimilation (Halperin et al. 2013). In 2013, the National Hurricane Center (NHC) introduced a new operational product issuing the probability of tropical cyclogenesis within 5 days, underlying the increased confidence in genesis prediction (genesis probabilities out to 5 days are provided every 6 h by the NHC online at http://www.nhc.noaa.gov/gtwo_atl.shtml). Additionally, several recent studies have assessed the ability of ensemble forecasts to accurately predict genesis in medium-range forecasts (e.g., Snyder et al. 2010, 2011). Tsai et al. (2013) even find that the European Centre for Medium-Range Weather Forecasts (ECMWF) ensemble demonstrates some skill in predicting genesis out to several weeks. Despite a recent uptick in studies that examine the predictability of whether or not genesis will occur, few if any examine the predictability of the pregenesis environment.
The predictive skill and the intrinsic predictability of tropical cyclogenesis (and intensity change) are largely influenced by a range of tropospheric conditions, both associated with the incipient vortex and the environmental flow with which it is interacting. To the best of our knowledge, no published study has explored the predictability of such variables directly relevant to cyclogenesis in the tropical Atlantic. One of the goals of the National Science Foundation’s Pre-Depression Investigation of Cloud-systems in the Tropics (PREDICT; Montgomery et al. 2012) field campaign and research initiative has been to identify the extent to which genesis is predictable and the factors that limit its predictability. In this paper, we use a variety of metrics as a proxy for quantifying predictability on a basinwide scale, using the ECMWF ensemble prediction system for the 2010 Atlantic hurricane season. The predictability associated directly with individual tropical waves and genesis events is the subject of a companion paper. While it is premature in these papers to offer rigorous conclusions about predictability with only a single hurricane season, we hope to offer a pathway toward understanding predictive skill and predictability through a first examination of a variety of ensemble-based metrics.
To attempt to quantify predictability, several metrics have been proposed in the literature, though all possess limitations and are dependent on the quality of the predictive model. One basic method was introduced in the famous study by Lorenz (1982), in which globally averaged errors of 500-hPa geopotential height forecasts from the ECMWF global model were computed to estimate lower and upper bounds of predictability. Several conclusions from this study and others by Lorenz (1963, 1965, 1969) are well-held tenets of predictability. Error growth was found to be nonlinear in a dynamical system, generally growing most rapidly during the first 24 h of model integration due to errors at small scales. Thereafter, error growth begins to slow as errors of increasingly larger scales begin to saturate. The total error finally begins to asymptote toward a constant value after errors at all scales have saturated. At this point, which may be several weeks, any individual forecast has become completely indistinguishable from a random state of the system/atmosphere, and predictability is said to have been lost. A related major conclusion that is of particular relevance to this paper is that the predictability of the flow is dependent on both the scales of motion and the flow regime (Lorenz 1965, 1969).
In practice, the growth and saturation of errors with respect to different phenomena with complex physics is not well established. For example, it may be feasible for initially large error structures to transfer to smaller scales with time, which would essentially be a reversal of the classical Lorenz conceptual models. The initial error structures may be a consequence of the data assimilation technique and the observational data assimilated, and the ensemble initialization and model perturbation techniques.
Even when errors have not yet saturated at all scales, a prediction is not useful to forecasters unless its errors are lower than climatology, which is typically employed as the default “no skill” forecast in the absence of any additional information. Several studies have attempted to quantify the net predictability of a system by taking the ratio between the root-mean-square (RMS) error of the model prediction and the climatology (Shukla 1981, 1985; Hayashi 1986; Murphy 1988; Griffies and Bryan 1997). Many of these studies assumed the error distribution to be Gaussian, although Anderson and Stern (1996) demonstrate that this is not always the case. Another caveat in estimating the predictability by these means is that results are dependent upon the choice of variable(s) (Palmer 1996; Palmer et al. 1998). While many classical studies quantify errors associated with geopotential height at 500 hPa, the use of height at a different pressure level or a different variable such as wind may yield a different interpretation.
The above two measures can be applied to a single deterministic forecast with a point verification. However, given that the uncertainty of a forecast is an essential component of modern-day prediction and is also a direct estimate of the predictability of a given flow, it is imperative to devise and evaluate relevant measures. Predicted uncertainty in ensemble forecasts is commonly computed via the ensemble forecast variance and its skill is evaluated against the variance or standard deviation of forecast errors (e.g., Wang and Bishop 2003; Grimit and Mass 2007). In a recent study by McMurdie and Ancell (2014), predictability is defined as the ensemble spread of surface pressure at the final forecast time, where large spread means low predictability. However, the aforementioned study focuses on midlatitude rather than tropical predictability.
Another metric that has been used more commonly in longer-range predictions is predictive power (PP; Schneider and Griffies 1999). The PP quantifies the forecast uncertainty by comparing the probability density function (PDF) based on an ensemble forecast with a climatological PDF, which represents a distribution of all historical outcomes. Predictability is said to be lost once the forecast PDF has become broader than the climatological PDF. Related metrics such as relative entropy (Kleeman 2002) and mutual information (Kleeman 2007) have also been devised. Several studies have utilized PP and similar methods to assess predictability (e.g., Stephenson and Doblas-Reyes 2000; Schneider and Held 2001; Waliser et al. 2003; DelSole 2004; Abramov et al. 2005; Giannakis and Majda 2012). However, in all the cited studies, computational constraints restricted the resolution at which PP and related metrics could be calculated. Also, the vast majority of these studies focused on more slowly varying systems than those considered in this study, such as the general circulation of the ocean or climate systems.
In this paper, we extend the error, error variance, and predictive power metrics into the domain of medium-range atmospheric predictions over the tropical Atlantic Ocean, using an operational ensemble of relatively high resolution (25 km) and relatively sophisticated physics compared with earlier studies. The primary purpose is to establish a basis for understanding predictability, via quantitative measures of a spectrum of kinematic variables relevant to tropical cyclogenesis. In section 2, the variables and metrics are formally introduced. In sections 3 and 4 the results using traditional metrics of RMS error and error growth and predictive power are presented, respectively. Conclusions are provided in section 5.
2. Methodology
a. ECMWF ensemble and climatology
Analyses and 10-day forecasts from the operational 50-member ECMWF Ensemble Prediction System at T639 L62 resolution, interpolated onto a horizontal grid with 0.25° spacing are used in this study. The period considered is from 1 June to 30 November 2010. The ensemble data, produced twice daily, were provided via The Observing System Research and Predictability Experiment (THORPEX) Interactive Grand Global Ensemble (TIGGE) database (Bougeault et al. 2010; http://tigge.ecmwf.int/). The ensemble is initialized through a combination of (i) initial-time dry global singular vectors (SVs) maximized over the Northern Hemisphere, (ii) an “ensemble of data assimilations” to create initial spread in the ensemble, and (iii) initial-time tropical SVs with a dry total energy norm and additional linearized physics relevant to tropical processes, optimized over the Caribbean area (0°–25°N, 100°–60°W). The SVs are not optimized on regions centered on tropical waves and disturbances prior to their existence as TCs. Model error perturbations are included via the stochastic kinetic-energy backscatter scheme (SKEB; Berner et al. 2009) and the stochastic perturbed parameterization tendencies scheme (SPPT; Palmer et al. 2009).
Relative vorticity: 850 hPa;
Vertical shear of horizontal wind: 850–200 and 850–500 hPa;
Divergence: 200 hPa;
Relative humidity: 700 hPa;
Velocity potential: 200 hPa;
Circulation: average 850–700-hPa relative vorticity within 200 km of each grid point; and
Thickness anomaly, defined by
The circulation and thickness anomaly were chosen since they were found to be relatively unambiguous in objectively identifying a tropical disturbance or cyclone in numerical model fields (Majumdar and Torn 2014, manuscript submitted to Wea. Forecasting). First, the circulation was found to be easier to unambiguously track temporally than relative vorticity, given its distinct local maxima in easterly waves and gyres compared with multiple filaments with adjacent local maxima of vorticity. Second, the 850–200-hPa thickness anomaly ΔZ yields a distinct and unambiguous signal of a local warm anomaly, which may correspond to a warm core building in a developing tropical cyclone. The exceedance of circulation and thickness anomaly by given threshold values in ECMWF analyses (investigated in Majumdar and Torn 2014, manuscript submitted to Wea. Forecasting) was found to be consistent with whether the tropical disturbance had attained tropical cyclone status (as declared by NHC), with relatively few “misses” (weak circulation and thickness anomaly, TC declared) or “false alarms” (strong circulation and thickness anomaly, no TC).
The domain used throughout this study comprises the Atlantic Ocean from 0°–25°N and 10°–100°W (Fig. 1a). Attention in this study is emphasized on tropical weather, incorporating tropical waves, dry air masses originating from Africa, and gyres and inverted troughs. Less emphasis is given to conditions that may lead to cyclogenesis of disturbances of baroclinic origin or of a subtropical nature. As will be discussed later, the latitudinal extent is restricted further to prevent baroclinic processes from contributing substantially to the variables under investigation.
(a) Geographic domain used in this study (blue) and land mask used to exclude Central America and part of the east Pacific (red); monthly-mean 850–200-hPa wind shear (m s−1) for 2010 for the months of (b) June, (c) July, (d) August, (e) September, (f) October, and (g) November.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
It should be noted that all calculations include a land mask to ensure that results are relevant to only the tropical Atlantic without contamination from South America or the east Pacific. The geographic area excluded by the land mask is depicted in Fig. 1a (anything west of the red line). However, it should be noted that results in sections 3 and 4 were not found to be particularly sensitive to whether or not the land mask was included.
Finally, the climatology is generated using 33 years of Interim ECMWF Re-Analysis (ERA-Interim) data from 1979 to 2011. ERA-Interim is a reanalysis dataset that uses the ECMWF data assimilation system and forecast model from 2006, but with T255 spectral resolution and 60 vertical levels (Dee et al. 2011). The climatological data, originally at the equivalent of 0.703° × 0.702° resolution on a Gaussian grid, are interpolated to a 0.25° fixed grid using spherical harmonics.
b. Metrics to quantify error growth and predictability
1) RMS error and error variance
In this paper, the RMS error 
Several different approaches have been taken to evaluate the reliability of ensemble predictions via their statistical consistency versus actual errors (reviewed in Grimit 2004). In other words, the ensemble needs to be able to discriminate between different error distributions, with forecasts of higher error corresponding to larger error spread in the ensemble predictions. A probabilistic approach that has been adopted over the past decade is to determine whether a relationship exists between the predicted error variance [Eq. (3), or the standard deviation which is its square root] and the corresponding variance (or standard deviation) of actual errors in the ensemble mean (e.g., Wang and Bishop 2003). This is accomplished via first plotting the distribution of forecast errors of the ensemble mean versus their corresponding predictions of the variance (or standard deviation) of the ensemble mean, for each forecast case. By averaging the data points in equally sized bins of increasing predicted forecast error variance, the variance (or standard deviation) of the actual errors is computed for each bin. Ideally, a linear, increasing relationship of slope 1 between the predicted and actual error variance (or standard deviation) is found. In this paper, we elect to plot the error of the ensemble mean, retaining the sign of the errors, versus the ensemble standard deviation, following the bottom panel of Fig. 5 of Grimit and Mass (2007). In this way, the distribution of forecast errors, including any biases and departures from Gaussianity are illustrated, while retaining the ability to formally evaluate the relationship between the predicted and actual error range.
2) Predictive power
To compute PP with so many data points (due to a combination of high resolution and large domain size), it is necessary to truncate the computations to find the determinants of 
3. Error growth
a. Monthly evolution of variables
To give perspective to the results, we first document the evolution of the relevant variables through the 2010 Atlantic hurricane season, based on ECMWF analysis fields. The evolution of the environment during 2010 is overall consistent with the climatological progression, although conditions were slightly more favorable during peak season than the climatology, likely contributing to anomalously high TC activity that year.
In June, much of the central Atlantic is dominated by relatively high values of 850–200-hPa vertical wind shear (exceeding 15 m s−1) (Fig. 1b). The shear decreases rapidly through July and remains favorable in the 7.5–12.5 m s−1 range for the majority of August and September (Figs. 1c–e). By October, shear again increases rapidly, especially north of 15°N (Fig. 1f), and high shear values dominate much of the tropical Atlantic in November (Fig. 1g). The 850–500-hPa vertical wind shear exhibits a similar spatial distribution and evolution (not shown). Comparison to the 1979–2011 climatology suggests that vertical wind shear was anomalously low in the central Caribbean during the 2010 hurricane season. Despite generally more favorable conditions, only one genesis event, Matthew, occurred in this region. However, this area is generally not associated with many genesis events, as persistent low-level mass divergence in this region induces a subsidence regime that weakens convection (Shieh and Colucci 2010).
An examination of 700-hPa relative humidity reveals that the northeast quadrant of the tropical Atlantic is generally driest throughout the season (Fig. 2). This minimum in moisture is particularly pronounced during June and July (Figs. 2a,b) when the Saharan air layer (SAL) is most prominent (Dunion 2011). The basin, and in particular the main development region (MDR) and the Caribbean, possesses the greatest average 700-hPa RH during August–October (Figs. 2c–e), with noticeable drying in the Gulf of Mexico and western Atlantic during October–November as midlatitude intrusions of dry air become increasingly frequent (Figs. 2e,f).
Monthly-mean 700-hPa relative humidity (%) for 2010 for the months of (a) June, (b) July, (c) August, (d) September, (e) October, and (f) November.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
Last, the evolution of 850–700-hPa circulation is examined. Negative values of circulation, corresponding to anticyclonic vorticity, dominate the northern half of the domain during June–August (Figs. 3a–c). Positive values of circulation progressively build with time during the first half of hurricane season, particularly within 10°–20°N, 15°–50°W (Figs. 3c,d), before diminishing rapidly in October–November (Figs. 3e,f). The African easterly wave train and tropical cyclones contribute substantially to the mean circulation field. Monthly-mean circulation values at or above 1.5 × 10−5 s−1 in August and September 2010 exceed climatological values as a result of easterly waves that tended to be larger and stronger than average (P. Klotzbach 2014, personal communication). A maximum in cyclonic circulation in the western Caribbean is also evident throughout the 2010 season. The genesis of Tropical Cyclones Alex, Karl, Matthew, Nicole, Paula, and Richard all occurred in this region.
Monthly-mean 850–700-hPa circulation (s−1) for 2010 for the months of (a) June, (b) July, (c) August, (d) September, (e) October, and (f) November.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
b. Evolution of errors
The next stage is to document errors in ensemble-mean forecasts between 0 and 10 days. Between days 2 and 4, errors in circulation tend to grow most quickly in the eastern MDR along the preferred path of easterly waves (Figs. 4a,b). Circulation errors throughout the rest of the basin begin to “catch up” by day 8, with errors along the path of recurving tropical cyclones becoming pronounced (Fig. 4c). Meanwhile, errors in both 700-hPa relative humidity (Figs. 4d–f) and deep-layer shear (not shown) are generally maximized north of 15°N. Differences in the phase, strength, and evolution of midlatitude dry air intrusions, upper-level troughs, and SAL outbreaks are likely culprits. Such events introduce errors in the forecast that rarely occur equatorward of 15°N, where the shear is more persistently low and moisture is more persistently high.
Spatial distribution of (a),(d) 48-h; (b),(e) 96-h; and (c),(f) 192-h forecast errors for (a)–(c) circulation (s−1) and (d)–(f) relative humidity (%) averaged during June–November 2010. Forecast errors are for the control run of the ECMWF ensemble.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
While there are geographically preferred regions for slightly faster error growth, the errors are growing everywhere with increasing lead time. By computing the mean of the forecast errors from June to November at every grid point in our Atlantic domain, we compare error growth characteristics against the hallmark study of error growth in the atmosphere (Lorenz 1982). Our results in Fig. 5 appear qualitatively similar to those of Fig. 1 of Lorenz (1982) who examined globally averaged 500-hPa geopotential height (Fig. 5a), in contrast to our regional averages of circulation, relative humidity, and vertical wind shear (Figs. 5b–d). In each case, the most rapid error growth occurs during the first 24 h of model integration due to the rapid growth of errors on the small scales, then declines as these small-scale errors saturate and/or energy is transferred to larger scales. In both the case of Lorenz (1982) and our study, the error growth rate becomes quasi-linear from around day 2. The error doubling time from this point is about 4 days in Lorenz (1982), whereas using our metrics the errors may saturate beyond 10 days without having doubled. In other words, errors are growing considerably more slowly in the tropics as expected. Beyond 8 days, as the errors at all spatial scales approach a maximum, the error curve slowly begins to saturate and asymptote toward a constant error. Note that there is significant variability in the error growth rate from one forecast to the next, indicated by the large 95% confidence interval accounting for the error growth curve for all forecasts during June–November 2010. If forecast errors were calculated beyond 10 days, eventually a maximum error state would be reached. Beyond this timeframe, a particular deterministic forecast becomes no more skillful than a forecast made with arbitrary initial conditions, and all predictability has been lost (Lorenz 1965).
Error growth for (a) global forecasts of 500-hPa heights from Lorenz (1982), and from 2010 using the ECMWF ensemble: (b) 850–700-hPa circulation, (c) 700-hPa relative humidity, and (d) 850–200-hPa wind shear. Forecast errors are for the control run of the ECMWF ensemble. A linear best-fit line (dashed) is extrapolated from the day 4–6 error growth in each curve.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
A linear best-fit trend line is extrapolated from the day 4 to 6 error growth in each curve. By 240 h, the error growth rate has slowed below the linear rate for all our metrics as large-scale errors saturate. It has dropped the most substantially for circulation (Fig. 5b), while it remains close to linear for shear (Fig. 5d). One may interpret this result as suggesting that basinwide shear on average possesses a more extended range of predictability than circulation.
The error growth rates also vary from month to month (Fig. 5). For circulation, the greatest errors occur during August and September, where higher uncertainty associated with an abundance of mature TCs contributes to larger errors. It is hypothesized that circulation errors for 2010 are overall greater than average due to heightened wave activity. For shear and RH, the greatest errors occur during October and November, where uncertainty that is likely associated with fronts and midlatitude systems affecting the tropics results in greater forecast uncertainty. Also note that since the much larger 95% confidence interval is for the full spread of all days within the 6-month period and not just the monthly averages, the variability from one forecast to the next is even greater than what the variability of the monthly means suggests. Therefore, it appears that total error and error growth rates are highly regime dependent.
The philosophy introduced by Lorenz (1982) abides by one particular definition of predictability, in which one may interpret a forecast as being useful if errors have not yet saturated. However, even if errors are still growing, the forecast may have lost skill compared against climatology, thereby eliminating its practical utility. To quantify this, we compare ECMWF control and ensemble-mean forecasts for August–October 2010 against “forecasts” during that same time frame using the 1979–2011 climatology as the forecast (Fig. 6). It is found that the point at which the ECMWF control run forecast errors exceed errors of the climatological forecast varies greatly depending upon choice of variable. By this criterion, forecast skill as a proxy for predictability is lost in the medium range, or 120 h, for circulation (Fig. 6a). For some variables, predictability is lost much more rapidly, with forecast error of the control run exceeding that of the climatological forecast within 48 h for 850-hPa vorticity (Fig. 6b) or only 24 h for 200-hPa divergence (Fig. 6c). Meanwhile, other variables are associated with much greater estimated predictability, with forecast error not exceeding climatological error until >240 h for 850–200-hPa thickness anomaly (not shown), 240 h for 200-hPa velocity potential (Fig. 6d), 192 h for 850–200-hPa wind shear (Fig. 6e), 180 h for 700-hPa relative humidity (Fig. 6f), and 108 h for 850–500-hPa wind shear (not shown). The much lower errors for circulation versus vorticity forecasts in comparison to climatology are likely a result of the fact that it is much easier to forecast an area-average quantity than it is to correctly predict a value at every possible grid point. The lower predictability associated with 850–500-hPa shear than 850–200-hPa shear is likely due to the greater uncertainty in the overall steering pattern at 500 than at 200 hPa (along with, to some degree, the fact that forecasting climatology for midlevel shear in 2010 actually made a good forecast: only a 2.6 m s−1 error on average). As expected, forecasts of velocity potential are associated with much lower error than divergence forecasts. The order of magnitude difference between the predictability “limit” (by this definition) of divergence and velocity potential is a direct consequence of the variability of their respective spatial and temporal scales.
Comparison of error growth rates for (a) 850–700-hPa circulation, (b) 850-hPa vorticity, (c) 200-hPa divergence, (d) 200-hPa velocity potential, (e) 850–200-hPa wind shear, and (f) 700-hPa relative humidity. Forecasts are made using the ECMWF control forecast (blue), the ECMWF ensemble mean (red), and a climatological forecast (green).
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
The ensemble mean forecast performs notably better on average than the control forecast. For example, using the ensemble mean forecast rather than the control forecast greatly extends the predictive skill for circulation from 108 to 240 h (Fig. 6a), or from 180 to >240 h for 700-hPa relative humidity (Fig. 6f). The lead time at which a skillful forecast can be made versus climatology only increases by a small amount for 850–200-hPa wind shear, although the control forecast is already very skillful to begin with (Fig. 6e). These results suggest that the ensemble mean forecast could be useful in predicting environments favorable for tropical cyclones to develop with greater than one week lead time.
While comparing errors in the ensemble mean forecast to climatological errors addresses the mean error and error growth rate, it does not address the fact that there is significant variability from the climatological mean state. The ensemble forecast, with each member constructed to represent an equal-likelihood event, is also associated with an increasing variance with increasing lead time. Therefore, it is perhaps more appropriate to be comparing the probability distribution, or more specifically, the PDF of the ensemble forecast to the PDF of all possible states comprising the climatology. We address these issues by evaluating forecast error variance and predictive power.
c. Forecast error variance
Compiling all forecasts of 850–200-hPa wind shear during August–October 2010 in 10° × 10° boxes and plotting the absolute error of the ensemble mean versus the square root of the predicted forecast error variance (the standard deviation), a modest but unconvincing positive relationship is observed (Fig. 7a). Low values of predicted error standard deviation should yield a low error of the ensemble mean. High values of predicted error standard deviation can yield low and high errors of the ensemble mean, though the average error of the ensemble mean over a large sample should be larger than when the predicted error standard deviation is low. Not surprisingly, the distribution changes as a function of forecast time, with generally lower (higher) predicted error standard deviations and errors of the ensemble mean at shorter (longer) forecast times. This relationship is mostly monotonic increasing and possesses a slope near one for 0–72-h (Fig. 7b), 84–156-h (Fig. 7c), and 168–240-h forecasts (Fig. 7d). Additionally, all of the predicted forecast error standard deviations lie above the line of unit slope, indicating that the variance of actual outcomes is greater than the variance of the forecasts. In other words, the ensemble is underdispersive given typical forecast errors.
Evaluation of ensemble variance prediction for 850–200-hPa shear forecasts. (a) Standard deviation of ensemble forecasts vs the absolute error of the ensemble mean forecast as a function of forecast lead-time, in days (colored). Ensemble standard deviation vs error of the ensemble mean forecast (purple dots), ensemble-mean forecast error standard deviations (black circles), and 1-to-1 line (dashed) for (b) 0–72 h, (c) 84–156 h, and (d) 168–240 h.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
A similar near-monotonic increasing relationship is found for all other chosen variables, such as 850–700-hPa circulation (Fig. 8), although the slopes for the different times vary. The slope is steeper than one for 0–72-h forecasts (excluding the final bin containing the largest forecast standard deviations) and flattens to a slope less than one as the forecast time increases. Averaging within 10° × 10° boxes tends to filter out the small scales in the variance predictions as well as mean error, such that the results for vorticity and circulation are nearly identical. Despite overall positive results, the ensemble is found to be underdispersive for all variables.
As in Fig. 7, but for 850–700-hPa circulation.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
4. Predictive power
The PP is developed such that it is invariant under linear coordinate transformations and applies to multivariate predictions irrespective of whether or not error distributions are Gaussian (Schneider and Griffies 1999). However, for the form used in this study, along with confidence intervals and significance levels, it is assumed that error PDFs are approximately Gaussian. Climatological error distributions, 
Probability density functions for (a),(b) climatological differences from the mean or “climatological errors” during 1979–2011 and (c),(d) 120-h forecast error distributions during June–November 2010, with Gaussian approximation based upon variance of sample (solid line). Probability density functions are for (a),(c) 850–700-hPa circulation and (b),(d) 700-hPa relative humidity.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
From Fig. 9, it is apparent that the overall variance or width of the climatological distribution is greater than that of the 120-h forecast error distribution. However, if error PDFs at multiple lead times were plotted, they would gradually widen with time from t = 0 to t = 240 h, eventually reaching or exceeding the width of the climatological PDF. There are also several cases in which the circulation error significantly exceeds the climatological mean, corresponding to a peak in the distribution along the tails of the PDFs in Figs. 9a,c. The rightmost peak along the positive tail in Fig. 9a corresponds to times when tropical cyclones pass through the domain, as the circulation becomes much greater than climatology. The positive (negative) peak at the end of the PDF tail in Fig. 9c corresponds to when a TC forms (fails to form) inside the domain when the ensemble does not predict it (predicts it), or when a TC inside the specified domain is stronger (weaker) than predicted. Since the width of the plotted Gaussian fit curve is based upon the variance of the distribution, and the variance is somewhat skewed by high and low outliers, much of the PDF in Fig. 9c lies “inside” the bell curve.
To estimate the predictability of 850–700-hPa circulation versus climatology using the ECMWF ensemble, PP is computed from t = 0 to t = 240 h in 12-h increments twice per day during August–October 2010 (Fig. 10a). Just as Schneider and Griffies (1999) run a Monte Carlo simulation to produce a confidence interval for the predictive power given a small number of actual forecasts, each of these independent forecasts from 2010 is treated as a sample of the true PP. The resulting mean PP crosses the 5% significance level, or the point at which the forecast error distribution and the climatological error distribution are no longer statistically distinguishable datasets at 180 h (Fig. 10b). This implies a predictability limit for circulation at the previous forecast time of 168 h. This is shorter than the predictability for circulation found when using the saturation of error growth to determine predictability, but longer than the limit of predictability found when comparing the RMS forecast error to the RMS climatological error. However, there is significant variability within the 95% confidence interval for PP, and the true predictability for circulation could range between 84 and >240 h. This broad range can potentially be reduced in future work if additional years are included.
Diagnosing the predictability of 850–700-hPa circulation and 850-hPa vorticity using predictive power. (a) Predictive power of circulation for each 0000 and 1200 UTC forecast from August (blue), September (magenta), and October (red) 2010. (b) As in (a), but with the 95% confidence interval and 5% significance level included and individual forecasts removed. (c) The eigenvalue spectrum of forecast error covariance for 10-day through 0-day circulation forecasts of circulation (colored) valid at 0000 UTC 15 Sep 2010, compared to the eigenvalue spectrum of the climatology (black). (d) As in (b) but for vorticity. In (a), (b), and (d), the mean for all forecasts is indicated by the solid black line.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
The PP calculation can be understood further by examining the eigenvalue spectrum (Fig. 10c). The PP becomes negative once the product of the first 50 eigenvalues (since we only have 51 ensemble members) of the forecast error exceeds the product of the eigenvalues of climatological error. This corresponds to a broadening of the forecast PDF beyond the width of the climatological PDF. All eigenvalues grow as errors grow with time from day 0 (blue) through day 10 (red), while the climatological eigenvalue distribution remains fixed.
The PP for relative vorticity is also included for comparison (Fig. 10d). Consistent with our results from section 3, PP suggests a lack of predictability for relative vorticity in <24 h, suggesting again that area-averaged quantities are substantially more predictable than nonaveraged quantities.
Predictive information is calculated in order to quantify the broadening of the forecast PDF with increasing forecast time and compare it to the relative width of the climatological PDF (DelSole 2004). As such, some insight should be provided by examining the structure of the forecast error variance over the tropical Atlantic. Recall that while the mean PP for circulation became negative at 180 h, there were individual forecasts where the PP became negative at <48 h while other forecasts had positive PP through 240 h (Fig. 10a). The 156-h predicted forecast error variance for the highest PP valid at 0000 UTC 5 September 2010 forecast (Fig. 11a, denoted “A”) is compared with the 48-h predicted forecast error variance for the lowest PP forecast valid at 0000 UTC 14 September 2010 forecast (Fig. 11b, denoted “B”). It is apparent that, despite the fact that A is from a much longer-range forecast than B, there is overall much lower forecast error variance in A than in B. On 5 September, only a small error variance exists associated with the location and strength of a tropical wave over the Leeward Islands and Puerto Rico, which happens to be the remnants of Tropical Storm Gaston, along with another wave that has yet to emerge off of Africa. On the other hand, on 14 September, there is a high amount of error variance associated with three TCs across the basin: Igor, Julia, and Karl. This comparison illustrates a common example of when forecasts associated with greater (lesser) error variance are associated with a greater (lesser) determinant of the forecast error covariance matrix and thereby lower (greater) PP. Note that a high ensemble mean circulation does not guarantee a high forecast error variance, as Karl in the Bay of Campeche was associated with much lower forecast error variance than Julia (near 42°W), despite the fact that Karl had a stronger ensemble-mean forecast circulation than Julia (Fig. 11b). Forecast error variance is instead a more direct measure of the spread, and thereby the uncertainty, in the ensemble forecasts.
Ensemble mean 850–700-hPa circulation (magenta contours, 2.5 × 10−5 s−1 increments) and circulation forecast error variance (shaded) for (a) a low-uncertainty 156-h forecast valid at 0000 UTC 5 Sep 2010 and (b) a high-uncertainty 72-h forecast valid at 0000 UTC 14 Sep 2010. Error variance for circulation from (c) ERA-Interim 1979–2011 is compared against the mean (d) 0-h, (e) 120-h, and (f) 240-h forecast error variance from the ECMWF ensemble for August–October 2010.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
The above example demonstrates the utility of PP in individual test cases. However, in order to abide by the Gaussian assumption and for the 5% confidence interval to have meaning, it is necessary to examine the season as a whole and compare against the 33-yr climatology. The 33-yr August–October climatological error variance for circulation shows a local maximum right off the coast of Africa near the Cape Verde islands, as well as along the dominant recurving TC track within the northern MDR (Fig. 11c). The mean August–October 2010 0-h forecast error variance (i.e., analysis error variance) for circulation indicates overall low uncertainty across the basin, but a maximum near the Cape Verde islands (Fig. 11d). The mean forecast error variance increases rapidly through 120 h (Fig. 11e), continuing to increase but at a slightly lower growth rate through 240 h (Fig. 11f). The overall structures of the climatological and forecast error variance plots are similar, but with greater error variance closer to 25°N in the forecast distribution and greater error variance closer to 10°N in the climatological error variance (along and to the north of the mean September circulation exceeding 5 × 10−6 s−1 from Fig. 3d). This is likely a result of there being only a few strong low-latitude tropical waves, and numerous stronger waves and tropical cyclones farther north in 2010. The basin mean forecast error variance does not exceed the climatological error variance at 0 h, is of comparable magnitude at 120 h, and definitely does so by 240 h. Therefore, it is not surprising that the determinant of the forecast error covariance matrix exceeds the determinant of the climatological error covariance matrix in the PP calculation at 180 h.
Next, PP for 850–200-hPa wind shear is examined. Overall, the estimated predictability limit for shear is very similar to that of circulation, approximately 168 h (Fig. 12a). However, the upper and lower bounds of the PP estimate for shear are significantly narrower than for circulation. This is likely due to the fact that, while uncertainties associated with tropical waves and cyclones introduce extreme local maxima in error variance for circulation (and pronounced minima in PP) there is no analog for shear. However, there is still some flow-dependent variability associated with PP for shear. For example, during September 2010, most ensemble forecasts associated with greatest PP are during 1–10 September 2010, and those ensemble forecasts associated with the lowest PP are during 21–30 September 2010. Correspondingly, the average 500-hPa geopotential height pattern during the first 10 days of September (Fig. 12b) was much less amplified than the average pattern during the final 10 days (Fig. 12c). Similar results were found for various 10-day periods during August and October 2010. These results demonstrate that PP is highly dependent on the flow regime, with a general trend that PP for shear and thereby predictability of shear decrease during more amplified flow regimes. The greater forecast error variance for shear is found in the northern half of the domain (not shown), where greater day-to-day variability will lead to greater uncertainty in the forecast. This is also the region associated with the greatest month-to-month variability for shear (Fig. 1).
(a) Predictive power for 850–200-hPa wind shear for consecutive 10-day time intervals from September 2010. 500-hPa geopotential height anomalies (m) for (b) 1–10 Sep and (c) 21–30 Sep 2010.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
Predictive power suggests a predictability limit of <144 h for 700-hPa relative humidity (Fig. 13a), lower than that for either circulation or shear. It is likely that smaller-scale variability associated with regions of convection lead to faster growth of forecast error variance for RH than for either circulation or shear, for which the error variance is dominated by mesoscale or synoptic scale variability. However, in practice, it can be very difficult to separate the component of error variance for RH associated with convection from that which is associated with the synoptic-scale movement of moist and dry air masses. The overall structure of the August–October 2010 forecast error variance (Fig. 13b) closely resembles the August–October 1979–2011 climatological error variance across the basin (Fig. 13c). The greatest error variance for RH is generally confined to the northern half of the domain, where there are alternating moist and dry periods. Farther south, particularly along the ITCZ into the Caribbean, error variance for RH is very low. This is due to the fact that these regions are almost always very moist (Fig. 2), leading to both a relatively uniform climatology as well as low uncertainty in the forecast.
(a) Predictive power for 700-hPa relative humidity during August–October 2010, with the 95% confidence interval and the 5% significance level. (b) The 180-h forecast error variance during those 3 months as well as (c) the 1979–2011 climatological error variance for relative humidity are also indicated.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
Predictive power for 200-hPa divergence drops sharply and drastically within the first 24 h of the ensemble forecast (Fig. 14a). Beyond 24 h, PP for divergence is relatively steady. This is in agreement with the fact that forecast error variance for divergence grows very large by t = 24 h (Fig. 14b), particularly along the ITCZ and within the southern MDR, but barely amplifies at all and even decreases in some regions by 240 h (Fig. 14c). Since convective-scale predictability is closely tied to the small scales (Zhang et al. 2007) and errors at the small scales grow and saturate the most rapidly (Lorenz 1982), it is not surprising that divergence becomes unpredictable in less than 24 h due to the natural association between divergence and areas of convection.
(a) Predictive power for 200-hPa divergence during August–October 2010, with the 95% confidence interval and the 5% significance level included. The (b) 24- and (c) 240-h mean forecast error variance for divergence during those 3 months are indicated.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
In sharp contrast to the results for divergence yet consistent with the aforementioned results examining error growth (Fig. 6), the 200-hPa velocity potential is found to have much greater predictive power than divergence (Fig. 15a). Velocity potential is found to be predictable out to 180 h, on average. Regions of maximum forecast error variance for velocity potential (Figs. 15b,c) correspond with regions that had the greatest forecast error variance for divergence (Figs. 14b,c), which also happen to be regions that are the most persistently convectively active. However, unlike 200-hPa divergence, forecast error variance for velocity potential is extremely small at 24 h in comparison with what it grows to be by 240 h.
As in Fig. 14, but for 200-hPa velocity potential.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
Last, 850–200-hPa thickness anomaly is associated with the greatest predictive power of all the chosen variables, to beyond 240 h (Fig. 16a). Correspondingly, there is a remarkable increase in forecast error variance from t = 24 (Fig. 16b) to t = 240 h (Fig. 16c). Similar to both RH and shear, the greatest forecast error variance for thickness anomaly is confined to the northern half of the domain, where greater uncertainty associated with synoptic features decreases predictability. Unlike for circulation, in which tropical cyclones are a significant contributor to the total error variance, the majority of the error variance for thickness anomalies does not appear to be associated with the warm cores of tropical cyclones.
As in Fig. 14, but for 850–200-hPa thickness anomaly.
Citation: Monthly Weather Review 142, 8; 10.1175/MWR-D-13-00370.1
5. Conclusions
Several metrics to evaluate predictive skill and attempt to quantify predictability have been explored using the ECMWF Ensemble Prediction System during the 2010 Atlantic hurricane season, in the context of large-scale variables in the tropical Atlantic basin that are relevant to tropical cyclogenesis. These metrics include 1) growth and saturation of RMS error, 2) RMS errors versus climatology, 3) predicted forecast error standard deviation (or variance), and 4) predictive power.
When error growth was computed in the same vein as the classical study of Lorenz (1982), basinwide errors were found to grow linearly beyond day 2, and began to saturate slowly after day 8. On average, the errors in the ensemble mean forecast were smaller than those from the corresponding control forecast, thereby extending the range of predictability. For variables including 850–700-hPa circulation, 200-hPa velocity potential, 700-hPa relative humidity, and 850–200-hPa vertical wind shear, the ensemble mean still exhibited skill (cf. climatology) beyond 9 days. In contrast, the 200-hPa divergence and 850-hPa relative vorticity were virtually indistinguishable from climatology beyond 2 days. Similar results were obtained using predictive power, in that the most predictable variables were 850–200-hPa thickness anomaly, 850–200-hPa wind shear, and 200-hPa velocity potential, with predictability estimated to be greater than one week. Relative humidity at 700- and 850–700-hPa circulation were also found to have predictive power out to about a week. Last, 200-hPa divergence and 850-hPa vorticity were found to be the least predictable, only to about 12 h using predictive power. An evaluation of the ability of the ensemble to predict the standard deviation of the forecast error in 10° × 10° boxes across the domain revealed a mostly linear increasing relationship between the predicted and actual standard deviations for all investigated variables, for forecast times up to and beyond a week.
There exists a considerable degree of variability from day-to-day and between successive initial conditions. This is the result of the evolving flow, with some flow regimes being more predictable than others as first put forth by Lorenz (1965). The degree of variability also depends on the atmospheric variables under consideration. While the predictive power of metrics such as wind shear exhibited relative low variability, the predictive power for circulation was found to decrease considerably in the presence of multiple tropical waves and tropical cyclones. Whether the relationship between predictability of variables relevant to tropical cyclogenesis and the subseasonal potential predictability may be connected to the Madden–Julian oscillation (Madden and Julian 1971) or other equatorial waves requires investigation. Future studies that estimate large-scale predictability in the tropical Atlantic using additional hurricane seasons are encouraged to determine the robustness of the presented results.
Although the conclusions may differ as a result of the metric used and the flow, the predictability is generally dependent on the scales of motion (following Lorenz 1969). Those metrics typically associated with larger spatial scales (and slower temporal scales of variability) were generally found to be more predictable than those associated with smaller spatial scales (and faster temporal scales). For these reasons, properties of the atmospheric flow that are mostly driven by the large scales (such as tropical cyclone track) are more predictable than those quantities that are mostly driven by the small scales (such as tropical cyclone intensity). For tropical cyclogenesis itself, the local scales of motion most directly relevant to the genesis process remain to be quantified and understood, together with the associated predictability in a frame of reference centered on the tropical disturbance under investigation. A forthcoming companion paper will explore these issues within a storm-relative framework.
Acknowledgments
The authors gratefully acknowledge funding from the National Science Foundation Grant ATM-0848753. The authors thank Rich Rotunno and Chris Davis of NCAR, Ryan Torn of SUNY at Albany, Brian Mapes and David Nolan of the University of Miami, Philip Klotzbach of Colorado State University, and two anonymous reviewers for their comments and suggestions, which have certainly improved this study. The first author would also like to thank and acknowledge support from the ASP student visitor program at NCAR. Last, we thank TIGGE for the availability of the data used in this study.
REFERENCES
Abramov, R., A. Majda, and R. Kleeman, 2005: Information theory and predictability for low-frequency variability. J. Atmos. Sci., 62, 65–87, doi:10.1175/JAS-3373.1.
Anderson, J. L., and W. F. Stern, 1996: Evaluating the potential predictive utility of ensemble forecasts. J. Climate, 9, 260–269, doi:10.1175/1520-0442(1996)009<0260:ETPPUO>2.0.CO;2.
Berner, J., G. J. Shutts, M. Leutbecher, and T. N. Palmer, 2009: A spectral stochastic kinetic energy backscatter scheme and its impact on flow-dependent predictability in the ECMWF ensemble prediction system. J. Atmos. Sci., 66, 603–626, doi:10.1175/2008JAS2677.1.
Bijlsma, J. S., L. M. Hafkenscheid, and P. Lynch, 1986: Computation of the streamfunction and velocity potential and reconstruction of the wind field. Mon. Wea. Rev., 114, 1547–1551, doi:10.1175/1520-0493(1986)114<1547:COTSAV>2.0.CO;2.
Bougeault, P., and Coauthors, 2010: The THORPEX Interactive Grand Global Ensemble. Bull. Amer. Meteor. Soc., 91, 1059–1072, doi:10.1175/2010BAMS2853.1.
Dee, D. P., and Coauthors, 2011: The ERA-Interim Reanalysis: Configuration and performance of the data assimilation system. Quart. J. Roy. Meteor. Soc., 137, 553–597, doi:10.1002/qj.828.
DelSole, T., 2004: Predictability and information theory. Part I: Measures of predictability. J. Atmos. Sci., 61, 2425–2440, doi:10.1175/1520-0469(2004)061<2425:PAITPI>2.0.CO;2.
DeMaria, J., A. Knaff, and B. H. Connell, 2001: A tropical cyclone genesis parameter for the tropical Atlantic. Wea. Forecasting, 16, 219–233, doi:10.1175/1520-0434(2001)016<0219:ATCGPF>2.0.CO;2.
Dunion, J. P., 2011: Rewriting the climatology of the tropical North Atlantic and Caribbean Sea atmosphere. J. Climate, 24, 893–908, doi:10.1175/2010JCLI3496.1.
Giannakis, D., and A. J. Majda, 2012: Quantifying the predictive skill in long-range forecasting. Part I: Coarse-grained predictions in a simple ocean model. J. Climate, 25, 1793–1813, doi:10.1175/2011JCLI4143.1.
Golub, G. H., and C. F. Van Loan, 1996: Matrix Computations. 3rd ed. Johns Hopkins University Press, 694 pp.
Gray, W. M., 1979: Hurricanes: Their formation, structure and likely role in the tropical circulation. Meteorology over the Tropical Oceans, D. B. Shaw, Ed., Royal Meteorological Society, 155–218.
Griffies, S. M., and K. Bryan, 1997: A predictability study of simulated North Atlantic multidecadal variability. Climate Dyn., 13, 459–488, doi:10.1007/s003820050177.
Grimit, E. P., 2004: Probabilistic mesoscale forecast error prediction using short-range ensembles. Ph.D. dissertation, University of Washington, 146 pp.
Grimit, E. P., and C. F. Mass, 2007: Measuring the ensemble spread–error relationship with a probabilistic approach: Stochastic ensemble results. Mon. Wea. Rev., 135, 203–221, doi:10.1175/MWR3262.1.
Halperin, D. J., H. E. Fuelberg, R. E. Hart, J. H. Cossuth, P. Sura, and R. J. Pasch, 2013: An evaluation of tropical cyclone genesis forecasts from global numerical models. Wea. Forecasting, 28, 1423–1445, doi:10.1175/WAF-D-13-00008.1.
Hayashi, Y., 1986: Statistical interpretation of ensemble-time mean predictability. J. Meteor. Soc. Japan, 64, 167–181.
Kleeman, R., 2002: Measuring dynamical prediction utility using relative entropy. J. Atmos. Sci., 59, 2057–2072, doi:10.1175/1520-0469(2002)059<2057:MDPUUR>2.0.CO;2.
Kleeman, R., 2007: Information flow in ensemble weather predictions. J. Atmos. Sci., 64, 1005–1016, doi:10.1175/JAS3857.1.
Lorenz, E. N., 1963: Deterministic non-periodic flow. J. Atmos. Sci., 20, 130–141, doi:10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2.
Lorenz, E. N., 1965: A new study of the predictability of a 28-variable atmospheric model. Tellus, 17, 321–333, doi:10.1111/j.2153-3490.1965.tb01424.x.
Lorenz, E. N., 1969: Predictability of a flow which possesses many scales of motion. Tellus, 21, 289–387, doi:10.1111/j.2153-3490.1969.tb00444.x.
Lorenz, E. N., 1982: Atmospheric predictability experiments with a large numerical-model. Tellus, 34, 505–513, doi:10.1111/j.2153-3490.1982.tb01839.x.
Madden, R. A., and P. R. Julian, 1971: Detection of a 40–50 day oscillation in the zonal wind in the tropical Pacific. J. Atmos. Sci., 28, 702–708, doi:10.1175/1520-0469(1971)028<0702:DOADOI>2.0.CO;2.
McMurdie, L. A., and B. Ancell, 2014: Predictability characteristics of landfalling cyclones along the North American west coast. Mon. Wea. Rev., 142, 301–319, doi:10.1175/MWR-D-13-00141.1.
Montgomery, M. T., and Coauthors, 2012: The Pre-Depression Investigation of Cloud Systems in the Tropics (PREDICT) experiment: Scientific basis, new analysis tools, and some first results. Bull. Amer. Meteor. Soc., 93, 153–172, doi:10.1175/BAMS-D-11-00046.1.
Murphy, A. H., 1988: Skill scores based on the mean square error and their relationships to the correlation coefficient. Mon. Wea. Rev., 116, 2417–2424, doi:10.1175/1520-0493(1988)116<2417:SSBOTM>2.0.CO;2.
Palmer, T. N., 1996: Predictability of the atmosphere and oceans: From days to decades. Decadal Climate Variability: Dynamics and Predictability, D. L. T. Anderson and J. Willebrand, Eds., NATO ASI Series, Vol. I, No. 44, Springer, 83–155.
Palmer, T. N., R. Gelaro, J. Barkmeijer, and R. Buizza, 1998: Singular vectors, metrics, and adaptive observations. J. Atmos. Sci., 55, 633–653, doi:10.1175/1520-0469(1998)055<0633:SVMAAO>2.0.CO;2.
Palmer, T. N., R. Buizza, F. Doblas-Reyes, T. Jung, M. Leutbecher, G. J. Shutts, M. Steinheimer, and A. Weisheimer, 2009: Stochastic parametrization and model uncertainty. ECMWF Tech. Memo. 598, ECMWF, Reading, United Kingdom, 42 pp.
Schneider, T., and S. M. Griffies, 1999: A conceptual framework for predictability studies. J. Climate, 12, 3133–3155, doi:10.1175/1520-0442(1999)012<3133:ACFFPS>2.0.CO;2.
Schneider, T., and I. M. Held, 2001: Discriminants of twentieth-century changes in earth surface temperatures. J. Climate, 14, 249–254, doi:10.1175/1520-0442(2001)014<0249:LDOTCC>2.0.CO;2.
Schumacher, A. B., M. DeMaria, and J. A. Knaff, 2009: Objective estimation of 24-h probability of tropical cyclone formation. Wea. Forecasting, 24, 456–471, doi:10.1175/2008WAF2007109.1.
Shieh, O. H., and S. J. Colucci, 2010: Local minimum of tropical cyclogenesis in the eastern Caribbean. Bull. Amer. Meteor. Soc., 91, 185–196, doi:10.1175/2009BAMS2822.1.
Shukla, J., 1981: Dynamical predictability of monthly means. J. Atmos. Sci., 38, 2547–2572, doi:10.1175/1520-0469(1981)038<2547:DPOMM>2.0.CO;2.
Shukla, J., 1985: Predictability. Advances in Geophysics, Vol. 28b, Academic Press, 87–122.
Snyder, A., Z. Pu, and Y. Zhu, 2010: Tracking and verification of the east Atlantic tropical cyclone genesis in NCEP global ensemble: Case studies during NASA African monsoon multi-disciplinary analyses. Wea. Forecasting, 25, 1397–1411, doi:10.1175/2010WAF2222332.1.
Snyder, A., Z. Pu, and C. A. Reynolds, 2011: Impact of stochastic convection on ensemble forecasts of tropical cyclone development. Mon. Wea. Rev., 139, 620–626, doi:10.1175/2010MWR3341.1.
Stephenson, D., and F. Doblas-Reyes, 2000: Statistical methods for interpreting Monte Carlo ensemble forecasts. Tellus, 52A, 300–322, doi:10.1034/j.1600-0870.2000.d01-5.x.
Tsai, H.-C., R. L. Elsberry, M. S. Jordan, and F. Vitart, 2013: Objective verifications and false alarm analyses of western North Pacific tropical cyclone event forecasts by the ECMWF 32-day ensemble. Asia-Pac. J. Atmos. Sci., 49, 409–420, doi:10.1007/s13143-013-0038-6.
Ventrice, M. J., M. C. Wheeler, H. H. Hendon, C. J. Schreck III, C. D. Thorncroft, and G. N. Kiladis, 2013: A modified multivariate Madden Julian Oscillation index using velocity potential. Mon. Wea. Rev., 141, 4197–4210, doi:10.1175/MWR-D-12-00327.1.
Waliser, D. E., W. Stern, S. Schubert, and K. M. Lau, 2003: Dynamic predictability of intraseasonal variability associated with the Asian summer monsoon. Quart. J. Roy. Meteor. Soc., 129, 2897–2925, doi:10.1256/qj.02.51.
Wang, X. G., and C. H. Bishop, 2003: A comparison of breeding and ensemble transform Kalman filter ensemble forecast schemes. J. Atmos. Sci., 60, 1140–1158, doi:10.1175/1520-0469(2003)060<1140:ACOBAE>2.0.CO;2.
Zhang, F., N. Bei, R. Rotunno, C. Snyder, and C. C. Epifanio, 2007: Mesoscale predictability of moist baroclinic waves: Convection-permitting experiments and multistage error growth dynamics. J. Atmos. Sci., 64, 3579–3594, doi:10.1175/JAS4028.1.
The area from 100°–80°W south of 15°N is excluded to ensure that the east Pacific does not factor into these calculations.
