Search Results

You are looking at 1 - 10 of 10 items for :

  • Author or Editor: Elizabeth E. Ebert x
  • Weather and Forecasting x
  • All content x
Clear All Modify Search
Elizabeth E. Ebert

Abstract

High-resolution forecasts may be quite useful even when they do not match the observations exactly. Neighborhood verification is a strategy for evaluating the “closeness” of the forecast to the observations within space–time neighborhoods rather than at the grid scale. Various properties of the forecast within a neighborhood can be assessed for similarity to the observations, including the mean value, fractional coverage, occurrence of a forecast event sufficiently near an observed event, and so on. By varying the sizes of the neighborhoods, it is possible to determine the scales for which the forecast has sufficient skill for a particular application. Several neighborhood verification methods have been proposed in the literature in the last decade. This paper examines four such methods in detail for idealized and real high-resolution precipitation forecasts, highlighting what can be learned from each of the methods. When applied to idealized and real precipitation forecasts from the Spatial Verification Methods Intercomparison Project, all four methods showed improved forecast performance for neighborhood sizes larger than grid scale, with the optimal scale for each method varying as a function of rainfall intensity.

Full access
Chermelle Engel and Elizabeth E. Ebert

Abstract

This paper describes an extension of an operational consensus forecasting (OCF) scheme from site forecasts to gridded forecasts. OCF is a multimodel consensus scheme including bias correction and weighting. Bias correction and weighting are done on a scale common to almost all multimodel inputs (1.25°), which are then downscaled using a statistical approach to an approximately 5-km-resolution grid. Local and international numerical weather prediction model inputs are found to have coarse scale biases that respond to simple bias correction, with the weighted average consensus at 1.25° outperforming all models at that scale. Statistical downscaling is found to remove the systematic representativeness error when downscaling from 1.25° to 5 km, though it cannot resolve scale differences associated with transient small-scale weather.

Full access
John L. McBride and Elizabeth E. Ebert

Abstract

Real-time gridded 24-h quantitative precipitation forecasts from seven operational NWP models are verified over the Australian continent. All forecasts have been mapped to a 1° latitude–longitude grid and have been verified against an operational daily rainfall analysis, mapped to the same grid. The verification focuses on two large subregions: the northern tropical monsoon regime and the southeastern subtropical regime. Statistics are presented of the bias score, probability of detection, and false alarm ratio for a range of rainfall threshold values. The basic measure of skill used in this study, however, is the Hanssen and Kuipers (HK) score and its two components: accuracy for events and accuracy for nonevents.

For both regimes the operational models tend to overestimate rainfall in summer and to underestimate it in winter. In the southeastern region the models have HK scores ranging from 0.5 to 0.7, and easily outperform a forecast of persistence. Thus for the current operational NWP models, the 24-h rain forecasts can be considered quite skillful in the subtropics. On the other hand, model skill is quite low in the northern regime with HK values of only 0.2–0.6. During the summer wet season the low skill is associated with an inability to simulate the behavior of tropical convective rain systems. During the winter dry season, it is associated with a low probability of detection for the occasional rainfall event. Thus it could be said that models have no real skill at rainfall forecasts in this monsoonal wet season regime.

Model skill falls dramatically for occurrence thresholds greater than 10 mm day−1. This implies that the models are much better at predicting the occurrence of rain than they are at predicting the magnitude and location of the peak values.

Full access
Elizabeth E. Ebert and William A. Gallus Jr.

Abstract

The contiguous rain area (CRA) method for spatial forecast verification is a features-based approach that evaluates the properties of forecast rain systems, namely, their location, size, intensity, and finescale pattern. It is one of many recently developed spatial verification approaches that are being evaluated as part of a Spatial Forecast Verification Methods Intercomparison Project. To better understand the strengths and weaknesses of the CRA method, it has been tested here on a set of idealized geometric and perturbed forecasts with known errors, as well as nine precipitation forecasts from three high-resolution numerical weather prediction models.

The CRA method was able to identify the known errors for the geometric forecasts, but only after a modification was introduced to allow nonoverlapping forecast and observed features to be matched. For the perturbed cases in which a radar rain field was spatially translated and amplified to simulate forecast errors, the CRA method also reproduced the known errors except when a high-intensity threshold was used to define the CRA (≥10 mm h−1) and a large translation error was imposed (>200 km). The decomposition of total error into displacement, volume, and pattern components reflected the source of the error almost all of the time when a mean squared error formulation was used, but not necessarily when a correlation-based formulation was used.

When applied to real forecasts, the CRA method gave similar results when either best-fit criteria, minimization of the mean squared error, or maximization of the correlation coefficient, was chosen for matching forecast and observed features. The diagnosed displacement error was somewhat sensitive to the choice of search distance. Of the many diagnostics produced by this method, the errors in the mean and peak rain rate between the forecast and observed features showed the best correspondence with subjective evaluations of the forecasts, while the spatial correlation coefficient (after matching) did not reflect the subjective judgments.

Full access
David Ahijevych, Eric Gilleland, Barbara G. Brown, and Elizabeth E. Ebert

Abstract

Several spatial forecast verification methods have been developed that are suited for high-resolution precipitation forecasts. They can account for the spatial coherence of precipitation and give credit to a forecast that does not necessarily match the observation at any particular grid point. The methods were grouped into four broad categories (neighborhood, scale separation, features based, and field deformation) for the Spatial Forecast Verification Methods Intercomparison Project (ICP). Participants were asked to apply their new methods to a set of artificial geometric and perturbed forecasts with prescribed errors, and a set of real forecasts of convective precipitation on a 4-km grid. This paper describes the intercomparison test cases, summarizes results from the geometric cases, and presents subjective scores and traditional scores from the real cases.

All the new methods could detect bias error, and the features-based and field deformation methods were also able to diagnose displacement errors of precipitation features. The best approach for capturing errors in aspect ratio was field deformation. When comparing model forecasts with real cases, the traditional verification scores did not agree with the subjective assessment of the forecasts.

Full access
Eric Gilleland, David Ahijevych, Barbara G. Brown, Barbara Casati, and Elizabeth E. Ebert

Abstract

Advancements in weather forecast models and their enhanced resolution have led to substantially improved and more realistic-appearing forecasts for some variables. However, traditional verification scores often indicate poor performance because of the increased small-scale variability so that the true quality of the forecasts is not always characterized well. As a result, numerous new methods for verifying these forecasts have been proposed. These new methods can mostly be classified into two overall categories: filtering methods and displacement methods. The filtering methods can be further delineated into neighborhood and scale separation, and the displacement methods can be divided into features based and field deformation. Each method gives considerably more information than the traditional scores, but it is not clear which method(s) should be used for which purpose.

A verification methods intercomparison project has been established in order to glean a better understanding of the proposed methods in terms of their various characteristics and to determine what verification questions each method addresses. The study is ongoing, and preliminary qualitative results for the different approaches applied to different situations are described here. In particular, the various methods and their basic characteristics, similarities, and differences are described. In addition, several questions are addressed regarding the application of the methods and the information that they provide. These questions include (i) how the method(s) inform performance at different scales; (ii) how the methods provide information on location errors; (iii) whether the methods provide information on intensity errors and distributions; (iv) whether the methods provide information on structure errors; (v) whether the approaches have the ability to provide information about hits, misses, and false alarms; (vi) whether the methods do anything that is counterintuitive; (vii) whether the methods have selectable parameters and how sensitive the results are to parameter selection; (viii) whether the results can be easily aggregated across multiple cases; (ix) whether the methods can identify timing errors; and (x) whether confidence intervals and hypothesis tests can be readily computed.

Full access
Jeremy S. Grams, Willam A. Gallus Jr., Steven E. Koch, Linda S. Wharton, Andrew Loughe, and Elizabeth E. Ebert

Abstract

The Ebert–McBride technique (EMT) is an entity-oriented method useful for quantitative precipitation verification. The EMT was modified to optimize its ability to identify contiguous rain areas (CRAs) during the 2002 International H2O Project (IHOP). This technique was then used to identify systematic sources of error as a function of observed convective system morphology in three 12-km model simulations run over the IHOP domain: Eta, the fifth-generation Pennsylvania State University–NCAR Mesoscale Model (MM5), and the Weather Research and Forecasting (WRF). The EMT was fine-tuned to optimize the pattern matching of forecasts to observations for the scales of precipitation systems observed during IHOP. To investigate several error measures provided by the EMT, a detailed morphological analysis of observed systems was performed using radar data for all CRAs identified in the IHOP domain. The modified EMT suggests that the Eta Model produced average rain rates, peak rainfall amounts, and total rain volumes that were lower than observed for almost all types of convective systems, likely because of its production of overly smoothed and low-variability quantitative precipitation forecasts. The MM5 and WRF typically produced average rain rates and peak rainfall amounts that were larger than observed in most linear convective systems. However, the rain volume for these models was too low for almost all types of convective systems, implying a sizeable underestimate in areal coverage. All three models forecast rainfall too far northwest for linear systems. The results for the WRF and MM5 are consistent with previous observations of mesoscale models run with explicit microphysics and no convective parameterization scheme, suggesting systematic problems with the prediction of mesoscale convective system cold pool dynamics.

Full access
Elizabeth E. Ebert, Laurence J. Wilson, Barbara G. Brown, Pertti Nurmi, Harold E. Brooks, John Bally, and Matthias Jaeneke

Abstract

The verification phase of the World Weather Research Programme (WWRP) Sydney 2000 Forecast Demonstration Project (FDP) was intended to measure the skill of the participating nowcast algorithms in predicting the location of convection, rainfall rate and occurrence, wind speed and direction, severe thunderstorm wind gusts, and hail location and size. An additional question of interest was whether forecasters could improve the quality of the nowcasts compared to the FDP products alone.

The nowcasts were verified using a variety of statistical techniques. Observational data came from radar reflectivity and rainfall analyses, a network of rain gauges, and human (spotter) observations. The verification results showed that the cell tracking algorithms predicted the location of the strongest cells with a mean error of about 15–30 km for a 1-h forecast, and were usually more accurate than an extrapolation (Lagrangian persistence) forecast. Mean location errors for the area tracking schemes were on the order of 20 km.

Almost all of the algorithms successfully predicted the frequency of rain throughout the forecast period, although most underestimated the frequency of high rain rates. The skill in predicting rain occurrence decreased very quickly into the forecast period. In particular, the algorithms could not predict the precise location of heavy rain beyond the first 10–20 min. Using radar analyses as verification, the algorithms' spatial forecasts were consistently more skillful than simple persistence. However, when verified against rain gauge observations at point locations, the algorithms had difficulty beating persistence, mainly due to differences in spatial and temporal resolution.

Only one algorithm attempted to forecast gust fronts. The results for a limited sample showed a mean absolute error of 7 km h−1 and mean bias of 3 km h−1 in the speed of the gust fronts during the FDP. The errors in sea-breeze front forecasts were half as large, with essentially no bias. Verification of the hail associated with the 3 November tornadic storm showed that the two algorithms that estimated hail size and occurrence successfully diagnosed the onset and cessation of the hail to within 30 min of the reported sightings. The time evolution of hail size was reasonably well captured by the algorithms, and the predicted mean and maximum hail diameters were consistent with the observations.

The Thunderstorm Interactive Forecast System (TIFS) allowed forecasters to modify the output of the cell tracking nowcasts, primarily using it to remove cells that were insignificant or diagnosed with incorrect motion. This manual filtering resulted in markedly reduced mean cell position errors when compared to the unfiltered forecasts. However, when forecasters attempted to adjust the storm tracks for a small number of well-defined intense cells, the position errors increased slightly, suggesting that in such cases the objective guidance is probably the best estimate of storm motion.

Full access
Elizabeth E. Ebert, Michael Turk, Sheldon J. Kusselson, Jianbin Yang, Matthew Seybold, Peter R. Keehn, and Robert J. Kuligowski

Abstract

Ensemble tropical rainfall potential (eTRaP) has been developed to improve short-range forecasts of heavy rainfall in tropical cyclones. Evolving from the tropical rainfall potential (TRaP), a 24-h rain forecast based on estimated rain rates from microwave sensors aboard polar-orbiting satellites, eTRaP combines all single-pass TRaPs generated within ±3 h of 0000, 0600, 1200, and 1800 UTC to form a simple ensemble. This approach addresses uncertainties in satellite-derived rain rates and spatial rain structures by using estimates from different sensors observing the cyclone at different times. Quantitative precipitation forecasts (QPFs) are produced from the ensemble mean field using a probability matching approach to recalibrate the rain-rate distribution against the ensemble members (e.g., input TRaP forecasts) themselves. ETRaPs also provide probabilistic forecasts of heavy rain, which are potentially of enormous benefit to decision makers. Verification of eTRaP forecasts for 16 Atlantic hurricanes making landfall in the United States between 2004 and 2008 shows that the eTRaP rain amounts are more accurate than single-sensor TRaPs. The probabilistic forecasts have useful skill, but the probabilities should be interpreted within a spatial context. A novel concept of a “radius of uncertainty” compensates for the influence of location error in the probability forecasts. The eTRaPs are produced in near–real time for all named tropical storms and cyclones around the globe. They can be viewed online (http://www.ssd.noaa.gov/PS/TROP/etrap.html) and are available in digital form to users.

Full access
James W. Wilson, Elizabeth E. Ebert, Thomas R. Saxen, Rita D. Roberts, Cynthia K. Mueller, Michael Sleigh, Clive E. Pierce, and Alan Seed

Abstract

Five of the nowcasting systems that were available during the Sydney 2000 Forecast Demonstration Project (FDP) were selected for evaluation. These systems, from the United States, the United Kingdom, and Australia, had the capability to nowcast the location and, with one exception, the intensity of convective storms. Six of the most significant convective storm cases from the 3-month FDP were selected for evaluating the performance of these state-of-the-art nowcasting systems, which extrapolated storms using a variety of methods, including cell and area tracking, model winds, and sounding winds. Three of the systems had the ability to forecast the initiation and growth of storms. Nowcasts for 30 and 60 min were evaluated, and it was found that even for such short time periods the skill of the extrapolation-only systems was often very low. Extrapolation techniques that allowed for differential motion performed slightly better, since high-impact storms often have motions different than surrounding storms. The ability to forecast initiation, growth, and dissipation is in its infancy. However, it was demonstrated that significant improvement in forecast accuracy was obtained for several of these cases when the locations of boundary layer convergence lines (sea breeze and gust fronts) were used in the nowcasts.

Based on the experiences during the FDP, and in forecast offices in the United States, a discussion is provided of the overall status of nowcasting convective storms. In addition, proposed future directions are discussed concerning the specificity of nowcast products, experimental test beds, and additional observations and research required.

Full access