Search Results
You are looking at 1 - 10 of 32 items for
- Author or Editor: S. Richardson x
- Refine by Access: All Content x
Abstract
The relative operating characteristic (ROC) curve is a popular diagnostic tool in forecast verification, with the area under the ROC curve (AUC) used as a verification metric measuring the discrimination ability of a forecast. Along with calibration, discrimination is deemed as a fundamental probabilistic forecast attribute. In particular, in ensemble forecast verification, AUC provides a basis for the comparison of potential predictive skill of competing forecasts. While this approach is straightforward when dealing with forecasts of common events (e.g., probability of precipitation), the AUC interpretation can turn out to be oversimplistic or misleading when focusing on rare events (e.g., precipitation exceeding some warning criterion). How should we interpret AUC of ensemble forecasts when focusing on rare events? How can changes in the way probability forecasts are derived from the ensemble forecast affect AUC results? How can we detect a genuine improvement in terms of predictive skill? Based on verification experiments, a critical eye is cast on the AUC interpretation to answer these questions. As well as the traditional trapezoidal approximation and the well-known binormal fitting model, we discuss a new approach that embraces the concept of imprecise probabilities and relies on the subdivision of the lowest ensemble probability category.
Abstract
The relative operating characteristic (ROC) curve is a popular diagnostic tool in forecast verification, with the area under the ROC curve (AUC) used as a verification metric measuring the discrimination ability of a forecast. Along with calibration, discrimination is deemed as a fundamental probabilistic forecast attribute. In particular, in ensemble forecast verification, AUC provides a basis for the comparison of potential predictive skill of competing forecasts. While this approach is straightforward when dealing with forecasts of common events (e.g., probability of precipitation), the AUC interpretation can turn out to be oversimplistic or misleading when focusing on rare events (e.g., precipitation exceeding some warning criterion). How should we interpret AUC of ensemble forecasts when focusing on rare events? How can changes in the way probability forecasts are derived from the ensemble forecast affect AUC results? How can we detect a genuine improvement in terms of predictive skill? Based on verification experiments, a critical eye is cast on the AUC interpretation to answer these questions. As well as the traditional trapezoidal approximation and the well-known binormal fitting model, we discuss a new approach that embraces the concept of imprecise probabilities and relies on the subdivision of the lowest ensemble probability category.
Abstract
Recent high-resolution numerical simulations of supercells have identified a feature referred to as the streamwise vorticity current (SVC). Some have presumed the SVC to play a role in tornadogenesis and maintenance, though observations of such a feature have been limited. To this end, 125-m dual-Doppler wind syntheses and mobile mesonet observations are used to examine three observed supercells for evidence of an SVC. Two of the three supercells are found to contain a feature similar to an SVC, while the other supercell contains an antistreamwise vorticity ribbon on the southern fringe of the forward flank. A closer examination of the two supercells with SVCs reveals that the SVCs are located on the cool side of boundaries within the forward flank that separate colder, more turbulent flow from warmer, more laminar flow, similar to numerical simulations. Furthermore, the observed SVCs are similar to those in simulations in that they appear to be associated with baroclinic vorticity generation and have similar appearances in vertical cross sections. Aside from some apparent differences in the location of the maximum streamwise vorticity between simulated and observed SVCs, the SVCs seen in numerical simulations are indeed similar to reality. The SVC, however, may not be essential for tornadogenesis, at least for weak tornadoes, because the supercell that did not have a well-defined SVC produced at least one brief, weak tornado during the analysis period.
Abstract
Recent high-resolution numerical simulations of supercells have identified a feature referred to as the streamwise vorticity current (SVC). Some have presumed the SVC to play a role in tornadogenesis and maintenance, though observations of such a feature have been limited. To this end, 125-m dual-Doppler wind syntheses and mobile mesonet observations are used to examine three observed supercells for evidence of an SVC. Two of the three supercells are found to contain a feature similar to an SVC, while the other supercell contains an antistreamwise vorticity ribbon on the southern fringe of the forward flank. A closer examination of the two supercells with SVCs reveals that the SVCs are located on the cool side of boundaries within the forward flank that separate colder, more turbulent flow from warmer, more laminar flow, similar to numerical simulations. Furthermore, the observed SVCs are similar to those in simulations in that they appear to be associated with baroclinic vorticity generation and have similar appearances in vertical cross sections. Aside from some apparent differences in the location of the maximum streamwise vorticity between simulated and observed SVCs, the SVCs seen in numerical simulations are indeed similar to reality. The SVC, however, may not be essential for tornadogenesis, at least for weak tornadoes, because the supercell that did not have a well-defined SVC produced at least one brief, weak tornado during the analysis period.
Abstract
Early awareness of extreme precipitation can provide the time necessary to make adequate event preparations. At the European Centre for Medium-Range Weather Forecasts (ECMWF), one tool that condenses the forecast information from the Integrated Forecasting System ensemble (ENS) is the extreme forecast index (EFI), an index that highlights regions that are forecast to have potentially anomalous weather conditions compared to the local climate. This paper builds on previous findings by undertaking a global verification throughout the medium-range forecast horizon (out to 15 days) on the ability of the EFI for water vapor transport [integrated vapor transport (IVT)] and precipitation to capture extreme observed precipitation. Using the ECMWF ENS for winters 2015/16 and 2016/17 and daily surface precipitation observations, the relative operating characteristic is used to show that the IVT EFI is more skillful than the precipitation EFI in forecast week 2 over Europe and western North America. It is the large-scale nature of the IVT, its higher predictability, and its relationship with extreme precipitation that result in its potential usefulness in these regions, which, in turn, could provide earlier awareness of extreme precipitation. Conversely, at shorter lead times the precipitation EFI is more useful, although the IVT EFI can provide synoptic-scale understanding. For the whole globe, the extratropical Northern Hemisphere, the tropics, and North America, the precipitation EFI is more useful throughout the medium range, suggesting that precipitation processes not captured in the IVT are important (e.g., tropical convection). Following these results, the operational implementation of the IVT EFI is currently being planned.
Abstract
Early awareness of extreme precipitation can provide the time necessary to make adequate event preparations. At the European Centre for Medium-Range Weather Forecasts (ECMWF), one tool that condenses the forecast information from the Integrated Forecasting System ensemble (ENS) is the extreme forecast index (EFI), an index that highlights regions that are forecast to have potentially anomalous weather conditions compared to the local climate. This paper builds on previous findings by undertaking a global verification throughout the medium-range forecast horizon (out to 15 days) on the ability of the EFI for water vapor transport [integrated vapor transport (IVT)] and precipitation to capture extreme observed precipitation. Using the ECMWF ENS for winters 2015/16 and 2016/17 and daily surface precipitation observations, the relative operating characteristic is used to show that the IVT EFI is more skillful than the precipitation EFI in forecast week 2 over Europe and western North America. It is the large-scale nature of the IVT, its higher predictability, and its relationship with extreme precipitation that result in its potential usefulness in these regions, which, in turn, could provide earlier awareness of extreme precipitation. Conversely, at shorter lead times the precipitation EFI is more useful, although the IVT EFI can provide synoptic-scale understanding. For the whole globe, the extratropical Northern Hemisphere, the tropics, and North America, the precipitation EFI is more useful throughout the medium range, suggesting that precipitation processes not captured in the IVT are important (e.g., tropical convection). Following these results, the operational implementation of the IVT EFI is currently being planned.
Abstract
The effects of observation errors on rank histograms and reliability diagrams are investigated using a perfect model approach. The three-variable Lorenz-63 model was used to simulate an idealized ensemble prediction system (EPS) with 50 perturbed ensemble members and one control forecast. Observation errors at verification time were introduced by adding normally distributed noise to the true state at verification time. Besides these simulations, a theoretical analysis was also performed. One of the major findings was that rank histograms are very sensitive to the presence of observation errors, leading to overpopulated upper- and lowermost ranks. This sensitivity was shown to grow for larger ensemble sizes. Reliability diagrams were far less sensitive in this respect. The resulting u-shaped rank histograms can easily be misinterpreted as indicating too little spread in the ensemble prediction system. To account for this effect when real observations are used to assess an ensemble prediction system, normally distributed noise based on the verifying observation error can be added to the ensemble members before the statistics are calculated. The method has been tested for the ECMWF ensemble forecasts of ocean waves and forecasts of the geopotential at 500 hPa. The EPS waves were compared with buoy observations from the Global Telecommunication System (GTS) for a period of almost 3 yr. When the buoy observations were taken as the true value, more than 25% of the observations appeared in the two extreme ranks for the day 3 forecast range. This number was reduced to less than 10% when observation errors were added to the ensemble members. Ensemble forecasts of the 500-hPa geopotential were verified against the ECMWF analysis. When analysis errors were neglected, the maximum number of outliers was more than 10% for most areas except for Europe, where the analysis errors are relatively smaller. Introducing noise to the ensemble members, based on estimates of analysis errors, reduced the number of outliers, particularly in the short range, where a peak around day 1 more or less vanished.
Abstract
The effects of observation errors on rank histograms and reliability diagrams are investigated using a perfect model approach. The three-variable Lorenz-63 model was used to simulate an idealized ensemble prediction system (EPS) with 50 perturbed ensemble members and one control forecast. Observation errors at verification time were introduced by adding normally distributed noise to the true state at verification time. Besides these simulations, a theoretical analysis was also performed. One of the major findings was that rank histograms are very sensitive to the presence of observation errors, leading to overpopulated upper- and lowermost ranks. This sensitivity was shown to grow for larger ensemble sizes. Reliability diagrams were far less sensitive in this respect. The resulting u-shaped rank histograms can easily be misinterpreted as indicating too little spread in the ensemble prediction system. To account for this effect when real observations are used to assess an ensemble prediction system, normally distributed noise based on the verifying observation error can be added to the ensemble members before the statistics are calculated. The method has been tested for the ECMWF ensemble forecasts of ocean waves and forecasts of the geopotential at 500 hPa. The EPS waves were compared with buoy observations from the Global Telecommunication System (GTS) for a period of almost 3 yr. When the buoy observations were taken as the true value, more than 25% of the observations appeared in the two extreme ranks for the day 3 forecast range. This number was reduced to less than 10% when observation errors were added to the ensemble members. Ensemble forecasts of the 500-hPa geopotential were verified against the ECMWF analysis. When analysis errors were neglected, the maximum number of outliers was more than 10% for most areas except for Europe, where the analysis errors are relatively smaller. Introducing noise to the ensemble members, based on estimates of analysis errors, reduced the number of outliers, particularly in the short range, where a peak around day 1 more or less vanished.
Abstract
From time to time atmospheric flows become organized and form coherent long-lived structures. Such structures could be propagating, quasi-stationary, or recur in place. We investigate the ability of principal components analysis (PCA) and archetypal analysis (AA) to identify long-lived events, excluding propagating forms. Our analysis is carried out on the Southern Hemisphere midtropospheric flow represented by geopotential height at 500 hPa (Z 500). The leading basis patterns of Z 500 for PCA and AA are similar and describe structures representing (or similar to) the southern annular mode (SAM) and Pacific–South American (PSA) pattern. Long-lived events are identified here from sequences of 8 days or longer where the same basis pattern dominates for PCA or AA. AA identifies more long-lived events than PCA using this approach. The most commonly occurring long-lived event for both AA and PCA is the annular SAM-like pattern. The second most commonly occurring event is the PSA-like Pacific wave train for both AA and PCA. For AA the flow at any given time is approximated as weighted contributions from each basis pattern, which lends itself to metrics for discriminating among basis patterns. These show that the longest long-lived events are in general better expressed than shorter events. Case studies of long-lived events featuring a blocking structure and an annular structure show that both PCA and AA can identify and discriminate the dominant basis pattern that most closely resembles the flow event.
Abstract
From time to time atmospheric flows become organized and form coherent long-lived structures. Such structures could be propagating, quasi-stationary, or recur in place. We investigate the ability of principal components analysis (PCA) and archetypal analysis (AA) to identify long-lived events, excluding propagating forms. Our analysis is carried out on the Southern Hemisphere midtropospheric flow represented by geopotential height at 500 hPa (Z 500). The leading basis patterns of Z 500 for PCA and AA are similar and describe structures representing (or similar to) the southern annular mode (SAM) and Pacific–South American (PSA) pattern. Long-lived events are identified here from sequences of 8 days or longer where the same basis pattern dominates for PCA or AA. AA identifies more long-lived events than PCA using this approach. The most commonly occurring long-lived event for both AA and PCA is the annular SAM-like pattern. The second most commonly occurring event is the PSA-like Pacific wave train for both AA and PCA. For AA the flow at any given time is approximated as weighted contributions from each basis pattern, which lends itself to metrics for discriminating among basis patterns. These show that the longest long-lived events are in general better expressed than shorter events. Case studies of long-lived events featuring a blocking structure and an annular structure show that both PCA and AA can identify and discriminate the dominant basis pattern that most closely resembles the flow event.
Abstract
Precipitation forecasts from five global numerical weather prediction (NWP) models are verified against rain gauge observations using the new stable equitable error in probability space (SEEPS) score. It is based on a 3 × 3 contingency table and measures the ability of a forecast to discriminate between “dry,” “light precipitation,” and “heavy precipitation.” In SEEPS, the threshold defining the boundary between the light and heavy categories varies systematically with precipitation climate. Results obtained for SEEPS are compared to those of more well-known scores, and are broken down with regard to individual contributions from the contingency table. It is found that differences in skill between the models are consistent for different scores, but are small compared to seasonal and geographical variations, which themselves can be largely ascribed to the varying prevalence of deep convection. Differences between the tropics and extratropics are quite pronounced. SEEPS scores at forecast day 1 in the tropics are similar to those at day 6 in the extratropics. It is found that the model ranking is robust with respect to choices in the score computation. The issue of observation representativeness is addressed using a “quasi-perfect model” approach. Results suggest that just under one-half of the current forecast error at day 1 in the extratropics can be attributed to the fact that gridbox values are verified against point observations.
Abstract
Precipitation forecasts from five global numerical weather prediction (NWP) models are verified against rain gauge observations using the new stable equitable error in probability space (SEEPS) score. It is based on a 3 × 3 contingency table and measures the ability of a forecast to discriminate between “dry,” “light precipitation,” and “heavy precipitation.” In SEEPS, the threshold defining the boundary between the light and heavy categories varies systematically with precipitation climate. Results obtained for SEEPS are compared to those of more well-known scores, and are broken down with regard to individual contributions from the contingency table. It is found that differences in skill between the models are consistent for different scores, but are small compared to seasonal and geographical variations, which themselves can be largely ascribed to the varying prevalence of deep convection. Differences between the tropics and extratropics are quite pronounced. SEEPS scores at forecast day 1 in the tropics are similar to those at day 6 in the extratropics. It is found that the model ranking is robust with respect to choices in the score computation. The issue of observation representativeness is addressed using a “quasi-perfect model” approach. Results suggest that just under one-half of the current forecast error at day 1 in the extratropics can be attributed to the fact that gridbox values are verified against point observations.
Abstract
Spatial variability of precipitation is analyzed to characterize to what extent precipitation observed at a single location is representative of precipitation over a larger area. Characterization of precipitation representativeness is made in probabilistic terms using a parametric approach, namely, by fitting a censored shifted gamma distribution to observation measurements. Parameters are estimated and analyzed for independent precipitation datasets, among which one is based on high-density gauge measurements. The results of this analysis serve as a basis for accounting for representativeness error in an ensemble verification process. Uncertainty associated with the scale mismatch between forecast and observation is accounted for by applying a perturbed-ensemble approach before the computation of scores. Verification results reveal a large impact of representativeness error on precipitation forecast reliability and skill estimates. The parametric model and estimated coefficients presented in this study could be used directly for forecast postprocessing to partly compensate for the limitation of any modeling system in terms of precipitation subgrid-scale variability.
Abstract
Spatial variability of precipitation is analyzed to characterize to what extent precipitation observed at a single location is representative of precipitation over a larger area. Characterization of precipitation representativeness is made in probabilistic terms using a parametric approach, namely, by fitting a censored shifted gamma distribution to observation measurements. Parameters are estimated and analyzed for independent precipitation datasets, among which one is based on high-density gauge measurements. The results of this analysis serve as a basis for accounting for representativeness error in an ensemble verification process. Uncertainty associated with the scale mismatch between forecast and observation is accounted for by applying a perturbed-ensemble approach before the computation of scores. Verification results reveal a large impact of representativeness error on precipitation forecast reliability and skill estimates. The parametric model and estimated coefficients presented in this study could be used directly for forecast postprocessing to partly compensate for the limitation of any modeling system in terms of precipitation subgrid-scale variability.
Abstract
A supercell produced a nearly tornadic vortex during an intercept by the Second Verification of the Origins of Rotation in Tornadoes Experiment on 26 May 2010. Using observations from two mobile radars performing dual-Doppler scans, a five-probe mobile mesonet, and a proximity sounding, factors that prevented this vortex from strengthening into a significant tornado are examined. Mobile mesonet observations indicate that portions of the supercell outflow possessed excessive negative buoyancy, likely owing in part to low boundary layer relative humidity, as indicated by a high environmental lifted condensation level. Comparisons to a tornadic supercell suggest that the Prospect Valley storm had enough far-field circulation to produce a significant tornado, but was unable to converge this circulation to a sufficiently small radius. Trajectories suggest that the weak convergence might be due to the low-level mesocyclone ingesting parcels with considerable crosswise vorticity from the near-storm environment, which has been found to contribute to less steady and weaker low-level updrafts in supercell simulations. Yet another factor that likely contributed to the weak low-level circulation was the inability of parcels rich in streamwise vorticity from the forward-flank precipitation region to reach the low-level mesocyclone, likely owing to an unfavorable pressure gradient force field. In light of these results, we suggest that future research should continue focusing on the role of internal, storm-scale processes in tornadogenesis, especially in marginal environments.
Abstract
A supercell produced a nearly tornadic vortex during an intercept by the Second Verification of the Origins of Rotation in Tornadoes Experiment on 26 May 2010. Using observations from two mobile radars performing dual-Doppler scans, a five-probe mobile mesonet, and a proximity sounding, factors that prevented this vortex from strengthening into a significant tornado are examined. Mobile mesonet observations indicate that portions of the supercell outflow possessed excessive negative buoyancy, likely owing in part to low boundary layer relative humidity, as indicated by a high environmental lifted condensation level. Comparisons to a tornadic supercell suggest that the Prospect Valley storm had enough far-field circulation to produce a significant tornado, but was unable to converge this circulation to a sufficiently small radius. Trajectories suggest that the weak convergence might be due to the low-level mesocyclone ingesting parcels with considerable crosswise vorticity from the near-storm environment, which has been found to contribute to less steady and weaker low-level updrafts in supercell simulations. Yet another factor that likely contributed to the weak low-level circulation was the inability of parcels rich in streamwise vorticity from the forward-flank precipitation region to reach the low-level mesocyclone, likely owing to an unfavorable pressure gradient force field. In light of these results, we suggest that future research should continue focusing on the role of internal, storm-scale processes in tornadogenesis, especially in marginal environments.
Abstract
A dryline and misocyclones have been simulated in a cloud-resolving model by applying specified initial and time-dependent lateral boundary conditions obtained from analyses of the 22 May 2002 International H2O Project (IHOP_2002) dataset. The initial and lateral boundary conditions were obtained from a combination of the time–spaced Lagrangian analyses for temperature and moisture with horizontal velocities from multiple-Doppler wind syntheses. The simulated dryline, horizontal dry-convective rolls (HCRs) and open cells (OCCs), misocyclones, and cumulus clouds are similar to the corresponding observed features. The misocyclones move northward at nearly the mean boundary layer (BL) wind speed, rotate dryline gradients owing to their circulations, and move the local dryline eastward via their passage. Cumuli develop along a secondary dryline, along HCR and OCC segments between the primary and secondary drylines, along HCR and OCC segments that have moved over the dryline, and within the dryline updraft. After the initial shearing instability develops, misocyclogenesis proceeds from tilting and stretching of vorticity by the persistent secondary dryline circulation. The resulting misocyclone evolution is discussed.
Abstract
A dryline and misocyclones have been simulated in a cloud-resolving model by applying specified initial and time-dependent lateral boundary conditions obtained from analyses of the 22 May 2002 International H2O Project (IHOP_2002) dataset. The initial and lateral boundary conditions were obtained from a combination of the time–spaced Lagrangian analyses for temperature and moisture with horizontal velocities from multiple-Doppler wind syntheses. The simulated dryline, horizontal dry-convective rolls (HCRs) and open cells (OCCs), misocyclones, and cumulus clouds are similar to the corresponding observed features. The misocyclones move northward at nearly the mean boundary layer (BL) wind speed, rotate dryline gradients owing to their circulations, and move the local dryline eastward via their passage. Cumuli develop along a secondary dryline, along HCR and OCC segments between the primary and secondary drylines, along HCR and OCC segments that have moved over the dryline, and within the dryline updraft. After the initial shearing instability develops, misocyclogenesis proceeds from tilting and stretching of vorticity by the persistent secondary dryline circulation. The resulting misocyclone evolution is discussed.
Abstract
While chaos ensures that probabilistic weather forecasts cannot always be “sharp,” it is important for users and developers that they are reliable. For example, they should not be overconfident or underconfident. The “spread–error” relationship is often used as a first-order assessment of the reliability of ensemble weather forecasts. This states that the ensemble standard deviation (a measure of forecast uncertainty) should match the root-mean-square error on the ensemble mean (when averaged over a sufficient number of forecast start dates). It is shown here that this relationship is now largely satisfied at the European Centre for Medium-Range Weather Forecasts (ECMWF) for ensemble forecasts of the midlatitude, midtropospheric flow out to lead times of at least 10 days when averaged over all flow situations throughout the year. This study proposes a practical framework for continued improvement in the reliability (and skill) of such forecasts. This involves the diagnosis of flow-dependent deficiencies in short-range (∼12 h) reliability for a range of synoptic-scale flow types and the prioritization of modeling research to address these deficiencies. The approach is demonstrated for a previously identified flow type, a trough over the Rockies with warm, moist air ahead. The mesoscale convective systems that can ensue are difficult to predict and, by perturbing the jet stream, are thought to lead to deterministic forecast “busts” for Europe several days later. The results here suggest that jet stream spread is insufficient during this flow type, and thus unreliable. This is likely to mean that the uncertain forecasts for Europe may, nevertheless, still be overconfident.
Abstract
While chaos ensures that probabilistic weather forecasts cannot always be “sharp,” it is important for users and developers that they are reliable. For example, they should not be overconfident or underconfident. The “spread–error” relationship is often used as a first-order assessment of the reliability of ensemble weather forecasts. This states that the ensemble standard deviation (a measure of forecast uncertainty) should match the root-mean-square error on the ensemble mean (when averaged over a sufficient number of forecast start dates). It is shown here that this relationship is now largely satisfied at the European Centre for Medium-Range Weather Forecasts (ECMWF) for ensemble forecasts of the midlatitude, midtropospheric flow out to lead times of at least 10 days when averaged over all flow situations throughout the year. This study proposes a practical framework for continued improvement in the reliability (and skill) of such forecasts. This involves the diagnosis of flow-dependent deficiencies in short-range (∼12 h) reliability for a range of synoptic-scale flow types and the prioritization of modeling research to address these deficiencies. The approach is demonstrated for a previously identified flow type, a trough over the Rockies with warm, moist air ahead. The mesoscale convective systems that can ensue are difficult to predict and, by perturbing the jet stream, are thought to lead to deterministic forecast “busts” for Europe several days later. The results here suggest that jet stream spread is insufficient during this flow type, and thus unreliable. This is likely to mean that the uncertain forecasts for Europe may, nevertheless, still be overconfident.
