Toward the Improvement of High-Impact Probabilistic Forecasts with a Sensitivity-Based Convective-Scale Ensemble Subsetting Technique

Austin Coleman Texas Tech University, Lubbock, Texas

Search for other papers by Austin Coleman in
Current site
Google Scholar
PubMed
Close
and
Brian Ancell Texas Tech University, Lubbock, Texas

Search for other papers by Brian Ancell in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Ensemble sensitivity analysis (ESA) is a useful and computationally inexpensive tool for analyzing how features in the flow at early forecast times affect different relevant forecast features later in the forecast. Given the frequency of observations measured between model initialization times that remain unused, ensemble sensitivity may be used to increase predictability and forecast accuracy through an objective ensemble subsetting technique. This technique identifies ensemble members with the smallest errors in regions of high sensitivity to produce a smaller, more accurate ensemble subset. Ensemble subsets can significantly reduce synoptic-scale forecast errors, but applying this strategy to convective-scale forecasts presents additional challenges. Objective verification of the sensitivity-based ensemble subsetting technique is conducted for ensemble forecasts of 2–5-km updraft helicity (UH) and simulated reflectivity. Many degrees of freedom are varied to identify the lead times, subset sizes, forecast thresholds, and atmospheric predictors that provide most forecast benefit. Subsets vastly reduce error of UH forecasts in an idealized framework but tend to degrade fractions skill scores and reliability in a real-world framework. Results reveal this discrepancy is a result of verifying probabilistic UH forecasts with storm-report-based observations, which effectively dampens technique performance. The potential of ensemble subsetting and likely other postprocessing techniques is limited by tuning UH forecasts to predict severe reports. Additional diagnostic ideas to improve postprocessing tool optimization for convection-allowing models are discussed.

Current affiliation: Texas Tech University, Lubbock, Texas.

© 2020 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Austin Coleman, austin.coleman@ttu.edu

Abstract

Ensemble sensitivity analysis (ESA) is a useful and computationally inexpensive tool for analyzing how features in the flow at early forecast times affect different relevant forecast features later in the forecast. Given the frequency of observations measured between model initialization times that remain unused, ensemble sensitivity may be used to increase predictability and forecast accuracy through an objective ensemble subsetting technique. This technique identifies ensemble members with the smallest errors in regions of high sensitivity to produce a smaller, more accurate ensemble subset. Ensemble subsets can significantly reduce synoptic-scale forecast errors, but applying this strategy to convective-scale forecasts presents additional challenges. Objective verification of the sensitivity-based ensemble subsetting technique is conducted for ensemble forecasts of 2–5-km updraft helicity (UH) and simulated reflectivity. Many degrees of freedom are varied to identify the lead times, subset sizes, forecast thresholds, and atmospheric predictors that provide most forecast benefit. Subsets vastly reduce error of UH forecasts in an idealized framework but tend to degrade fractions skill scores and reliability in a real-world framework. Results reveal this discrepancy is a result of verifying probabilistic UH forecasts with storm-report-based observations, which effectively dampens technique performance. The potential of ensemble subsetting and likely other postprocessing techniques is limited by tuning UH forecasts to predict severe reports. Additional diagnostic ideas to improve postprocessing tool optimization for convection-allowing models are discussed.

Current affiliation: Texas Tech University, Lubbock, Texas.

© 2020 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Austin Coleman, austin.coleman@ttu.edu
Save
  • Ancell, B. C., 2016: Improving high-impact forecasts through sensitivity-based ensemble subsets: Demonstration and initial tests. Wea. Forecasting, 31, 10191036, https://doi.org/10.1175/WAF-D-15-0121.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ancell, B. C., and G. J. Hakim, 2007: Comparing adjoint- and ensemble-sensitivity analysis with applications to observation targeting. Mon. Wea. Rev., 135, 41174134, https://doi.org/10.1175/2007MWR1904.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ancell, B. C., A. Bogusz, M. J. Lauridsen, and C. J. Nauert, 2018: Seeding chaos: The dire consequences of numerical noise in NWP perturbation experiments. Bull. Amer. Meteor. Soc., 99, 615628, https://doi.org/10.1175/BAMS-D-17-0129.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Anderson, J., T. Hoar, K. Raeder, H. Liu, N. Collins, R. Torn, and A. Avellano, 2009: The Data Assimilation Research Testbed: A community facility. Bull. Amer. Meteor. Soc., 90, 12831296, https://doi.org/10.1175/2009BAMS2618.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Atger, F., 2003: Spatial and interannual variability of the reliability of ensemble-based probabilistic forecasts: Consequences for calibration. Mon. Wea. Rev., 131, 15091523, https://doi.org/10.1175//1520-0493(2003)131<1509:SAIVOT>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bednarczyk, C. N., and B. C. Ancell, 2015: Ensemble sensitivity analysis applied to a southern plains convective event. Mon. Wea. Rev., 143, 230249, https://doi.org/10.1175/MWR-D-13-00321.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78, 13, https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bröcker, J., and L. A. Smith, 2007: Increasing the reliability of reliability diagrams. Wea. Forecasting, 22, 651661, https://doi.org/10.1175/WAF993.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brooks, H., and M. Kay, 1998: Objective limits on forecasting skill of rare events. Preprints, 19th Conf. on Severe Local Storms, Minneapolis, MN, Amer. Meteor. Soc., 552–555.

  • Bryan, G. H., J. C. Wyngaard, and J. M. Fritsch, 2003: Resolution requirements for the simulation of deep moist convection. Mon. Wea. Rev., 131, 23942416, https://doi.org/10.1175/1520-0493(2003)131<2394:RRFTSO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Clark, A. J., and Coauthors, 2011: Probabilistic precipitation forecast skill as a function of ensemble size and spatial scale in a convection-allowing ensemble. Mon. Wea. Rev., 139, 14101418, https://doi.org/10.1175/2010MWR3624.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Clark, A. J., J. S. Kain, P. T. Marsh, J. Correia, M. Xue, and F. Kong, 2012: Forecasting tornado pathlengths using a three-dimensional object identification algorithm applied to convection-allowing forecasts. Wea. Forecasting, 27, 10901113, https://doi.org/10.1175/WAF-D-11-00147.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Clark, A. J., and Coauthors, 2020: A real-time, simulated forecasting experiment for advancing the prediction of hazardous convective weather. Bull. Amer. Meteor. Soc., 101, E2022E2024, https://doi.org/10.1175/BAMS-D-19-0298.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Coniglio, M. C., G. S. Romine, D. D. Turner, and R. D. Torn, 2019: Impacts of targeted AERI and Doppler lidar wind retrievals on short-term forecasts of the initiation and early evolution of thunderstorms. Mon. Wea. Rev., 147, 11491170, https://doi.org/10.1175/MWR-D-18-0351.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Done, J., C. A. Davis, and M. Weisman, 2004: The next generation of NWP: Explicit forecasts of convection using the Weather Research and Forecasting (WRF) model. Atmos. Sci. Lett., 5, 110117, https://doi.org/10.1002/asl.72.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duda, J. D., and W. A. Gallus, 2010: Spring and summer Midwestern severe weather reports in supercells compared to other morphologies. Wea. Forecasting, 25, 190206, https://doi.org/10.1175/2009WAF2222338.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gallo, B. T., A. J. Clark, and S. R. Dembek, 2016: Forecasting tornadoes using convection-permitting ensembles. Wea. Forecasting, 31, 273295, https://doi.org/10.1175/WAF-D-15-0134.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hakim, G. J., and R. D. Torn, 2008: Ensemble synoptic analysis. Synoptic–Dynamic Meteorology and Weather Analysis and Forecasting: A Tribute to Fred Sanders, Meteor. Monogr., No. 55, Amer. Meteor. Soc., 147–161, https://doi.org/10.1007/978-0-933876-68-2_7.

    • Crossref
    • Export Citation
  • Hill, A. J., C. C. Weiss, and B. C. Ancell, 2016: Ensemble sensitivity analysis for mesoscale forecasts of dryline convection initiation. Mon. Wea. Rev., 144, 41614182, https://doi.org/10.1175/MWR-D-15-0338.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hitchens, N. M., H. E. Brooks, and M. P. Kay, 2013: Objective limits on forecasting skill of rare events. Wea. Forecasting, 28, 525534, https://doi.org/10.1175/WAF-D-12-00113.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Homeyer, C., and K. Bowman, 2017: Algorithm description document for version 3.1 of the three-dimensional gridded NEXRAD WSR-88D radar (GridRad) dataset. Tech. Rep., 23 pp.

  • Kain, J. S., S. J. Weiss, J. J. Levit, M. E. Baldwin, and D. R. Bright, 2006: Examination of convection-allowing configurations of the WRF Model for the prediction of severe convective weather: The SPC/NSSL spring program 2004. Wea. Forecasting, 21, 167181, https://doi.org/10.1175/WAF906.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kain, J. S., and Coauthors, 2008: Some practical considerations regarding horizontal resolution in the first generation of operational convection-allowing NWP. Wea. Forecasting, 23, 931952, https://doi.org/10.1175/WAF2007106.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kain, J. S., S. R. Dembek, S. J. Weiss, J. L. Case, J. J. Levit, and R. A. Sobash, 2010: Extracting unique information from high-resolution forecast models: Monitoring selected fields and phenomena every time step. Wea. Forecasting, 25, 15361542, https://doi.org/10.1175/2010WAF2222430.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kerr, C. A., D. J. Stensrud, and X. Wang, 2019: Diagnosing convective dependencies on near-storm environments using ensemble sensitivity analyses. Mon. Wea. Rev., 147, 495517, https://doi.org/10.1175/MWR-D-18-0140.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Krocak, M. J., and H. E. Brooks, 2020: An analysis of subdaily severe thunderstorm probabilities for the United States. Wea. Forecasting, 35, 107112, https://doi.org/10.1175/WAF-D-19-0145.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Roberts, N. M., 2005: An investigation of the ability of a storm scale configuration of the Met Office NWP model to predict flood producing rainfall. Met Office Tech. Rep. 455, 80 pp., https://library.metoffice.gov.uk/portal/Default/en-GB/RecordView/Index/251914.

  • Roberts, N. M., and H. W. Lean, 2008: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev., 136, 7897, https://doi.org/10.1175/2007MWR2123.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Schwartz, C. S., and R. A. Sobash, 2017: Generating probabilistic forecasts from convection-allowing ensembles using neighborhood approaches: A review and recommendations. Mon. Wea. Rev., 145, 33973418, https://doi.org/10.1175/MWR-D-16-0400.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Schwartz, C. S., and Coauthors, 2010: Toward improved convection-allowing ensembles: Model physics sensitivities and optimizing probabilistic guidance with small ensemble membership. Wea. Forecasting, 25, 263280, https://doi.org/10.1175/2009WAF2222267.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Skamarock, W. C., and Coauthors, 2008: A description of the Advanced Research WRF version 3. NCAR Tech. Note NCAR/TN-475+STR, 113 pp., https://doi.org/10.5065/D68S4MVH.

    • Crossref
    • Export Citation
  • Skinner, P. S., and Coauthors, 2018: Object-based verification of a prototype Warn-on-Forecast System. Wea. Forecasting, 33, 12251250, https://doi.org/10.1175/WAF-D-18-0020.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Smith, N. H., and B. C. Ancell, 2017: Ensemble sensitivity analysis of wind ramp events with applications to observation targeting. Mon. Wea. Rev., 145, 25052522, https://doi.org/10.1175/MWR-D-16-0306.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Snider, C. R., 1977: A look at Michigan tornado statistics. Mon. Wea. Rev., 105, 13411342, https://doi.org/10.1175/1520-0493(1977)105<1341:ALAMTS>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sobash, R. A., and J. S. Kain, 2017: Seasonal variations in severe weather forecast skill in an experimental convection-allowing model. Wea. Forecasting, 32, 18851902, https://doi.org/10.1175/WAF-D-17-0043.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sobash, R. A., J. S. Kain, D. R. Bright, A. R. Dean, M. C. Coniglio, and S. J. Weiss, 2011: Probabilistic forecast guidance for severe thunderstorms based on the identification of extreme phenomena in convection-allowing model forecasts. Wea. Forecasting, 26, 714728, https://doi.org/10.1175/WAF-D-10-05046.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sobash, R. A., C. S. Schwartz, G. S. Romine, K. R. Fossell, and M. L. Weisman, 2016: Severe weather prediction using storm surrogates from an ensemble forecasting system. Wea. Forecasting, 31, 255271, https://doi.org/10.1175/WAF-D-15-0138.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sobash, R. A., C. S. Schwartz, G. S. Romine, and M. L. Weisman, 2019: Next-day prediction of tornadoes using convection-allowing models with 1-km horizontal grid spacing. Wea. Forecasting, 34, 11171135, https://doi.org/10.1175/WAF-D-19-0044.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Starzec, M., G. L. Mullendore, and P. A. Kucera, 2018: Using radar reflectivity to evaluate the vertical structure of forecast convection. J. Appl. Meteor. Climatol., 57, 28352849, https://doi.org/10.1175/JAMC-D-18-0116.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Weisman, M. L., W. C. Skamarock, and J. B. Klemp, 1997: The resolution dependence of explicitly modeled convective systems. Mon. Wea. Rev., 125, 527548, https://doi.org/10.1175/1520-0493(1997)125<0527:TRDOEM>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 239 0 0
Full Text Views 3374 2587 176
PDF Downloads 724 149 13