• Barthold, F. E., T. E. Workoff, B. A. Cosgrove, J. J. Gourley, D. R. Novak, and K. M. Mahoney, 2015: Improving flash flood forecasts: The HMT-WPC flash flood and intense rainfall experiment. Bull. Amer. Meteor. Soc., 96, 18591866, https://doi.org/10.1175/BAMS-D-14-00201.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Baxter, M. A., G. M. Lackmann, K. M. Mahoney, T. E. Workoff, and T. M. Hamill, 2014: Verification of quantitative precipitation reforecasts over the southeastern United States. Wea. Forecasting, 29, 11991207, https://doi.org/10.1175/WAF-D-14-00055.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brennan, M. J., J. Clark, and M. Klein, 2008: Verification of quantitative precipitation forecast guidance from NWP models and the Hydrometeorological Prediction Center for 2005–2007 tropical cyclones with continental U.S. rainfall impacts. 28th Conf. on Hurricanes and Tropical Meteorology, Orlando, FL, Amer. Meteor. Soc., P2H.9, https://ams.confex.com/ams/pdfpapers/138022.pdf.

  • Bullock, R., B. Brown, and T. Fowler, 2016: Method for object-based diagnostic evaluation. NCAR Tech. Note NCAR/TN-532+STR, 84 pp., https://doi.org/10.5065/D61V5CBS.

    • Crossref
    • Export Citation
  • Clark, P., N. Roberts, H. Lean, S. P. Ballard, and C. Charlton-Perez, 2016: Convection-permitting models: A step-change in rainfall forecasting. Meteor. Appl., 23, 165181, https://doi.org/10.1002/met.1538.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Clark, R. A., J. J. Gourley, Z. L. Flamig, Y. Hong, and E. Clark, 2014: CONUS-wide evaluation of National Weather Service flash flood guidance products. Wea. Forecasting, 29, 377392, https://doi.org/10.1175/WAF-D-12-00124.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cookson-Hills, P., D. J. Kirshbaum, M. Surcel, J. G. Doyle, L. Fillion, D. Jacques, and S.-J. Baek, 2017: Verification of 24-h quantitative precipitation forecasts over the Pacific Northwest from a high resolution ensemble Kalman filter system. Wea. Forecasting, 32, 11851208, https://doi.org/10.1175/WAF-D-16-0180.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cooley, D. S., 1978: The excessive rainfall potential outlook. NOAA/NWS Tech. Procedures Bull. 239, 7 pp.

  • Cosgrove, B., and C. Klymmer, 2016: The National Water Model. NOAA, accessed 29 July 2019, https://water.noaa.gov/about/nwm.

  • Cuo, L., T. C. Pagano, and Q. J. Wang, 2011: A review of quantitative precipitation forecasts and their use in short- to medium-range streamflow forecasting. J. Hydrometeor., 12, 713728, https://doi.org/10.1175/2011JHM1347.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Erickson, M. J., and J. A. Nelson Jr., 2018: Verifying, calibrating, and redefining the excessive rainfall outlook at the weather prediction center. Eighth Conf. on Transition of Research to Operations, Austin, TX, Amer. Meteor. Soc., 7.5, https://ams.confex.com/ams/98Annual/webprogram/Paper327404.html.

  • Erickson, M. J., J. S. Kastman, B. Albright, S. Perfater, J. A. Nelson, R. S. Schumacher, and G. R. Herman, 2019: Verification results from the 2017 HMT-WPC flash flood and intense rainfall experiment. J. Appl. Meteor. Climatol., 58, 25912604, https://doi.org/10.1175/JAMC-D-19-0097.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fritsch, J. M., and R. E. Carbone, 2004: Improving quantitative precipitation forecasts in the warm season: A USWRP research and development strategy. Bull. Amer. Meteor. Soc., 85, 955964, https://doi.org/10.1175/BAMS-85-7-955.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gourley, J. J., and Coauthors, 2013: A unified flash flood database across the United States. Bull. Amer. Meteor. Soc., 94, 799805, https://doi.org/10.1175/BAMS-D-12-00198.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gourley, J. J., and Coauthors, 2017: The FLASH project: Improving the tools for flash flood monitoring and prediction across the United States. Bull. Amer. Meteor. Soc., 98, 361372, https://doi.org/10.1175/BAMS-D-15-00247.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Guirguis, K., A. Gershunov, T. Shulgina, R. E. S. Clemesha, and F. Martin Ralph, 2019: Atmospheric rivers impacting Northern California and their modulation by a variable climate. Climate Dyn., 52, 65696583, https://doi.org/10.1007/s00382-018-4532-5.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Halley Gotway, J., and Coauthors, 2018: Model Evaluation Tools version 8.0 (METv8.0): User’s guide. Developmental Testbed Center Rep., 431 pp., https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf.

  • Hamill, T. M., G. T. Bates, J. S. Whitaker, D. R. Murray, M. Fiorino, T. J. Galarneau Jr., Y. Zhu, and W. Lapenta, 2013: NOAA’s second-generation global medium-range ensemble reforecast dataset. Bull. Amer. Meteor. Soc., 94, 15531565, https://doi.org/10.1175/BAMS-D-12-00014.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., E. Engle, D. Myrick, M. Peroutka, C. Finan, and M. Scheuerer, 2017: The U.S. national blend of models for statistical postprocessing of probability of precipitation and deterministic precipitation amount. Mon. Wea. Rev., 145, 34413463, https://doi.org/10.1175/MWR-D-16-0331.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Herman, G. R., and R. S. Schumacher, 2018a: Money doesn’t grow on trees, but forecasts do: Forecasting extreme precipitation with random forests. Mon. Wea. Rev., 146, 15711600, https://doi.org/10.1175/MWR-D-17-0250.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Herman, G. R., and R. S. Schumacher, 2018b: “Dendrology” in numerical weather prediction: What random forests and logistic regression tell us about forecasting extreme precipitation. Mon. Wea. Rev., 146, 17851812, https://doi.org/10.1175/MWR-D-17-0307.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hitchens, N. M., H. E. Brooks, and M. P. Kay, 2013: Objective limits on forecasting skill of rare events. Wea. Forecasting, 28, 525534, https://doi.org/10.1175/WAF-D-12-00113.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Iyer, E. R., A. J. Clark, M. Xue, and F. Kong, 2016: A comparison of 36–60-h precipitation forecasts from convection-allowing and convection-parameterizing ensembles. Wea. Forecasting, 31, 647661, https://doi.org/10.1175/WAF-D-15-0143.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kendall, M. G., 1975: Rank Correlation Measures. Charles Griffin, 202 pp.

  • Lamers, A., 2019: Pre-monsoon season summary of important information. WPC-WFO-RFC Southwest Monsoon Collaboration Call.

  • Lincoln, W. S., and R. F. L. Thomason, 2018: A preliminary look at using rainfall average recurrence interval to characterize flash flood events for real-time warning forecasting. J. Oper. Meteor., 6, 1322, https://doi.org/10.15191/nwajom.2018.0602.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Luitel, B., G. Villarini, and G. A. Vecchi, 2018: Verification of the skill of numerical weather prediction models in forecasting rainfall from U.S. landfalling tropical cyclones. J. Hydrol., 556, 10261037, https://doi.org/10.1016/j.jhydrol.2016.09.019.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ma, S., C. Chen, H. He, D. Wu, and C. Zhang, 2018: Assessing the skill of convection-allowing ensemble forecasts of precipitation by optimization of spatial-temporal neighborhoods. Science, 9, 43, https://doi.org/10.3390/ATMOS9020043.

    • Search Google Scholar
    • Export Citation
  • Mann, H. B., 1945: Nonparametric tests against trend. Econometrica, 13, 245259, https://doi.org/10.2307/1907187.

  • Marchok, T., R. Rogers, and R. Tuleya, 2007: Validation schemes for tropical cyclone quantitative precipitation forecasts: Evaluation of operational models for U.S. landfall cases. Wea. Forecasting, 22, 726746, https://doi.org/10.1175/WAF1024.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • National Weather Service, 2017: Summary of natural hazard statistics for 2017 in the United States. NOAA, 3 pp., https://www.weather.gov/media/hazstat/sum17.pdf.

  • NCEP, 2019: Ensemble situational awareness table. Accessed 29 July 2019, https://satable.ncep.noaa.gov/naefs/.

  • Nelson, B., O. Prat, D. Seo, and E. Habib, 2016: Assessment and implications of NCEP stage IV quantitative precipitation estimates for product comparisons. Wea. Forecasting, 31, 371394, https://doi.org/10.1175/WAF-D-14-00112.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Newman, K., and Coauthors, 2019: Model Evaluation Tools version 8.1 (METv8.1): User’s guide. Developmental Testbed Center Rep., 437 pp., https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.1.pdf.

  • Novak, D. R., C. Bailey, K. F. Brill, P. Burke, W. A. Hogsett, R. Rausch, and M. Schichtel, 2014: Precipitation and temperature forecast performance at the Weather Prediction Center. Wea. Forecasting, 29, 489504, https://doi.org/10.1175/WAF-D-13-00066.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Perfater, S., and B. Albright, 2017: 2017 flash flood and intense rainfall experiment. NOAA/NWS/WPC, Weather Prediction Center Rep., 95 pp., https://www.wpc.ncep.noaa.gov/hmt/2017_FFaIR_final_report.pdf.

  • Perica, S., and Coauthors, 2013: Precipitation-Frequency Atlas of the United States. Vol. 9, NOAA Atlas NESDIS 14, 171 pp.

  • Pyle, M. E., and G. S. Manikin, 2018: The High-Resolution Ensemble Forecast (HREF) System and applications for aviation forecasting. Sixth Aviation, Range, and Aerospace Meteorology Special Symp., Austin, TX, Amer. Meteor. Soc., 2.3, https://ams.confex.com/ams/98Annual/webprogram/Paper334406.html.

  • Scheuerer, M., and T. M. Hamill, 2015: Statistical postprocessing of ensemble precipitation forecasts by fitting censored, shifted gamma distributions. Mon. Wea. Rev., 143, 45784596, https://doi.org/10.1175/MWR-D-15-0061.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Schmidt, J. A., A. J. Anderson, and J. H. Paul, 2007: Spatially-variable, physically-derived flash flood guidance. 21st Conf. on Hydrology, San Antonio, TX, Amer. Meteor. Soc., 6B.2, https://ams.confex.com/ams/pdfpapers/120022.pdf.

  • Sharma, S., and Coauthors, 2017: Eastern U.S. verification of ensemble precipitation forecasts. Wea. Forecasting, 32, 117139, https://doi.org/10.1175/WAF-D-16-0094.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sukovich, E. M., F. M. Ralph, F. E. Barthold, D. W. Reynolds, and D. R. Novak, 2014: Extreme quantitative precipitation forecast performance at the Weather Prediction Center from 2001 to 2011. Wea. Forecasting, 29, 894911, https://doi.org/10.1175/WAF-D-13-00061.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Water Resources Services, 2017: NWS annual flood loss summary reports to U.S. Army Corps of Engineers. 2 pp., https://www.weather.gov/media/water/WY17%20Flood%20Deaths%20and%20Direct%20Damagesv2.pdf.

  • Wick, G. A., P. J. Neiman, and F. M. Ralph, 2013: Description and validation of an automated objective technique for identification and characterization of integrated water vapor signature of atmospheric rivers IEEE Trans. Geosci. Remote Sens., 51, 21662176, https://doi.org/10.1109/TGRS.2012.2211024.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2011: Statistical Methods in the Atmospheric Sciences. 3rd ed. International Geophysics Series, Vol. 100, Academic Press, 704 pp.

  • WPC, 2013: About the excessive rainfall forecast. Accessed 29 July 2019, https://www.wpc.ncep.noaa.gov/qpf/about_excess.shtml.

  • WPC, 2016: What’s new at WPC. Accessed 29 July 2019, https://www.wpc.ncep.noaa.gov/html/whats_new.shtml.

  • WPC, 2017: WPC verification page. Accessed 29 July 2019, https://www.wpc.ncep.noaa.gov/images/hpcvrf/WPCmdlsd110yrly.gif.

  • WPC, 2019: Product information. Accessed 29 July 2019, https://www.wpc.ncep.noaa.gov/html/fam2.shtml#excessrain.

  • Yang, F., and V. Tallapragada, 2018: Implementation and evaluation of the NOAA next generation global prediction system with FV3 dynamical core and advanced physics. Eighth Conf. on Transition of Research to Operations, Austin, TX, Amer. Meteor. Soc., 1.4, https://ams.confex.com/ams/98Annual/webprogram/Paper329963.html.

  • Zheng, M., E. K. Chang, B. A. Colle, Y. Luo, and Y. Zhu, 2017: Applying fuzzy clustering to a multimodel ensemble for U.S. East Coast winter storms: Scenario identification and forecast verification. Wea. Forecasting, 32, 881903, https://doi.org/10.1175/WAF-D-16-0112.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 486 0 0
Full Text Views 563 305 20
PDF Downloads 556 256 28

Verifying and Redefining the Weather Prediction Center’s Excessive Rainfall Outlook Forecast Product

Michael J. Erickson Cooperative Institute for Research in Environmental Sciences, University of Colorado Boulder, Boulder, Colorado
NOAA/NWS/NCEP/Weather Prediction Center, College Park, Maryland

Search for other papers by Michael J. Erickson in
Current site
Google Scholar
PubMed
Close
,
Benjamin Albright NOAA/NWS/NCEP/Weather Prediction Center, College Park, Maryland
Systems Research Group, Inc., College Park, Maryland

Search for other papers by Benjamin Albright in
Current site
Google Scholar
PubMed
Close
, and
James A. Nelson NOAA/NWS/NCEP/Weather Prediction Center, College Park, Maryland

Search for other papers by James A. Nelson in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

The Weather Prediction Center’s Excessive Rainfall Outlook (ERO) forecasts the probability of rainfall exceeding flash flood guidance within 40 km of a point. This study presents a comprehensive ERO verification between 2015 and 2019 using a combination of flooding observations and proxies. ERO spatial issuance frequency plots are developed to provide situational awareness for forecasters. Reliability of the ERO is assessed by computing fractional coverage of the verification within each probabilistic category. Probabilistic forecast skill is evaluated using the Brier skill score (BSS) and area under the relative operating characteristic (AUC). A “probabilistic observation” called practically perfect (PP) is developed and compared to the ERO as an additional measure of skill. The areal issuance frequency of the ERO varies spatially with the most abundant issuances spanning from the Gulf Coast to the Midwest and the Appalachians. ERO issuances occur most often in the summer and are associated with the Southwestern monsoon, mesoscale convective systems, and tropical cyclones. The ERO exhibits good reliability on average, although more recent trends suggest some ERO-defined probabilistic categories should be issued more frequently. AUC and BSS are useful bulk skill metrics, while verification against PP is useful in bulk and for shorter-term ERO evaluation. ERO forecasts are generally more skillful at shorter lead times in terms of AUC and BSS. There is no trend in ERO area size over 5 years, although ERO forecasts may be getting slightly more skillful in terms of critical success index when verified against the PP.

© 2021 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Michael Erickson, mjaerickson@gmail.com

Abstract

The Weather Prediction Center’s Excessive Rainfall Outlook (ERO) forecasts the probability of rainfall exceeding flash flood guidance within 40 km of a point. This study presents a comprehensive ERO verification between 2015 and 2019 using a combination of flooding observations and proxies. ERO spatial issuance frequency plots are developed to provide situational awareness for forecasters. Reliability of the ERO is assessed by computing fractional coverage of the verification within each probabilistic category. Probabilistic forecast skill is evaluated using the Brier skill score (BSS) and area under the relative operating characteristic (AUC). A “probabilistic observation” called practically perfect (PP) is developed and compared to the ERO as an additional measure of skill. The areal issuance frequency of the ERO varies spatially with the most abundant issuances spanning from the Gulf Coast to the Midwest and the Appalachians. ERO issuances occur most often in the summer and are associated with the Southwestern monsoon, mesoscale convective systems, and tropical cyclones. The ERO exhibits good reliability on average, although more recent trends suggest some ERO-defined probabilistic categories should be issued more frequently. AUC and BSS are useful bulk skill metrics, while verification against PP is useful in bulk and for shorter-term ERO evaluation. ERO forecasts are generally more skillful at shorter lead times in terms of AUC and BSS. There is no trend in ERO area size over 5 years, although ERO forecasts may be getting slightly more skillful in terms of critical success index when verified against the PP.

© 2021 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Michael Erickson, mjaerickson@gmail.com
Save