Observation Definitions and Their Implications in Machine Learning–Based Predictions of Excessive Rainfall

Aaron J. Hill aDepartment of Atmospheric Science, Colorado State University, Fort Collins, Colorado
bSchool of Meteorology, University of Oklahoma, Norman, Oklahoma

Search for other papers by Aaron J. Hill in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0002-0337-8864
,
Russ S. Schumacher aDepartment of Atmospheric Science, Colorado State University, Fort Collins, Colorado

Search for other papers by Russ S. Schumacher in
Current site
Google Scholar
PubMed
Close
, and
Mitchell L. Green cDepartment of Earth and Atmospheric Sciences, Central Michigan University, Mount Pleasant, Michigan

Search for other papers by Mitchell L. Green in
Current site
Google Scholar
PubMed
Close
Restricted access

We are aware of a technical issue preventing figures and tables from showing in some newly published articles in the full-text HTML view.
While we are resolving the problem, please use the online PDF version of these articles to view figures and tables.

Abstract

The implications of definitions of excessive rainfall observations on machine learning model forecast skill are assessed using the Colorado State University Machine Learning Probabilities (CSU-MLP) forecast system. The CSU-MLP uses historical observations along with reforecasts from a global ensemble to train random forests to probabilistically predict excessive rainfall events. Here, random forest models are trained using two distinct rainfall datasets, one that is composed of fixed-frequency (FF) average recurrence intervals exceedances and flash flood reports and the other a compilation of flooding and rainfall proxies [Unified Flood Verification System (UFVS)]. Both models generate 1–3-day forecasts and are evaluated against a climatological baseline to characterize their overall skill as a function of lead time, season, and region. Model comparisons suggest that regional frequencies in excessive rainfall observations contribute to when and where the ML models issue forecasts and subsequently their skill and reliability. Additionally, the spatiotemporal distribution of observations has implications for ML model training requirements, notably, how long of an observational record is needed to obtain skillful forecasts. Experiments reveal that shorter-trained UFVS-based models can be as skillful as longer-trained FF-based models. In essence, the UFVS dataset exhibits a more robust characterization of excessive rainfall and impacts, and machine learning models trained on more representative datasets of meteorological hazards may not require as extensive training to generate skillful forecasts.

Significance Statement

Machine learning (ML) models have shown significant promise in recent years when used to predict high-impact weather hazards. Here, we explore two similarly trained ML models tasked with predicting excessive rainfall, but they use datasets that define excessive rainfall differently. We explore how definitions of excessive rainfall, for example, an amount of rainfall that would be expected to fall once per year, contribute to forecast skill through where these observations are reported. Generally, we find that the two models have substantial skill relative to climatology out to 3 days, but skill varies by geographical region and season in part because of the distribution of observations geographically. These results suggest that careful attention should be paid to how ML models are trained to predict meteorological hazards like excessive rainfall.

© 2024 American Meteorological Society. This published article is licensed under the terms of the default AMS reuse license. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Aaron J. Hill, ahill@ou.edu

Abstract

The implications of definitions of excessive rainfall observations on machine learning model forecast skill are assessed using the Colorado State University Machine Learning Probabilities (CSU-MLP) forecast system. The CSU-MLP uses historical observations along with reforecasts from a global ensemble to train random forests to probabilistically predict excessive rainfall events. Here, random forest models are trained using two distinct rainfall datasets, one that is composed of fixed-frequency (FF) average recurrence intervals exceedances and flash flood reports and the other a compilation of flooding and rainfall proxies [Unified Flood Verification System (UFVS)]. Both models generate 1–3-day forecasts and are evaluated against a climatological baseline to characterize their overall skill as a function of lead time, season, and region. Model comparisons suggest that regional frequencies in excessive rainfall observations contribute to when and where the ML models issue forecasts and subsequently their skill and reliability. Additionally, the spatiotemporal distribution of observations has implications for ML model training requirements, notably, how long of an observational record is needed to obtain skillful forecasts. Experiments reveal that shorter-trained UFVS-based models can be as skillful as longer-trained FF-based models. In essence, the UFVS dataset exhibits a more robust characterization of excessive rainfall and impacts, and machine learning models trained on more representative datasets of meteorological hazards may not require as extensive training to generate skillful forecasts.

Significance Statement

Machine learning (ML) models have shown significant promise in recent years when used to predict high-impact weather hazards. Here, we explore two similarly trained ML models tasked with predicting excessive rainfall, but they use datasets that define excessive rainfall differently. We explore how definitions of excessive rainfall, for example, an amount of rainfall that would be expected to fall once per year, contribute to forecast skill through where these observations are reported. Generally, we find that the two models have substantial skill relative to climatology out to 3 days, but skill varies by geographical region and season in part because of the distribution of observations geographically. These results suggest that careful attention should be paid to how ML models are trained to predict meteorological hazards like excessive rainfall.

© 2024 American Meteorological Society. This published article is licensed under the terms of the default AMS reuse license. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Aaron J. Hill, ahill@ou.edu
Save
  • Agresti, A., and B. A. Coull, 1998: Approximate is better than “exact” for interval estimation of binomial proportions. Amer. Stat., 52, 119126, https://doi.org/10.2307/2685469.

    • Search Google Scholar
    • Export Citation
  • Arcodia, M., E. A. Barnes, P. J. Durack, P. Keys, and J. Rocha, 2024: Sea surface salinity provides subseasonal predictability for forecasts of opportunity of U.S. summertime precipitation. Authorea, 26 pp., https://doi.org/10.22541/au.170602087.73596352/v1.

  • Breiman, L., 2001: Random forests. Mach. Learn., 45, 532, https://doi.org/10.1023/A:1010933404324.

  • Chase, R. J., D. R. Harrison, A. Burke, G. M. Lackmann, and A. McGovern, 2022: A machine learning tutorial for operational meteorology. Part I: Traditional machine learning. Wea. Forecasting, 37, 15091529, https://doi.org/10.1175/WAF-D-22-0070.1.

    • Search Google Scholar
    • Export Citation
  • Chase, R. J., D. R. Harrison, G. M. Lackmann, and A. McGovern, 2023: A machine learning tutorial for operational meteorology. Part II: Neural networks and deep learning. Wea. Forecasting, 38, 12711293, https://doi.org/10.1175/WAF-D-22-0187.1.

    • Search Google Scholar
    • Export Citation
  • Clark, R. A., J. J. Gourley, Z. L. Flamig, Y. Hong, and E. Clark, 2014: CONUS-wide evaluation of National Weather Service flash flood guidance products. Wea. Forecasting, 29, 377392, https://doi.org/10.1175/WAF-D-12-00124.1.

    • Search Google Scholar
    • Export Citation
  • Doswell, C. A., H. E. Brooks, and R. A. Maddox, 1996: Flash flood forecasting: An ingredients-based methodology. Wea. Forecasting, 11, 560581, https://doi.org/10.1175/1520-0434(1996)011<0560:FFFAIB>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Erickson, M. J., J. S. Kastman, B. Albright, S. Perfater, J. A. Nelson, R. S. Schumacher, and G. R. Herman, 2019: Verification results from the 2017 HMT–WPC Flash Flood and Intense Rainfall Experiment. J. Appl. Meteor. Climatol., 58, 25912604, https://doi.org/10.1175/JAMC-D-19-0097.1.

    • Search Google Scholar
    • Export Citation
  • Erickson, M. J., B. Albright, and J. A. Nelson, 2021: Verifying and redefining the Weather Prediction Center’s Excessive Rainfall Outlook forecast product. Wea. Forecasting, 36, 325340, https://doi.org/10.1175/WAF-D-20-0020.1.

    • Search Google Scholar
    • Export Citation
  • Fritsch, J. M., and R. E. Carbone, 2004: Improving quantitative precipitation forecasts in the warm season: A USWRP research and development strategy. Bull. Amer. Meteor. Soc., 85, 955966, https://doi.org/10.1175/BAMS-85-7-955.

    • Search Google Scholar
    • Export Citation
  • Gagne, D. J., II, A. McGovern, and M. Xue, 2014: Machine learning enhancement of storm-scale ensemble probabilistic quantitative precipitation forecasts. Wea. Forecasting, 29, 10241043, https://doi.org/10.1175/WAF-D-13-00108.1.

    • Search Google Scholar
    • Export Citation
  • Ghazvinian, M., Y. Zhang, D.-J. Seo, M. He, and N. Fernando, 2021: A novel hybrid artificial neural network—Parametric scheme for postprocessing medium-range precipitation forecasts. Adv. Water Resour., 151, 103907, https://doi.org/10.1016/j.advwatres.2021.103907.

    • Search Google Scholar
    • Export Citation
  • Gourley, J. J., and H. Vergara, 2021: Comments on “flash flood verification: Pondering precipitation proxies”. J. Hydrometeor., 22, 739747, https://doi.org/10.1175/JHM-D-20-0215.1.

    • Search Google Scholar
    • Export Citation
  • Gourley, J. J., and Coauthors, 2017: The FLASH project: Improving the tools for flash flood monitoring and prediction across the United States. Bull. Amer. Meteor. Soc., 98, 361372, https://doi.org/10.1175/BAMS-D-15-00247.1.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., and Coauthors, 2022: The reanalysis for the Global Ensemble Forecast System, version 12. Mon. Wea. Rev., 150, 5979, https://doi.org/10.1175/MWR-D-21-0023.1.

    • Search Google Scholar
    • Export Citation
  • Herman, G. R., and R. S. Schumacher, 2018a: Flash flood verification: Pondering precipitation proxies. J. Hydrometeor., 19, 17531776, https://doi.org/10.1175/JHM-D-18-0092.1.

    • Search Google Scholar
    • Export Citation
  • Herman, G. R., and R. S. Schumacher, 2018b: Money doesn’t grow on trees, but forecasts do: Forecasting extreme precipitation with random forests. Mon. Wea. Rev., 146, 15711600, https://doi.org/10.1175/MWR-D-17-0250.1.

    • Search Google Scholar
    • Export Citation
  • Hill, A. J., and R. S. Schumacher, 2021: Forecasting excessive rainfall with random forests and a deterministic convection-allowing model. Wea. Forecasting, 36, 16931711, https://doi.org/10.1175/WAF-D-21-0026.1.

    • Search Google Scholar
    • Export Citation
  • Hill, A. J., and R. S. Schumacher, 2024: Medium-range excessive rainfall prediction with machine learning. EGU General Assembly 2024, Vienna, Austria, EGU24-11708, https://doi.org/10.5194/egusphere-egu24-11708.

    • Search Google Scholar
    • Export Citation
  • Hill, A. J., G. R. Herman, and R. S. Schumacher, 2020: Forecasting severe weather with random forests. Mon. Wea. Rev., 148, 21352161, https://doi.org/10.1175/MWR-D-19-0344.1.

    • Search Google Scholar
    • Export Citation
  • Hill, A. J., R. S. Schumacher, and I. L. Jirak, 2023: A new paradigm for medium-range severe weather forecasts: Probabilistic random forest–based predictions. Wea. Forecasting, 38, 251272, https://doi.org/10.1175/WAF-D-22-0143.1.

    • Search Google Scholar
    • Export Citation
  • Hou, D., and Coauthors, 2014: Climatology-calibrated precipitation analysis at fine scales: Statistical adjustment of Stage IV toward CPC gauge-based analysis. J. Hydrometeor., 15, 25422557, https://doi.org/10.1175/JHM-D-11-0140.1.

    • Search Google Scholar
    • Export Citation
  • Hugeback, K. K., K. J. Franz, and W. A. Gallus Jr., 2023: Short-term ensemble streamflow prediction using spatially shifted QPF informed by displacement errors. J. Hydrometeor., 24, 2134, https://doi.org/10.1175/JHM-D-21-0252.1.

    • Search Google Scholar
    • Export Citation
  • James, E. P., and R. S. Schumacher, 2024: Precipitation proxies for flash flooding: A seven-year analysis over the contiguous United States. J. Hydrometeor., 25, 13231344, https://doi.org/10.1175/JHM-D-23-0203.1.

    • Search Google Scholar
    • Export Citation
  • James, E. P., R. S. Schumacher, and A. J. Hill, 2023: Testing random forests for prediction of excessive rainfall based on the high-resolution rapid refresh (HRRR). 32nd Conf. on Weather Analysis and Forecasting, Madison, WI, Amer. Meteor. Soc., 16.3, https://ams.confex.com/ams/WAFNWPMS/meetingapp.cgi/Paper/425347.

  • Li, X., and Coauthors, 2021: Evaluating precipitation, streamflow, and inundation forecasting skills during extreme weather events: A case study for an urban watershed. J. Hydrol., 603, 127126, https://doi.org/10.1016/j.jhydrol.2021.127126.

    • Search Google Scholar
    • Export Citation
  • Lin, Y., and K. E. Mitchell, 2005: The NCEP Stage II/IV hourly precipitation analyses: Development and applications. 19th Conf. Hydrology, San Diego, CA, Amer. Meteor. Soc., 102, http://ams.confex.com/ams/Annual2005/techprogram/paper_83847.htm.

  • Loken, E. D., A. J. Clark, A. McGovern, M. Flora, and K. Knopfmeier, 2019: Postprocessing next-day ensemble probabilistic precipitation forecasts using random forests. Wea. Forecasting, 34, 20172044, https://doi.org/10.1175/WAF-D-19-0109.1.

    • Search Google Scholar
    • Export Citation
  • McGovern, A., R. J. Chase, M. Flora, D. J. Gagne II, R. Lagerquist, C. K. Potvin, N. Snook, and E. Loken, 2023: A review of machine learning for convective weather. Artif. Intell. Earth Syst., 2, e220077, https://doi.org/10.1175/AIES-D-22-0077.1.

    • Search Google Scholar
    • Export Citation
  • McGovern, A., A. Bostrom, M. McGraw, R. J. Chase, D. J. Gagne II, I. Ebert-Uphoff, K. D. Musgrave, and A. Schumacher, 2024: Identifying and categorizing bias in AI/ML for Earth sciences. Bull. Amer. Meteor. Soc., 105, E567E583, https://doi.org/10.1175/BAMS-D-23-0196.1.

    • Search Google Scholar
    • Export Citation
  • NOAA, 2020: Precipitation prediction grand challenge strategy. NOAA Tech. Rep., 44 pp.

  • NOAA Office of Water Prediction, 2021: Current NWS precipitation frequency documents. Accessed 17 October 2023, https://www.weather.gov/owp/hdsc_currentpf.

  • Pedregosa, F., and Coauthors, 2011: Scikit-learn: Machine learning in Python. J. Mach. Learn. Res., 12, 28252830.

  • Ralph, F. M., E. Sukovich, D. Reynolds, M. Dettinger, S. Weagle, W. Clark, and P. J. Neiman, 2010: Assessment of extreme quantitative precipitation forecasts and development of regional extreme event thresholds using data from HMT-2006 and COOP observers. J. Hydrometeor., 11, 12861304, https://doi.org/10.1175/2010JHM1232.1.

    • Search Google Scholar
    • Export Citation
  • Refaeilzadeh, P., L. Tang, and H. Liu, 2009: Cross-validation. Encyclopedia of Database Systems, Springer, 532–538, https://doi.org/10.1007/978-0-387-39940-9_565.

  • Schmidt, J. A., A. J. Anderson, and J. H. Paul, 2007: Spatially-variable, physically-derived flash flood guidance. 21st Conf. on Hydrology, San Antonio, TX, Amer. Meteor. Soc., 6B.2, https://ams.confex.com/ams/pdfpapers/120022.pdf.

  • Schumacher, R. S., and G. R. Herman, 2021: Reply to “comments on ‘flash flood verification: Pondering precipitation proxies’”. J. Hydrometeor., 22, 749752, https://doi.org/10.1175/JHM-D-20-0275.1.

    • Search Google Scholar
    • Export Citation
  • Schumacher, R. S., A. J. Hill, M. Klein, J. A. Nelson, M. J. Erickson, S. M. Trojniak, and G. R. Herman, 2021: From random forests to flood forecasts: A research to operations success story. Bull. Amer. Meteor. Soc., 102, E1742E1755, https://doi.org/10.1175/BAMS-D-20-0186.1.

    • Search Google Scholar
    • Export Citation
  • Sharma, K., R. Ashrit, R. Bhatla, A. K. Mitra, G. R. Iyengar, and E. N. Rajagopal, 2017: Skill of predicting heavy rainfall over India: Improvement in recent years using UKMO Global Model. Pure Appl. Geophys., 174, 42414250, https://doi.org/10.1007/s00024-017-1640-9.

    • Search Google Scholar
    • Export Citation
  • Sukovich, E. M., F. M. Ralph, F. E. Barthold, D. W. Reynolds, and D. R. Novak, 2014: Extreme quantitative precipitation forecast performance at the Weather Prediction Center from 2001 to 2011. Wea. Forecasting, 29, 894911, https://doi.org/10.1175/WAF-D-13-00061.1.

    • Search Google Scholar
    • Export Citation
  • Ukkonen, P., and A. Mäkelä, 2019: Evaluation of machine learning classifiers for predicting deep convection. J. Adv. Model. Earth Syst., 11, 17841802, https://doi.org/10.1029/2018MS001561.

    • Search Google Scholar
    • Export Citation
  • Viterbo, F., and Coauthors, 2020: A multiscale, hydrometeorological forecast evaluation of National Water Model forecasts of the May 2018 Ellicott City, Maryland, flood. J. Hydrometeor., 21, 475499, https://doi.org/10.1175/JHM-D-19-0125.1.

    • Search Google Scholar
    • Export Citation
  • Williams, J. K., D. A. Ahijevych, C. J. Kessinger, T. R. Saxen, M. Steiner, and S. Dettling, 2008: A machine learning approach to finding weather regimes and skillful predictor combinations for short-term storm forecasting. Sixth Conf. on Artificial Intelligence Applications to Environmental Science and 13th Conf. on Aviation, Range and Aerospace Meteorology, New Orleans, LA, Amer. Meteor. Soc., J1.4, https://ams.confex.com/ams/88Annual/webprogram/Paper135663.html.

  • Yue, H., and M. Gebremichael, 2020: Evaluation of High-Resolution Rapid Refresh (HRRR) forecasts for extreme precipitation. Environ. Res. Commun., 2, 065004, https://doi.org/10.1088/2515-7620/ab9002.

    • Search Google Scholar
    • Export Citation
  • Zhou, X., and Coauthors, 2022: The development of the NCEP Global Ensemble Forecast System version 12. Wea. Forecasting, 37, 10691084, https://doi.org/10.1175/WAF-D-21-0112.1.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 6850 6850 1011
Full Text Views 211 211 34
PDF Downloads 182 182 20