Object-Based Verification of Atmospheric River Predictions in the Northeast Pacific

Laurel L. DeHaan aCenter for Western Weather and Water Extremes, Scripps Institution of Oceanography, University of California, San Diego, La Jolla, California

Search for other papers by Laurel L. DeHaan in
Current site
Google Scholar
PubMed
Close
,
Andrew C. Martin bDepartment of Geography, Portland State University, Portland, Oregon

Search for other papers by Andrew C. Martin in
Current site
Google Scholar
PubMed
Close
,
Rachel R. Weihs aCenter for Western Weather and Water Extremes, Scripps Institution of Oceanography, University of California, San Diego, La Jolla, California

Search for other papers by Rachel R. Weihs in
Current site
Google Scholar
PubMed
Close
,
Luca Delle Monache aCenter for Western Weather and Water Extremes, Scripps Institution of Oceanography, University of California, San Diego, La Jolla, California

Search for other papers by Luca Delle Monache in
Current site
Google Scholar
PubMed
Close
, and
F. Martin Ralph aCenter for Western Weather and Water Extremes, Scripps Institution of Oceanography, University of California, San Diego, La Jolla, California

Search for other papers by F. Martin Ralph in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Accurate forecasts of atmospheric rivers (ARs) provide advance warning of flood and landslide hazards and greatly aid effective water management. It is, therefore, critical to evaluate the skill of AR forecasts in numerical weather prediction (NWP) models. A new verification framework is proposed that leverages freely available software and metrics previously used for different applications. Specifically, AR detection and statistics are computed for the first time using the Method for Object-Based Diagnostic Evaluation (MODE). In addition, the measure of effectiveness (MoE) is introduced as a new metric for understanding AR forecast skill in terms of size and location. The MoE provides a quantitative measure of the position of an entire forecast AR relative to observation, regardless of whether the AR is making landfall. In addition, the MoE can provide qualitative information about the evolution of a forecast by lead time, with implications about the predictability of an AR. We analyze AR forecast verification and skill using 11 years of cold-season forecasts from two NWP models: one global and one regional. Four different thresholds of integrated vapor transport (IVT) are used in the verification, revealing differences in forecast skill that are based on the strength of an AR. In addition to MoE, AR forecast skill is also addressed in terms of intensity error, landfall position error, and contingency-table metrics.

© 2021 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Laurel L. DeHaan, ldehaan@ucsd.edu

Abstract

Accurate forecasts of atmospheric rivers (ARs) provide advance warning of flood and landslide hazards and greatly aid effective water management. It is, therefore, critical to evaluate the skill of AR forecasts in numerical weather prediction (NWP) models. A new verification framework is proposed that leverages freely available software and metrics previously used for different applications. Specifically, AR detection and statistics are computed for the first time using the Method for Object-Based Diagnostic Evaluation (MODE). In addition, the measure of effectiveness (MoE) is introduced as a new metric for understanding AR forecast skill in terms of size and location. The MoE provides a quantitative measure of the position of an entire forecast AR relative to observation, regardless of whether the AR is making landfall. In addition, the MoE can provide qualitative information about the evolution of a forecast by lead time, with implications about the predictability of an AR. We analyze AR forecast verification and skill using 11 years of cold-season forecasts from two NWP models: one global and one regional. Four different thresholds of integrated vapor transport (IVT) are used in the verification, revealing differences in forecast skill that are based on the strength of an AR. In addition to MoE, AR forecast skill is also addressed in terms of intensity error, landfall position error, and contingency-table metrics.

© 2021 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Laurel L. DeHaan, ldehaan@ucsd.edu
Save
  • Bullock, R., B. Brown, and T. Fowler, 2016: Method for Object-Based Diagnostic Evaluation. NCAR Tech. Note NCAR/TN-532+STR, https://doi.org/10.5065/D61V5CBS.

    • Crossref
    • Export Citation
  • Cannon, F., C. W. Hecht, J. M. Cordeira, and F. M. Ralph, 2018: Synoptic and mesoscale forcing of Southern California extreme precipitation. J. Geophys. Res. Atmos., 123, 13 71413 730, https://doi.org/10.1029/2018JD029045.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cobb, A., L. Delle Monache, F. Cannon, and F. M. Ralph, 2021: Representation of dropsonde-observed atmospheric river conditions in reanalyses. Geophys. Res. Lett., https://doi.org/10.1029/2021GL093357, in press.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cordeira, J. M., F. M. Ralph, A. Martin, N. Gaggini, J. R. Spackman, P. J. Neiman, J. J. Rutz, and R. Pierce, 2017: Forecasting atmospheric rivers during CalWater 2015. Bull. Amer. Meteor. Soc., 98, 449459, https://doi.org/10.1175/BAMS-D-15-00245.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Corringham, T. W., F. M. Ralph, A. Gershunov, D. R. Cayan, C. A. Talbot, 2019: Atmospheric rivers drive flood damages in the western United States. Sci. Adv., 5, eaax4631, https://doi.org/10.1126/sciadv.aax4631.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Davis, C., A. Brown, and R. Bullock, 2006: Object-based verification of precipitation forecasts. Part I: Methodology and application to mesoscale rain areas. Mon. Wea. Rev., 134, 17721784, https://doi.org/10.1175/MWR3145.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Davis, C., A. Brown, R. Bullock, and J. Halley-Gotway, 2009: The Method for Object-Based Diagnostic Evaluation (MODE) applied to numerical forecasts from the 2005 NSSL/SPC spring program. Wea. Forecasting, 24, 12521267, https://doi.org/10.1175/2009WAF2222241.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • DeFlorio, M. J., D. E. Waliser, B. Guan, D. A. Lavers, F. M. Ralph, and F. Vitart, 2018: Global assessment of atmospheric river prediction skill. J. Hydrometeor., 19, 409426, https://doi.org/10.1175/JHM-D-17-0135.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Dettinger, M. D., 2013: Atmospheric rivers as drought busters on the U.S. West Coast. J. Hydrometeor., 14, 17211732, https://doi.org/10.1175/JHM-D-13-02.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Dettinger, M. D., F. M. Ralph, T. Das, P. J. Neiman, and D. R. Cayan, 2011: Atmospheric rivers, floods and the water resources of California. Water, 3, 445478, https://doi.org/10.3390/w3020445.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gallus, W. A., Jr, 2010: Application of object-based verification techniques to ensemble precipitation forecasts. Wea. Forecasting, 25, 144158, https://doi.org/10.1175/2009WAF2222274.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gelaro, R., and Coauthors, 2017: The Modern-Era Retrospective Analysis for Research and Applications, version 2 (MERRA-2). J. Climate, 30, 54195454, https://doi.org/10.1175/JCLI-D-16-0758.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gershunov, A., T. Shulgina, F. M. Ralph, D. A. Lavers, and J. J. Rutz, 2017: Assessing the climate-scale variability of atmospheric rivers affecting western North America. Geophys. Res. Lett., 44, 79007908, https://doi.org/10.1002/2017GL074175.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gilleland, E., D. Ahijevych, B. G. Brown, B. Casati, and E. E. Ebert, 2009: Intercomparison of spatial forecast verification methods. Wea. Forecasting, 24, 14161430, https://doi.org/10.1175/2009WAF2222269.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Guan, B., and D. E. Waliser, 2015: Detection of atmospheric rivers: Evaluation and application of an algorithm for global studies. J. Geophys. Res. Atmos., 120, 12 51412 535, https://doi.org/10.1002/2015JD024257.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hecht, C. W., and J. M. Cordeira, 2017: Characterizing the influence of atmospheric river orientation and intensity on precipitation distributions over North Coastal California. Geophys. Res. Lett., 44, 90489058, https://doi.org/10.1002/2017GL074179.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hersbach, H., and Coauthors, 2020: The ERA5 global reanalysis. Quart. J. Roy. Meteor. Soc., 146, 19992049, https://doi.org/10.1002/qj.3803.

  • Jolliffe, I. T., and D. B. Stephenson, Eds., 2012: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. 2nd ed. John Wiley and Sons, 296 pp.

    • Search Google Scholar
    • Export Citation
  • Kanamitsu, M., and L. DeHaan, 2011: The added value index: A new metric to quantify the added value of regional models. J. Geophys. Res., 116, D11106, https://doi.org/10.1029/2011JD015597.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Martin, A., F. M. Ralph, R. Demirdjian, L. DeHaan, R. Weihs, J. Helly, D. Reynolds, and S. Iacobellis, 2018: Evaluation of atmospheric river predictions by the WRF model using aircraft and regional mesonet observations of orographic precipitation and its forcing. J. Hydrometeor., 19, 10971113, https://doi.org/10.1175/JHM-D-17-0098.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mass, C. F., D. Ovens, K. Westrick, and B. A. Colle, 2002: Does increasing horizontal resolution produce more skillful forecasts? Bull. Amer. Meteor. Soc., 83, 407430, https://doi.org/10.1175/1520-0477(2002)083<0407:DIHRPM>2.3.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mundhenk, B. D., E. A. Barnes, E. D. Maloney, and K. M. Nardi, 2016: Modulation of atmospheric rivers near Alaska and the U.S. West Coast by northeast Pacific height anomalies. J. Geophys. Res. Atmos., 121, 12 75112 765, https://doi.org/10.1002/2016JD025350.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Nardi, K. M., E. A. Barnes, and F. M. Ralph, 2018: Assessment of numerical weather prediction model reforecasts of the occurrence, intensity, and location of atmospheric rivers along the West Coast of North America. Mon. Wea. Rev., 146, 33433362, https://doi.org/10.1175/MWR-D-18-0060.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Nayak, M. A., G. Villarini, and D. A. Lavers, 2014: On the skill of numerical weather prediction models to forecast atmospheric rivers over the central United States. Geophys. Res. Lett., 41, 43544362, https://doi.org/10.1002/2014GL060299.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ralph, F. M., E. Sukovich, D. Reynolds, M. Dettinger, S. Weagle, W. Clark, and P. J. Neiman, 2010: Assessment of extreme quantitative precipitation forecasts and development of regional extreme event thresholds using data from HMT-2006 and COOP observers. J. Hydrometeor., 11, 12861304, https://doi.org/10.1175/2010JHM1232.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ralph, F. M., T. Coleman, P. J. Neiman, R. J. Zamora, and M. D. Dettinger, 2013a: Observed impacts of duration and seasonality of atmospheric-river landfalls on soil moisture and runoff in coastal Northern California. J. Hydrometeor., 14, 443459, https://doi.org/10.1175/JHM-D-12-076.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ralph, F. M., and Coauthors, 2013b: The emergence of weather-related test beds linking research and forecasting operations. Bull. Amer. Meteor. Soc., 94, 11871211, https://doi.org/10.1175/BAMS-D-12-00080.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ralph, F. M., and Coauthors, 2017: Dropsonde observations of total integrated water vapor transport within North Pacific atmospheric rivers. J. Hydrometeor., 18, 25772596, https://doi.org/10.1175/JHM-D-17-0036.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ralph, F. M., M. D. Dettinger, M. M. Cairns, T. J. Galarneau, and J. Eylander, 2018: Defining “atmospheric river”: How the Glossary of Meteorology helped resolve a debate. Bull. Amer. Meteor. Soc., 99, 837839, https://doi.org/10.1175/BAMS-D-17-0157.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ralph, F. M., J. J. Rutz, J. M. Cordeira, M. Dettinger, M. Anderson, D. Reynolds, L. J. Schick, and C. Smallcomb, 2019: A scale to characterize the strength and impacts of atmospheric rivers. Bull. Amer. Meteor. Soc., 100, 269289, https://doi.org/10.1175/BAMS-D-18-0023.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Reid, K. J., A. D. King, T. P. Lane, and E. Short, 2020: The sensitivity of atmospheric river identification to integrated water vapor transport threshold, resolution, and regridding method. J. Geophys. Res. Atmos., 125, e2020JD032897, https://doi.org/10.1029/2020JD032897.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Roebber, P. J., 2009: Visualizing multiple measures of forecast quality. Wea. Forecasting, 24, 601608, https://doi.org/10.1175/2008WAF2222159.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rutz, J. J., and Coauthors, 2019: The Atmospheric River Tracking Method Intercomparison Project (ARTMIP): Quantifying uncertainties in atmospheric river climatology. J. Geophys. Res. Atmos., 124, 13 77713 802, https://doi.org/10.1029/2019JD030936.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Shields, C. A., and Coauthors, 2018: Atmospheric River Tracking Method Intercomparison Project (ARTMIP): Project goals and experimental design. Geosci. Model Dev., 11, 24552474, https://doi.org/10.5194/gmd-11-2455-2018.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Skamarock, W. C., and Coauthors, 2008: A description of the Advanced Research WRF version 3. NCAR Tech. Note NCAR/TN-475+STR, 113 pp., https://doi.org/10.5065/D68S4MVH.

    • Crossref
    • Export Citation
  • Skok, G., J. Tribbia, J. Rakovec, and B. Brown, 2009: Object-based analysis of satellite-derived precipitation systems over the low- and midlatitude Pacific Ocean. Mon. Wea. Rev., 137, 31963218, https://doi.org/10.1175/2009MWR2900.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sukovich, E. M., F. M. Ralph, F. E. Barthold, D. W. Reynolds, and D. R. Novak, 2014: Extreme quantitative precipitation forecast performance at the weather prediction center from 2001 to 2011. Wea. Forecasting, 29, 894911, https://doi.org/10.1175/WAF-D-13-00061.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Warner, S., N. Platt, and J. F. Heagy, 2004: User-oriented two-dimensional measure of effectiveness for the evaluation of transport and dispersion models. J. Appl. Meteor., 43, 5873, https://doi.org/10.1175/1520-0450(2004)043<0058:UTMOEF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Weihs, R., D. Reynolds, R. Hartman, S. Sellars, D. Kozlowski, and F. Ralph, 2020: Assessing quantitative precipitation and inflow forecast skill for potential forecast informed reservoir operations for Lake Mendocino. ERDC/CHL Contract Rep., 42 pp.

  • Wick, G., P. Neiman, F. Ralph, and T. Hamill, 2013: Evaluation of forecasts of the water vapor signature of atmospheric rivers in operational numerical weather prediction models. Wea. Forecasting, 28, 13371352, https://doi.org/10.1175/WAF-D-13-00025.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2019: Statistical Methods in the Atmospheric Sciences. 4th ed. Elsevier, 840 pp.

  • Zhou, X., Y. Zhu, D. Hou, Y. Luo, J. Peng, and R. Wobus, 2017: Performance of the new NCEP global ensemble forecast system in a parallel experiment. Wea. Forecasting, 32, 19892004, https://doi.org/10.1175/WAF-D-17-0023.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 252 0 0
Full Text Views 353 199 13
PDF Downloads 332 150 9