Editorial Type:
Article
Article Type:
Research Article

Verification of Precipitation Forecasts from NCEP’s Short-Range Ensemble Forecast (SREF) System with Reference to Ensemble Streamflow Prediction Using Lumped Hydrologic Models

James D. Brown NOAA/National Weather Service, Office of Hydrologic Development, Silver Spring, Maryland, and University Corporation for Atmospheric Research, Boulder, Colorado

Search for other papers by James D. Brown in
Current site
Google Scholar
PubMed Close
,
Dong-Jun Seo Department of Civil Engineering, The University of Texas at Arlington, Arlington, Texas

Search for other papers by Dong-Jun Seo in
Current site
Google Scholar
PubMed Close
, and
Jun Du NOAA/NWS/NCEP/Environmental Modeling Center, Camp Springs, Maryland

Search for other papers by Jun Du in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Jun 2012
DOI:
https://doi.org/10.1175/JHM-D-11-036.1
Page(s):
808–836
Restricted access

Abstract

Precipitation forecasts from the Short-Range Ensemble Forecast (SREF) system of the National Centers for Environmental Prediction (NCEP) are verified for the period April 2006–August 2010. Verification is conducted for 10–20 hydrologic basins in each of the following: the middle Atlantic, the southern plains, the windward slopes of the Sierra Nevada, and the foothills of the Cascade Range in the Pacific Northwest. Mean areal precipitation is verified conditionally upon forecast lead time, amount of precipitation, season, forecast valid time, and accumulation period. The stationary block bootstrap is used to quantify the sampling uncertainties of the verification metrics. In general, the forecasts are more skillful for moderate precipitation amounts than either light or heavy precipitation. This originates from a threshold-dependent conditional bias in the ensemble mean forecast. Specifically, the forecasts overestimate low observed precipitation and underestimate high precipitation (a type-II conditional bias). Also, the forecast probabilities are generally overconfident (a type-I conditional bias), except for basins in the southern plains, where forecasts of moderate to high precipitation are reliable. Depending on location, different types of bias correction may be needed. Overall, the northwest basins show the greatest potential for statistical postprocessing, particularly during the cool season, when the type-I conditional bias and correlations are both high. The basins of the middle Atlantic and southern plains show less potential for statistical postprocessing, as the type-II conditional bias is larger and the correlations are weaker. In the Sierra Nevada, the greatest benefits of statistical postprocessing should be expected for light precipitation, specifically during the warm season, when the type-I conditional bias is large and the correlations are strong.

Corresponding author address: James D. Brown, NOAA/National Weather Service, Office of Hydrologic Development, 1325 East–West Highway, Silver Spring, MD 20910. E-mail: james.d.brown@noaa.gov

Abstract

Precipitation forecasts from the Short-Range Ensemble Forecast (SREF) system of the National Centers for Environmental Prediction (NCEP) are verified for the period April 2006–August 2010. Verification is conducted for 10–20 hydrologic basins in each of the following: the middle Atlantic, the southern plains, the windward slopes of the Sierra Nevada, and the foothills of the Cascade Range in the Pacific Northwest. Mean areal precipitation is verified conditionally upon forecast lead time, amount of precipitation, season, forecast valid time, and accumulation period. The stationary block bootstrap is used to quantify the sampling uncertainties of the verification metrics. In general, the forecasts are more skillful for moderate precipitation amounts than either light or heavy precipitation. This originates from a threshold-dependent conditional bias in the ensemble mean forecast. Specifically, the forecasts overestimate low observed precipitation and underestimate high precipitation (a type-II conditional bias). Also, the forecast probabilities are generally overconfident (a type-I conditional bias), except for basins in the southern plains, where forecasts of moderate to high precipitation are reliable. Depending on location, different types of bias correction may be needed. Overall, the northwest basins show the greatest potential for statistical postprocessing, particularly during the cool season, when the type-I conditional bias and correlations are both high. The basins of the middle Atlantic and southern plains show less potential for statistical postprocessing, as the type-II conditional bias is larger and the correlations are weaker. In the Sierra Nevada, the greatest benefits of statistical postprocessing should be expected for light precipitation, specifically during the warm season, when the type-I conditional bias is large and the correlations are strong.

Corresponding author address: James D. Brown, NOAA/National Weather Service, Office of Hydrologic Development, 1325 East–West Highway, Silver Spring, MD 20910. E-mail: james.d.brown@noaa.gov
Save
Cover Journal of Hydrometeorology
Journal of Hydrometeorology

  • Bradley, A. A., Schwartz S. S. , and Hashino T. , 2004: Distributions-oriented verification of ensemble streamflow predictions. J. Hydrometeor., 5, 532–545.

    • Search Google Scholar
    • Export Citation

  • Brier, G., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78, 1–3.

  • Brown, J. D., and Heuvelink G. , 2005: Assessing uncertainty propagation through physically based models of soil water flow and solute transport. The Encyclopedia of Hydrological Sciences, M. Anderson, Ed., John Wiley and Sons, 1181–1195.

  • Brown, J. D., and Seo D.-J. , 2010: A nonparametric postprocessor for bias correction of hydrometeorological and hydrologic ensemble forecasts. J. Hydrometeor., 11, 642–665.

    • Search Google Scholar
    • Export Citation

  • Brown, J. D., Demargne J. , Seo D.-J. , and Liu Y. , 2010: The Ensemble Verification System (EVS): A software tool for verifying ensemble forecasts of hydrometeorological and hydrologic variables at discrete locations. Environ. Modell. Software, 25, 854–872.

    • Search Google Scholar
    • Export Citation

  • Brussolo, E., Von Hardenberg J. , Ferraris L. , Rebora N. , and Provenzale A. , 2008: Verification of quantitative precipitation forecasts via stochastic downscaling. J. Hydrometeor., 9, 1084–1094.

    • Search Google Scholar
    • Export Citation

  • Buerger, G., Reusser D. , and Kneis D. , 2009: Early flood warnings from empirical (expanded) downscaling of the full ECMWF Ensemble Prediction System. Water Resour. Res., 45, W10443, doi:10.1029/2009WR007779.

    • Search Google Scholar
    • Export Citation

  • Buizza, R., Miller M. , and Palmer T. N. , 1999: Stochastic representation of model uncertainty in the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc., 125, 2887–2908.

    • Search Google Scholar
    • Export Citation

  • Charles, M. E., and Colle B. A. , 2009: Verification of extratropical cyclones within the NCEP operational models. Part II: The Short-Range Ensemble Forecast system. Wea. Forecasting, 24, 1191–1214.

    • Search Google Scholar
    • Export Citation

  • Clark, A. J., Gallus W. A. Jr., Xue M. , and Kong F. , 2009: A comparison of precipitation forecast skill between small convection-allowing and large convection-parameterizing ensembles. Wea. Forecasting, 24, 1121–1140.

    • Search Google Scholar
    • Export Citation

  • Clark, A. J., Gallus W. A. Jr., Xue M. , and Kong F. , 2010: Growth of spread in convection-allowing and convection-parameterizing ensembles. Wea. Forecasting, 25, 594–612.

    • Search Google Scholar
    • Export Citation

  • Clark, A. J., and Coauthors, 2011: Probabilistic precipitation forecast skill as a function of ensemble size and spatial scale in a convection-allowing ensemble. Mon. Wea. Rev., 139, 1410–1418.

    • Search Google Scholar
    • Export Citation

  • de Elia, R., and Laprise R. , 2003: Distribution-oriented verification of limited-area model forecasts in a perfect-model framework. Mon. Wea. Rev., 131, 2492–2509.

    • Search Google Scholar
    • Export Citation

  • Demargne, J., Brown J. , Liu Y. , Seo D.-J. , Wu L. , Toth Z. , and Zhu Y. , 2010: Diagnostic verification of hydrometeorological and hydrologic ensembles. Atmos. Sci. Lett., 11, 114–122.

    • Search Google Scholar
    • Export Citation

  • Du, J., and Tracton M. , 2001: Implementation of a real-time short-range ensemble forecasting system at NCEP: An update. Preprints, Ninth Conf. on Mesoscale Processes, Ft. Lauderdale, FL, Amer. Meteor. Soc., P4.9. [Available online at http://ams.confex.com/ams/WAF-NWP-MESO/techprogram/paper_23074.htm.]

  • Du, J., and Coauthors, 2009: NCEP Short-Range Ensemble Forecast (SREF) system upgrade in 2009. Extended Abstracts, 19th Conf. on Numerical Weather Prediction and 23rd Conf. on Weather Analysis and Forecasting, Omaha, NE, Amer. Meteor. Soc., 4A.4. [Available online at http://ams.confex.com/ams/23WAF19NWP/techprogram/paper_153264.htm.]

  • Eckel, F., and Mass C. , 2005: Aspects of effective mesoscale, short-range ensemble forecasting. Wea. Forecasting, 20, 328–350.

  • Epstein, E., 1969: Stochastic dynamic prediction. Tellus, 21, 739–759.

  • Gneiting, T., Balabdaoui F. , and Raftery A. E. , 2007: Probabilistic forecasts, calibration and sharpness. J. Roy. Stat. Soc., 69B, 243–268.

    • Search Google Scholar
    • Export Citation

  • Green, D., and Swets J. , 1966: Signal Detection Theory and Psychophysics. John Wiley and Sons, 521 pp.

  • Hamill, T. M., and Colucci S. J. , 1998: Evaluation of Eta–RSM ensemble probabilistic precipitation forecasts. Mon. Wea. Rev., 126, 711–724.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., and Juras J. , 2006: Measuring forecast skill: Is it real skill or is it the varying climatology? Quart. J. Roy. Meteor. Soc., 132, 2905–2923, doi:10.1256/qj.06.25.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., and Whitaker J. S. , 2006: Probabilistic quantitative precipitation forecasts based on reforecast analogs: Theory and application. Mon. Wea. Rev., 134, 3209–3229.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., Whitaker J. S. , and Mullen S. L. , 2006: Reforecasts: An important new dataset for improving weather predictions. Bull. Amer. Meteor. Soc., 87, 33–46.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., Hagedorn R. , and Whitaker J. S. , 2008: Probabilistic forecast calibration using ECMWF and GFS ensemble reforecasts. Part II: Precipitation. Mon. Wea. Rev., 136, 2620–2632.

    • Search Google Scholar
    • Export Citation

  • Hanley, J., 1988: The robustness of the “binormal” assumptions used in fitting ROC curves. Med. Decis. Making, 8, 197–203.

  • Harris, D., Foufoula-Georgiou E. , Droegemeier K. K. , and Levit J. J. , 2001: Multiscale statistical properties of a high-resolution precipitation forecast. J. Hydrometeor., 2, 406–418.

    • Search Google Scholar
    • Export Citation

  • Hersbach, H., 2000: Decomposition of the continuous ranked probability score for ensemble prediction systems. Wea. Forecasting, 15, 559–570.

    • Search Google Scholar
    • Export Citation

  • Houtekamer, P. L., Mitchell H. L. , and Deng X. , 2009: Model error representation in an operational ensemble Kalman filter. Mon. Wea. Rev., 137, 2126–2143.

    • Search Google Scholar
    • Export Citation

  • Hsu, W.-R., and Murphy A. , 1986: The attributes diagram: A geometrical framework for assessing the quality of probability forecasts. Int. J. Forecast., 2, 285–293.

    • Search Google Scholar
    • Export Citation

  • Jolliffe, I. T., and Stephenson D. B. , 2012: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. 2nd ed. John Wiley and Sons, 292 pp.

  • Jones, M. S., Colle B. A. , and Tongue J. S. , 2007: Evaluation of a mesoscale short-range ensemble forecast system over the northeast United States. Wea. Forecasting, 22, 36–55.

    • Search Google Scholar
    • Export Citation

  • Kobold, M., and Suselj K. , 2005: Precipitation forecasts and their uncertainty as input into hydrological models. Hydrol. Earth Syst. Sci., 9, 322–332.

    • Search Google Scholar
    • Export Citation

  • Lahiri, S., 2003: Resampling Methods for Dependent Data. Springer, 388 pp.

  • Mascaro, G., Vivoni E. R. , and Deidda R. , 2010: Implications of ensemble quantitative precipitation forecast errors on distributed streamflow forecasting. J. Hydrometeor., 11, 69–86.

    • Search Google Scholar
    • Export Citation

  • McCollor, D., and Stull R. , 2008: Hydrometeorological accuracy enhancement via postprocessing of numerical weather forecasts in complex terrain. Wea. Forecasting, 23, 131–144.

    • Search Google Scholar
    • Export Citation

  • Metz, C. E., and Pan X. , 1999: “Proper” binormal ROC curves: Theory and maximum-likelihood estimation. J. Math. Psychol., 43, 1–33.

    • Search Google Scholar
    • Export Citation

  • Murphy, A. H., and Winkler R. , 1987: A general framework for forecast verification. Mon. Wea. Rev., 115, 1330–1338.

  • Pappenberger, F., and Buizza R. , 2009: The skill of ECMWF precipitation and temperature predictions in the Danube basin as forcings of hydrological models. Wea. Forecasting, 24, 749–766.

    • Search Google Scholar
    • Export Citation

  • Politis, D. N., and Romano J. P. , 1994: The stationary bootstrap. J. Amer. Stat. Assoc., 89, 1303–1313.

  • Rabier, F., 2006: Overview of global data assimilation developments in numerical weather-prediction centres. Quart. J. Roy. Meteor. Soc., 131, 3215–3233.

    • Search Google Scholar
    • Export Citation

  • Raynaud, L., Berre L. , and Desroziers G. , 2012: Accounting for model error in the Météo-France ensemble data assimilation system. Quart. J. Roy. Meteor. Soc., 138, 249–262, doi:10.1002/qj.906.

    • Search Google Scholar
    • Export Citation

  • Ruiz, J. J., Saulo C. , and Kalnay E. , 2012: How sensitive are probabilistic precipitation forecasts to the choice of calibration algorithms and the ensemble generation method? Part II: sensitivity to ensemble generation method. Meteor. Appl., doi:10.1002/met.262, in press.

    • Search Google Scholar
    • Export Citation

  • Saha, S., and Coauthors, 2006: The NCEP Climate Forecast System. J. Climate, 19, 3483–3517.

  • Schaake, J., and Coauthors, 2007: Precipitation and temperature ensemble forecasts from single-value forecasts. Hydrol. Earth Syst. Sci., 4, 655–717.

    • Search Google Scholar
    • Export Citation

  • Schumacher, R. S., and Davis C. A. , 2010: Ensemble-based forecast uncertainty analysis of diverse heavy rainfall events. Wea. Forecasting, 25, 1103–1122.

    • Search Google Scholar
    • Export Citation

  • Schwartz, C. S., and Coauthors, 2010: Toward improved convection-allowing ensembles: Model physics sensitivities and optimizing probabilistic guidance with small ensemble membership. Wea. Forecasting, 25, 263–280.

    • Search Google Scholar
    • Export Citation

  • Seo, D.-J., 1998: Real-time estimation of rainfall fields using rain gauge data under fractional coverage conditions. J. Hydrol., 208, 25–36.

    • Search Google Scholar
    • Export Citation

  • Seo, D.-J., Herr H. D. , and Schaake J. C. , 2006: A statistical post-processor for accounting of hydrologic uncertainty in short-range ensemble streamflow prediction. Hydrol. Earth Syst. Sci., 3, 1987–2035.

    • Search Google Scholar
    • Export Citation

  • Sloughter, J. M., Raftery A. E. , Gneiting T. , and Fraley C. , 2007: Probabilistic quantitative precipitation forecasting using Bayesian model averaging. Mon. Wea. Rev., 135, 3209–3220.

    • Search Google Scholar
    • Export Citation

  • Stensrud, D. J., Bao J. W. , and Warner T. T. , 2000: Using initial condition and model physics perturbations in short-range ensemble simulations of mesoscale convective systems. Mon. Wea. Rev., 128, 2077–2107.

    • Search Google Scholar
    • Export Citation

  • Toth, Z., Kalnay E. , Tracton S. M. , Wobus R. , and Irwin J. , 1997: A synoptic evaluation of the NCEP ensemble. Wea. Forecasting, 12, 140–153.

    • Search Google Scholar
    • Export Citation

  • Unger, D. A., van den Dool H. , O’Lenic E. , and Collins D. , 2009: Ensemble regression. Mon. Wea. Rev., 137, 2365–2379.

  • Warner, T., 2011: Numerical Weather and Climate Prediction. Cambridge University Press, 548 pp.

  • Wei, M., Toth Z. , Wobus R. , and Zhu Y. , 2008: Initial perturbations based on the Ensemble Transform (ET) technique in the NCEP Global Operational Forecast System. Tellus, 60, 62–79.

    • Search Google Scholar
    • Export Citation

  • Weigel, A. P., Liniger M. A. , and Appenzeller C. , 2008: Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts? Quart. J. Roy. Meteor. Soc., 134, 241–260.

    • Search Google Scholar
    • Export Citation

  • Weigel, A. P., Liniger M. A. , and Appenzeller C. , 2009: Seasonal ensemble forecasts: Are recalibrated single models better than multimodels? Mon. Wea. Rev., 137, 1460–1479.

    • Search Google Scholar
    • Export Citation

  • Whitaker, J. S., Hamill T. M. , Wei X. , Song Y. , and Toth Z. , 2008: Ensemble data assimilation with the NCEP Global Forecast System. Mon. Wea. Rev., 136, 463–482.

    • Search Google Scholar
    • Export Citation

  • Wilks, D. S., 2006: Comparison of ensemble-MOS methods in the Lorenz ’96 setting. Meteor. Appl., 13, 243–256.

  • Wilks, D. S., 2009: Extending logistic regression to provide full-probability-distribution MOS forecasts. Meteor. Appl., 16, 361–368.

  • Wu, L., Seo D.-J. , Demargne J. , Brown J. D. , Cong S. , and Schaake J. , 2011: Generation of ensemble precipitation forecast from single-valued quantitative precipitation forecast for hydrologic ensemble prediction. J. Hydrol., 339 (3–4), 281–298, doi:10.1016/j.jhydrol.2011.01.013.

    • Search Google Scholar
    • Export Citation

  • Yuan, H., Mullen S. , Gao X. , Sorooshian S. , Du J. , and Juang H. , 2005: Verification of probabilistic quantitative precipitation forecasts over the southwest United States during winter 2002/03 by the RSM ensemble system. Mon. Wea. Rev., 133, 279–294.

    • Search Google Scholar
    • Export Citation

  • Yuan, H., Gao X. , Mullen S. L. , Sorooshian S. , Du J. , and Juang H.-M. H. , 2007: Calibration of probabilistic quantitative precipitation forecasts with an artificial neural network. Wea. Forecasting, 22, 1287–1303.

    • Search Google Scholar
    • Export Citation

  • Yuan, H., Lu C. , McGinley J. A. , Schultz P. J. , Jamison B. D. , Wharton L. , and Anderson C. J. , 2009: Evaluation of short-range quantitative precipitation forecasts from a time-lagged multimodel ensemble. Wea. Forecasting, 24, 18–38.

    • Search Google Scholar
    • Export Citation

  • Zappa, M., and Coauthors, 2010: Propagation of uncertainty from observing systems and NWP into hydrological models: COST-731 Working Group 2. Atmos. Sci. Lett., 11, 83–91.

    • Search Google Scholar
    • Export Citation

  • Zhao, L., Duan Q. , Schaake J. , Ye A. , and Xia J. , 2011: A hydrologic post-processor for ensemble streamflow predictions. Adv. Geosci., 29, 51–59, doi:10.5194/adgeo-29-51-2011.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 2408 1392 75
PDF Downloads 385 69 8