Verification with Variograms

Caren Marzban Applied Physics Laboratory and Department of Statistics, University of Washington, Seattle, Washington

Search for other papers by Caren Marzban in
Current site
Google Scholar
PubMed
Close
and
Scott Sandgathe Applied Physics Laboratory, University of Washington, Seattle, Washington, and College of Oceanic and Atmospheric Sciences, Oregon State University, Corvallis, Oregon

Search for other papers by Scott Sandgathe in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

The verification of a gridded forecast field, for example, one produced by numerical weather prediction (NWP) models, cannot be performed on a gridpoint-by-gridpoint basis; that type of approach would ignore the spatial structures present in both forecast and observation fields, leading to misinformative or noninformative verification results. A variety of methods have been proposed to acknowledge the spatial structure of the fields. Here, a method is examined that compares the two fields in terms of their variograms. Two types of variograms are examined: one examines correlation on different spatial scales and is a measure of texture; the other type of variogram is additionally sensitive to the size and location of objects in a field and can assess size and location errors. Using these variograms, the forecasts of three NWP model formulations are compared with observations/analysis, on a dataset consisting of 30 days in spring 2005. It is found that within statistical uncertainty the three formulations are comparable with one another in terms of forecasting the spatial structure of observed reflectivity fields. None, however, produce the observed structure across all scales, and all tend to overforecast the spatial extent and also forecast a smoother precipitation (reflectivity) field. A finer comparison suggests that the University of Oklahoma 2-km resolution Advanced Research Weather Research and Forecasting (WRF-ARW) model and the National Center for Atmospheric Research (NCAR) 4-km resolution WRF-ARW slightly outperform the 4.5-km WRF-Nonhydrostatic Mesoscale Model (NMM), developed by the National Oceanic and Atmospheric Administration/National Centers for Environmental Prediction (NOAA/NCEP), in terms of producing forecasts whose spatial structures are closer to that of the observed field.

Corresponding author address: Caren Marzban, Dept. of Statistics, University of Washington, Box 354322, Seattle, WA 98195-4322. Email: marzban@stat.washington.edu

Abstract

The verification of a gridded forecast field, for example, one produced by numerical weather prediction (NWP) models, cannot be performed on a gridpoint-by-gridpoint basis; that type of approach would ignore the spatial structures present in both forecast and observation fields, leading to misinformative or noninformative verification results. A variety of methods have been proposed to acknowledge the spatial structure of the fields. Here, a method is examined that compares the two fields in terms of their variograms. Two types of variograms are examined: one examines correlation on different spatial scales and is a measure of texture; the other type of variogram is additionally sensitive to the size and location of objects in a field and can assess size and location errors. Using these variograms, the forecasts of three NWP model formulations are compared with observations/analysis, on a dataset consisting of 30 days in spring 2005. It is found that within statistical uncertainty the three formulations are comparable with one another in terms of forecasting the spatial structure of observed reflectivity fields. None, however, produce the observed structure across all scales, and all tend to overforecast the spatial extent and also forecast a smoother precipitation (reflectivity) field. A finer comparison suggests that the University of Oklahoma 2-km resolution Advanced Research Weather Research and Forecasting (WRF-ARW) model and the National Center for Atmospheric Research (NCAR) 4-km resolution WRF-ARW slightly outperform the 4.5-km WRF-Nonhydrostatic Mesoscale Model (NMM), developed by the National Oceanic and Atmospheric Administration/National Centers for Environmental Prediction (NOAA/NCEP), in terms of producing forecasts whose spatial structures are closer to that of the observed field.

Corresponding author address: Caren Marzban, Dept. of Statistics, University of Washington, Box 354322, Seattle, WA 98195-4322. Email: marzban@stat.washington.edu

Save
  • Baldwin, M. E., and Elmore K. L. , 2005: Objective verification of high-resolution WRF forecasts during 2005 NSSL/SPC Spring Program. Preprints, 21st Conf. on Weather Analysis and Forecasting/17th Conf. on Numerical Weather Prediction, Washington, DC, Amer. Meteor. Soc., 11B.4. [Available online at http://ams.confex.com/ams/pdfpapers/95172.pdf].

    • Search Google Scholar
    • Export Citation
  • Baldwin, M. E., Lakshmivarahan S. , and Kain J. S. , 2002: Development of an “events-oriented” approach to forecast verification. Preprints, 19th Conf. on Weather Analysis and Forecasting/15th Conf. Numerical Weather Prediction, San Antonio, TX, Amer. Meteor. Soc., 7B.3. [Available online at http://ams.confex.com/ams/pdfpapers/47738.pdf].

    • Search Google Scholar
    • Export Citation
  • Barancourt, C., Creutin J. D. , and Rivoirard J. , 1992: A method for delineating and estimating rainfall fields. Water Resour. Res., 28 , 11331144.

  • Berrocal, V. J., Raftery A. E. , and Gneiting T. , 2007: Combining spatial statistical and ensemble information in probabilistic weather forecasts. Mon. Wea. Rev., 135 , 13861402.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Briggs, W. M., and Levine R. A. , 1997: Wavelets and field forecast verification. Mon. Wea. Rev., 125 , 13291341.

  • Brown, B. G., Mahoney J. L. , Davis C. A. , Bullock R. , and Mueller C. K. , 2002: Improved approaches for measuring the quality of convective weather forecasts. Preprints, 16th Conf. on Probability and Statistics in the Atmospheric Sciences, Orlando, FL, Amer. Meteor. Soc., 1.6. [Available online at http://ams.confex.com/ams/pdfpapers/29359.pdf].

    • Search Google Scholar
    • Export Citation
  • Brown, B. G., and Coauthors, 2004: New verification approaches for convective weather forecasts. Preprints, 11th Conf. on Aviation, Range, and Aerospace, Hyannis, MA, Amer. Meteor. Soc., 9.4. [Available online at http://ams.confex.com/ams/pdfpapers/82068.pdf].

    • Search Google Scholar
    • Export Citation
  • Bullock, R., Brown B. G. , Davis C. A. , Chapman M. , Manning K. W. , and Morss R. , 2004: An object-oriented approach to quantitative precipitation forecasts. Preprints, 17th Conf. on Probability and Statistics in the Atmospheric Sciences/20th Conf. on Weather Analysis and Forecasting/16th Conf. on Numerical Weather Prediction, Seattle, WA, Amer. Meteor. Soc., J12.4. [Available online at http://ams.confex.com/ams/pdfpapers/71819.pdf].

    • Search Google Scholar
    • Export Citation
  • Casati, B., Ross G. , and Stephenson D. B. , 2004: A new intensity-scale approach for the verification of spatial precipitation forecasts. Meteor. Appl., 11 , 141154.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Casati, B., and Coauthors, 2008: Forecast verification: Current status and future directions. Meteor. Appl., 15 , 318.

  • Chapman, M., Bullock R. , Brown B. G. , Davis C. A. , Manning K. W. , Morss R. , and Takacs A. , 2004: An object oriented approach to the verification of quantitative precipitation forecasts: Part II—Examples. Preprints, 17th Conf. on Probability and Statistics in the Atmospheric Sciences/20th Conf. on Weather Analysis and Forecasting/16th Conf. on Numerical Weather Prediction, Seattle, WA, Amer. Meteor. Soc., J12.5. [Available online at http://ams.confex.com/ams/pdfpapers/70881.pdf].

    • Search Google Scholar
    • Export Citation
  • Cressie, N. A. C., 1993: Statistics for Spatial Data. Wiley Series in Probability and Mathematical Statistics, John Wiley and Sons, 900 pp.

    • Search Google Scholar
    • Export Citation
  • Davis, A., Marshak A. , Wiscombe W. , and Cahalan R. , 1996: Multifractal characterization of intermittency in nonstationary geophysical signals and fields: A model-based perspective on ergocity issues illustrated with cloud data. Current Topics in Nonstationary Analysis, G. Treviño et al., Eds., World Scientific, 97–158.

    • Search Google Scholar
    • Export Citation
  • Du, J., Mullen S. L. , and Sanders F. , 2000: Removal of distortion error from an ensemble forecast. Mon. Wea. Rev., 128 , 33473351.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ebert, E. E., 2008: Fuzzy verification of high-resolution gridded forecasts: A review and proposed framework. Meteor. Appl., 15 , 5164.

  • Ebert, E. E., and McBride J. L. , 2000: Verification of precipitation in weather systems: Determination of systematic errors. J. Hydrol., 239 , 179202.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gebremichael, M., and Krajewski W. F. , 2004: Assessment of the statistical characterization of small-scale rainfall variability from radar: Analysis of TRMM ground validation datasets. J. Appl. Meteor., 43 , 11801199.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Germann, U., and Joss J. , 2001: Variograms of radar reflectivity to describe the spatial continuity of Alpine precipitation. J. Appl. Meteor., 40 , 10421059.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Germann, U., and Zawadzki I. , 2002: Scale-dependence of the predictability of precipitation from continental radar images. Part I: Description of the methodology. Mon. Wea. Rev., 130 , 28592873.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Greene, J. S., Cook W. E. , Knapp D. , and Haines P. , 2002: An examination of the uncertainty in interpolated winds and its effect on the validation and intercomparison of forecast models. J. Atmos. Oceanic Technol., 19 , 397401.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Harris, D., Foufoula-Georgiou E. , Droegemeier K. K. , and Levit J. J. , 2001: Multiscale statistical properties of a high-resolution precipitation forecast. J. Hydrol., 2 , 406418.

    • Search Google Scholar
    • Export Citation
  • Hoffman, R. N., Liu Z. , Louis J-F. , and Grassotti C. , 1995: Distortion representation of forecast errors. Mon. Wea. Rev., 123 , 27582770.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Journel, A. G., 1983: Non parametric estimation of spatial distributions. Math. Geol., 15 , 445467.

  • Kedem, B., Chiu L. S. , and North G. R. , 1990: Estimation of mean rain rate: Application to satellite observations. J. Geophys. Res., 95 , 19651972.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kundu, P., and Bell T. L. , 2003: A stochastic model of space–time variability of mesoscale rainfall: Statistics of spatial averages. Water Resour. Res., 39 , 13281343.

    • Search Google Scholar
    • Export Citation
  • Maillard, P., 2001: Developing methods of texture analysis in high resolution images of the Earth. Anais X SBSR, Foz do Iguacu, Brazil, INPE, 1309–1319.

    • Search Google Scholar
    • Export Citation
  • Marzban, C., and Sandgathe S. , 2006: Cluster analysis for verification of precipitation fields. Wea. Forecasting, 21 , 824838.

  • Marzban, C., and Sandgathe S. , 2007: Verification via optical flow. Proc. Third Int. Verification Methods Workshop, Reading, United Kingdom, ECMWF.

    • Search Google Scholar
    • Export Citation
  • Marzban, C., and Sandgathe S. , 2008: Cluster analysis for object-oriented verification of fields: A variation. Mon. Wea. Rev., 136 , 10131025.

  • Marzban, C., Sandgathe S. , and Lyons H. , 2008: An object-oriented verification of three NWP model formulations via cluster analysis: An objective and a subjective analysis. Mon. Wea. Rev., 136 , 33923407.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Matheron, G., 1963: Principles of geostatistics. Econ. Geol., 58 , 12461266.

  • Mela, K., and Louie J. N. , 2001: Correlation length and fractal dimension interpretation from seismic data using variograms and power spectra. Geophysics, 66 , 13721378.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Nachamkin, J. E., 2004: Mesoscale verification using meteorological composites. Mon. Wea. Rev., 132 , 941955.

  • Politis, D. N., and Romano J. P. , 1994: The stationary bootstrap. J. Amer. Stat. Assoc., 89 , 13031313.

  • Politis, D. N., Romano J. P. , and Wolf M. , 1999: Subsampling. Springer, 347 pp.

  • Ripley, B. D., 1991: Statistical Inference for Spatial Processes. Cambridge University Press, 148 pp.

  • Roberts, N. M., 2005: An investigation of the ability of a stormscale configuration of the Met Office NWP model to predict flood-producing rainfall. Forecasting Research Tech. Rep. 455, Met Office, Exeter, United Kingdom, 80 pp.

    • Search Google Scholar
    • Export Citation
  • Skamarock, W. C., 2004: Evaluating mesoscale NWP models using kinetic energy spectra. Mon. Wea. Rev., 132 , 30193032.

  • Skamarock, W. C., Baldwin M. E. , and Wang W. , 2004: Evaluating high-resolution NWP models using kinetic energy spectra. Proc. Fifth WRF/14th MM5 Users’ Workshop, Boulder, CO, NCAR, 53–56.

    • Search Google Scholar
    • Export Citation
  • Sapozhnikov, V. B., and Foufoula-Georgiou E. , 2007: An exponential Langevin-type model for rainfall exhibiting spatial and temporal scaling. Nonlinear Dynamics in Geosciences, A. A. Tsonis and J. B. Elsner, Eds., Springer, 87–100.

    • Search Google Scholar
    • Export Citation
  • Şen, Z., 1997: Objective analysis by cumulative semivariogram technique and its application in Turkey. J. Appl. Meteor., 36 , 17121724.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Venugopal, V., Basu S. , and Foufoula-Georgiou E. , 2005: A new metric for comparing precipitation patterns with an application to ensemble forecasts. J. Geophys. Res., 110 , D08111. doi:10.1029/2004JD005395.

    • Search Google Scholar
    • Export Citation
  • Weiss, S. J., Kain J. , Levit J. , Baldwin M. , Bright D. , Carbin G. , and Hart J. , 2005: NOAA Hazardous Weather Testbed: SPC/NSSL Spring Program 2005—Program overview and operations plan. Storm Prediction Center, Norman, OK, 61 pp. [Available online at http://www.spc.noaa.gov/exper/Spring_2005/2005_ops_plan.pdf].

    • Search Google Scholar
    • Export Citation
  • Yoo, C., and Ha E. , 2007: Effect of zero measurements on the spatial correlation structure of rainfall. Stoch. Environ. Res. Risk Assess., 21 , 287297.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zepeda-Arce, J., Foufoula-Georgiou E. , and Droegemeier K. , 2000: Space–time rainfall organization and its role in validating quantitative precipitation forecasts. J. Geophys. Res., 105 , 1012910146.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 859 380 16
PDF Downloads 365 80 1