A New Spatial-Scale Decomposition of the Brier Score: Application to the Verification of Lightning Probability Forecasts

B. Casati Recherche en Prévision Numérique, Meteorological Research Division, Environment Canada, Dorval, Quebec, Canada

Search for other papers by B. Casati in
Current site
Google Scholar
PubMed
Close
and
L. J. Wilson Recherche en Prévision Numérique, Meteorological Research Division, Environment Canada, Dorval, Quebec, Canada

Search for other papers by L. J. Wilson in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

A new scale decomposition of the Brier score for the verification of probabilistic forecasts defined on a spatial domain is introduced. The technique is illustrated on the Canadian Meteorological Centre (CMC) lightning probability forecasts. Probability forecasts of lightning occurrence in 3-h time windows and 24-km spatial resolution are verified against lightning observations from the North American Lightning Detection Network (NALDN) on a domain encompassing Canada and the northern United States. Verification is performed for lightning occurrences exceeding two different thresholds, to assess the forecast performance both for modest and intense lightning activity. Observation and forecast fields are decomposed into the sum of components on different spatial scales by performing a discrete 2D Haar wavelet decomposition. Wavelets, rather than Fourier transforms, were chosen because they are locally defined, and therefore more suitable for representing discontinuous spatial fields characterized by the presence of a few sparse nonzero values, such as lightning. Verification at different spatial scales is performed by evaluating Brier score and Brier skill score for each spatial-scale component. Reliability and resolution are also evaluated on different scales. Moreover, the bias on different scales is assessed, along with the ability of the forecasts to reproduce the observed-scale structure.

Corresponding author address: Dr. B. Casati, Environment Canada, 2121 Trans-Canada Highway, 5th floor, Dorval, QC H9P 1J3, Canada. Email: barbara.casati@ec.gc.ca

Abstract

A new scale decomposition of the Brier score for the verification of probabilistic forecasts defined on a spatial domain is introduced. The technique is illustrated on the Canadian Meteorological Centre (CMC) lightning probability forecasts. Probability forecasts of lightning occurrence in 3-h time windows and 24-km spatial resolution are verified against lightning observations from the North American Lightning Detection Network (NALDN) on a domain encompassing Canada and the northern United States. Verification is performed for lightning occurrences exceeding two different thresholds, to assess the forecast performance both for modest and intense lightning activity. Observation and forecast fields are decomposed into the sum of components on different spatial scales by performing a discrete 2D Haar wavelet decomposition. Wavelets, rather than Fourier transforms, were chosen because they are locally defined, and therefore more suitable for representing discontinuous spatial fields characterized by the presence of a few sparse nonzero values, such as lightning. Verification at different spatial scales is performed by evaluating Brier score and Brier skill score for each spatial-scale component. Reliability and resolution are also evaluated on different scales. Moreover, the bias on different scales is assessed, along with the ability of the forecasts to reproduce the observed-scale structure.

Corresponding author address: Dr. B. Casati, Environment Canada, 2121 Trans-Canada Highway, 5th floor, Dorval, QC H9P 1J3, Canada. Email: barbara.casati@ec.gc.ca

Save
  • Brieman, L., J. H. Friedman, R. A. Olshen, and C. J. Stone, 1984: Classification and Regression Trees. Chapmann and Hall/CRC Press, 368 pp.

    • Search Google Scholar
    • Export Citation
  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78 , 13.

  • Briggs, W. M., and R. A. Levine, 1997: Wavelets and field forecast verification. Mon. Wea. Rev., 125 , 13291341.

  • Burrows, W. R., C. Price, and L. J. Wilson, 2005: Warm season lightning probability prediction for Canada and the northern United States. Wea. Forecasting, 20 , 971988.

    • Search Google Scholar
    • Export Citation
  • Casati, B., G. Ross, and D. B. Stephenson, 2004: A new intensity-scale approach for the verification of spatial precipitation forecasts. Meteor. Appl., 11 , 141154.

    • Search Google Scholar
    • Export Citation
  • Côté, J., S. Gravel, A. Méthot, A. Patoine, M. Roch, and A. Staniforth, 1998: The operational CMC–MRB Global Environmental Multiscale (GEM) model: Part I: Design considerations and formulation. Mon. Wea. Rev., 126 , 13731395.

    • Search Google Scholar
    • Export Citation
  • Daubechies, I., 1992: Ten Lectures on Wavelets. SIAM, 357 pp.

  • De Elia, R., R. Laprise, and B. Denis, 2002: Forecasting skill limits of nested, limited-area models: A perfect model approach. Mon. Wea. Rev., 130 , 20062023.

    • Search Google Scholar
    • Export Citation
  • Denis, B., R. Laprise, and D. Caya, 2003: Sensitivity of a regional climate model to the resolution of the lateral boundary conditions. Climate Dyn., 20 , 107126.

    • Search Google Scholar
    • Export Citation
  • Efron, B., and R. J. Tibshirani, 1993: An Introduction to the Bootstrap. Chapman and Hall, 436 pp.

  • Harris, D., E. Foufoula-Georgiou, K. K. Droegemeier, and J. J. Levit, 2001: Multiscale statistical properties of a high-resolution precipitation forecast. J. Hydrometeor., 2 , 406418.

    • Search Google Scholar
    • Export Citation
  • Jolliffe, I. T., and D. B. Stephenson, 2003: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. John Wiley and Sons, 240 pp.

    • Search Google Scholar
    • Export Citation
  • Mallat, S. G., 1989: A theory for multiresolution signal decomposition: The wavelet representation. IEEE Trans. Pattern Anal. Mach. Intell., II , 674693.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1973: A new vector partition of the probability score. J. Appl. Meteor., 12 , 595600.

  • Nachamkin, J. E., 2004: Mesoscale verification using meteorological composites. Mon. Wea. Rev., 132 , 941955.

  • Orville, R. E., G. R. Huffines, W. R. Burrows, R. L. Holle, and K. L. Cummins, 2002: The North American Lightning Detection Network (NALDN)—First results: 1998–2000. Mon. Wea. Rev., 130 , 20982109.

    • Search Google Scholar
    • Export Citation
  • Roberts, N. M., and H. W. Lean, 2007: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev., in press.

    • Search Google Scholar
    • Export Citation
  • Theis, S. E., A. Hense, and U. Damrath, 2005: Probabilistic precipitation forecasts from a deterministic model: A pragmatic approach. Meteor. Appl., 12 , 257268.

    • Search Google Scholar
    • Export Citation
  • Zepeda-Arce, J., E. Foufoula-Georgiou, and K. K. Droegemeier, 2000: Space-time rainfall organization and its role in validating Quantitative Precipitation Forecasts. J. Geophys. Res., 105 , 1012910146.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 452 121 11
PDF Downloads 258 57 11