Information-Based Probabilistic Verification Scores for Two-Dimensional Ensemble Forecast Data: A Madden–Julian Oscillation Index Example

Yuhei Takaya aMeteorological Research Institute, Japan Meteorological Agency, Tsukuba, Ibaraki, Japan

Search for other papers by Yuhei Takaya in
Current site
Google Scholar
PubMed
Close
https://orcid.org/0000-0002-7359-8897
,
Kensuke K. Komatsu aMeteorological Research Institute, Japan Meteorological Agency, Tsukuba, Ibaraki, Japan
bAtmosphere and Ocean Research Institute, The University of Tokyo, Kashiwa, Chiba, Japan

Search for other papers by Kensuke K. Komatsu in
Current site
Google Scholar
PubMed
Close
,
Hideitsu Hino cThe Institute of Statistical Mathematics, Tachikawa, Tokyo, Japan

Search for other papers by Hideitsu Hino in
Current site
Google Scholar
PubMed
Close
, and
Frédéric Vitart dThe European Centre for Medium-Range Weather Forecasts, Reading, Berkshire, United Kingdom

Search for other papers by Frédéric Vitart in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Probabilistic forecasting is a common activity in many fields of the Earth sciences. Assessing the quality of probabilistic forecasts—probabilistic forecast verification—is therefore an essential task in these activities. Numerous methods and metrics have been proposed for this purpose; however, the probabilistic verification of vector variables of ensemble forecasts has received less attention than others. Here we introduce a new approach that is applicable for verifying ensemble forecasts of continuous, scalar, and two-dimensional vector data. The proposed method uses a fixed-radius near-neighbors search to compute two information-based scores, the ignorance score (the logarithmic score) and the information gain, which quantifies the skill gain from the reference forecast. Basic characteristics of the proposed scores were examined using idealized Monte Carlo simulations. The results indicated that both the continuous ranked probability score (CRPS) and the proposed score with a relatively small ensemble size (<25) are not proper in terms of the forecast dispersion. The proposed verification method was successfully used to verify the Madden–Julian oscillation index, which is a two-dimensional quantity. The proposed method is expected to advance probabilistic ensemble forecasts in various fields.

Significance Statement

In the Earth sciences, stochastic future states are estimated by solving a large number of forecasts (called ensemble forecasts) based on physical equations with slightly different initial conditions and stochastic parameters. The verification of probabilistic forecasts is an essential part of forecasting and modeling activity in the Earth sciences. However, there is no information-based probabilistic verification score applicable for vector variables of ensemble forecasts. The purpose of this study is to introduce a novel method for verifying scalar and two-dimensional vector variables of ensemble forecasts. The proposed method offers a new approach to probabilistic verification and is expected to advance probabilistic ensemble forecasts in various fields.

© 2023 American Meteorological Society. This published article is licensed under the terms of the default AMS reuse license. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Komatsu’s current affiliation: Osaka Regional Headquarters, Japan Meteorological Agency, Osaka, Japan.

Corresponding author: Yuhei Takaya, yuhei.takaya@mri-jma.go.jp

Abstract

Probabilistic forecasting is a common activity in many fields of the Earth sciences. Assessing the quality of probabilistic forecasts—probabilistic forecast verification—is therefore an essential task in these activities. Numerous methods and metrics have been proposed for this purpose; however, the probabilistic verification of vector variables of ensemble forecasts has received less attention than others. Here we introduce a new approach that is applicable for verifying ensemble forecasts of continuous, scalar, and two-dimensional vector data. The proposed method uses a fixed-radius near-neighbors search to compute two information-based scores, the ignorance score (the logarithmic score) and the information gain, which quantifies the skill gain from the reference forecast. Basic characteristics of the proposed scores were examined using idealized Monte Carlo simulations. The results indicated that both the continuous ranked probability score (CRPS) and the proposed score with a relatively small ensemble size (<25) are not proper in terms of the forecast dispersion. The proposed verification method was successfully used to verify the Madden–Julian oscillation index, which is a two-dimensional quantity. The proposed method is expected to advance probabilistic ensemble forecasts in various fields.

Significance Statement

In the Earth sciences, stochastic future states are estimated by solving a large number of forecasts (called ensemble forecasts) based on physical equations with slightly different initial conditions and stochastic parameters. The verification of probabilistic forecasts is an essential part of forecasting and modeling activity in the Earth sciences. However, there is no information-based probabilistic verification score applicable for vector variables of ensemble forecasts. The purpose of this study is to introduce a novel method for verifying scalar and two-dimensional vector variables of ensemble forecasts. The proposed method offers a new approach to probabilistic verification and is expected to advance probabilistic ensemble forecasts in various fields.

© 2023 American Meteorological Society. This published article is licensed under the terms of the default AMS reuse license. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Komatsu’s current affiliation: Osaka Regional Headquarters, Japan Meteorological Agency, Osaka, Japan.

Corresponding author: Yuhei Takaya, yuhei.takaya@mri-jma.go.jp
Save
  • Benedetti, R., 2010: Scoring rules for forecast verification. Mon. Wea. Rev., 138, 203211, https://doi.org/10.1175/2009MWR2945.1.

  • Box, G. E. P., and M. E. Muller, 1958: A note on the generation of random normal deviates. Ann. Math. Stat., 29, 610611, https://doi.org/10.1214/aoms/1177706645.

    • Search Google Scholar
    • Export Citation
  • Bröcker, J., and L. A. Smith, 2007: Scoring probabilistic forecast: The importance of being proper. Wea. Forecasting, 22, 382388, https://doi.org/10.1175/WAF966.1.

    • Search Google Scholar
    • Export Citation
  • Bröcker, J., and L. A. Smith, 2008: From ensemble forecasts to predictive distribution functions. Tellus, 60A, 663678, https://doi.org/10.1111/j.1600-0870.2008.00333.x.

    • Search Google Scholar
    • Export Citation
  • Casati, B., M. Dorninger, C. A. S. Coelho, E. E. Ebert, C. Marsigli, M. P. Mittermaier, and E. Gilleland, 2022: The 2020 International Verification Methods Workshop Online: Major outcomes and way forward. Bull. Amer. Meteor. Soc., 103, E899E910, https://doi.org/10.1175/BAMS-D-21-0126.1.

    • Search Google Scholar
    • Export Citation
  • Dawid, A. P., and P. Sebastiani, 1999: Coherent dispersion criteria for optimal experimental design. Ann. Stat., 27, 6581, https://doi.org/10.1214/aos/1018031101.

    • Search Google Scholar
    • Export Citation
  • Ferro, C. A. T., 2014: Fair scores for ensemble forecasts. Quart. J. Roy. Meteor. Soc., 140, 19171923, https://doi.org/10.1002/qj.2270.

    • Search Google Scholar
    • Export Citation
  • Gneiting, T., and A. E. Raftery, 2007: Strictly proper scoring rules, prediction, and estimation. J. Amer. Stat. Assoc., 102, 359378, https://doi.org/10.1198/016214506000001437.

    • Search Google Scholar
    • Export Citation
  • Gneiting, T., L. I. Stanberry, E. P. Grimit, L. Held, and N. A. Johnson, 2008: Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds. TEST, 17, 211235, https://doi.org/10.1007/s11749-008-0114-x.

    • Search Google Scholar
    • Export Citation
  • Gombos, D., J. A. Hansen, J. Du, and J. McQueen, 2007: Theory and applications of the minimum spanning tree rank histogram. Mon. Wea. Rev., 135, 14901505, https://doi.org/10.1175/MWR3362.1.

    • Search Google Scholar
    • Export Citation
  • Goria, M. N., N. N. Leonenko, V. V. Mergel, and P. L. Novi Inverardi, 2005: A new class of random vector entropy estimators and its applications in testing statistical hypotheses. J. Nonparametric Stat., 17, 277297, https://doi.org/10.1080/104852504200026815.

    • Search Google Scholar
    • Export Citation
  • Gottschalck, J., and Coauthors, 2010: A framework for assessing operational Madden–Julian oscillation forecasts. Bull. Amer. Meteor. Soc., 91, 12471258, https://doi.org/10.1175/2010BAMS2816.1.

    • Search Google Scholar
    • Export Citation
  • Grimit, E. P., T. Gneiting, V. J. Berrocal, and N. A. Johnson, 2006: The continuous ranked probability score for circular variables and its application to mesoscale forecast ensemble verification. Quart. J. Roy. Meteor. Soc., 132, 29252942, https://doi.org/10.1256/qj.05.235.

    • Search Google Scholar
    • Export Citation
  • Hersbach, H., 2000: Decomposition of the continuous ranked probability score for ensemble prediction systems. Wea. Forecasting, 15, 559570, https://doi.org/10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Hersbach, H., and Coauthors, 2020: The ERA5 global reanalysis. Quart. J. Roy. Meteor. Soc., 146, 19992049, https://doi.org/10.1002/qj.3803.

    • Search Google Scholar
    • Export Citation
  • Hino, H., and N. Murata, 2013: Information estimators for weighted observations. Neural Networks, 46, 260275, https://doi.org/10.1016/j.neunet.2013.06.005.

    • Search Google Scholar
    • Export Citation
  • Kalnay, E., 2019: Historical perspective: Earlier ensembles and forecasting forecast skill. Quart. J. Roy. Meteor. Soc., 145, 2534, https://doi.org/10.1002/qj.3595.

    • Search Google Scholar
    • Export Citation
  • Kim, H.-M., 2017: The impact of the mean moisture bias on the key physics of MJO propagation in the ECMWF reforecast. J. Geophys. Res. Atmos., 122, 77727784, https://doi.org/10.1002/2017JD027005.

    • Search Google Scholar
    • Export Citation
  • Kim, H.-M., P. J. Webster, V. E. Toma, and D. Kim, 2014: Predictability and prediction skill of the MJO in two operational forecasting systems. J. Climate, 27, 53645378, https://doi.org/10.1175/JCLI-D-13-00480.1.

    • Search Google Scholar
    • Export Citation
  • Kim, H.-M., F. Vitart, and D. E. Waliser, 2018: Prediction of the Madden–Julian oscillation: A review. J. Climate, 31, 94259443, https://doi.org/10.1175/JCLI-D-18-0210.1.

    • Search Google Scholar
    • Export Citation
  • Kozachenko, L. F., and N. N. Leonenko, 1987: Sample estimate of entropy of a random vector. Probl. Info. Transm., 23, 95101.

  • Kraskov, A., H. Stögbauer, and P. Grassberger, 2004: Estimating mutual information. Phys. Rev., 69E, 066138, https://doi.org/10.1103/PhysRevE.69.066138.

    • Search Google Scholar
    • Export Citation
  • Kumar, A., 2009: Finite samples and uncertainty estimates for skill measures for seasonal predictions. Mon. Wea. Rev., 137, 26222631, https://doi.org/10.1175/2009MWR2814.1.

    • Search Google Scholar
    • Export Citation
  • Leutbecher, M., 2019: Ensemble size: How suboptimal is less than infinity? Quart. J. Roy. Meteor. Soc., 145, 107128, https://doi.org/10.1002/qj.3387.

    • Search Google Scholar
    • Export Citation
  • Lim, Y., S.-W. Son, and D. Kim, 2018: MJO prediction skill of the subseasonal-to-seasonal prediction models. J. Climate, 31, 40754094, https://doi.org/10.1175/JCLI-D-17-0545.1.

    • Search Google Scholar
    • Export Citation
  • Loftsgaarden, D. O., and C. P. Quesenberry, 1965: A nonparametric estimate of multivariate density function. Ann. Math. Stat., 36, 10491051, https://doi.org/10.1214/aoms/1177700079.

    • Search Google Scholar
    • Export Citation
  • Lombardi, D., and S. Pant, 2016: Nonparametric k-nearest-neighbor entropy estimator. Phys. Rev., 93E, 013310, https://doi.org/10.1103/PhysRevE.93.013310.

    • Search Google Scholar
    • Export Citation
  • Marshall, A. G., H. H. Hendon, and D. Hudson, 2016: Visualizing and verifying probabilistic forecasts of the Madden–Julian Oscillation. Geophys. Res. Lett., 43, 12 27812 286, https://doi.org/10.1002/2016GL071423.

    • Search Google Scholar
    • Export Citation
  • Matsueda, M., and H. Endo, 2011: Verification of medium-range MJO forecasts with TIGGE. Geophys. Res. Lett., 38, L11801, https://doi.org/10.1029/2011GL047480.

    • Search Google Scholar
    • Export Citation
  • Murray, S. A., 2018: The importance of ensemble techniques for operational space weather forecasting. Space Wea., 16, 777783, https://doi.org/10.1029/2018SW001861.

    • Search Google Scholar
    • Export Citation
  • Palmer, T., 2019: The ECMWF ensemble prediction system: Looking back (more than) 25 years and projecting forward 25 years. Quart. J. Roy. Meteor. Soc., 145, 1224, https://doi.org/10.1002/qj.3383.

    • Search Google Scholar
    • Export Citation
  • Peirolo, R., 2011: Information gain as a score for probabilistic forecasts. Meteor. Appl., 18, 917, https://doi.org/10.1002/met.188.

  • Pérez-Cruz, F., 2008: Estimation of information theoretic measures for continuous random variables. Advances in Neural Information Processing Systems: Proceedings of the First 12 Conferences, M. I. Jordan, Y. LeCun, and S. A. Solla, Eds., Vol. 21, Curran Associates Inc., 1257–1264.

  • Pestov, V., 2000: On the geometry of similarity search: Dimensionality curse and concentration of measure. Info. Process. Lett., 73, 4751, https://doi.org/10.1016/S0020-0190(99)00156-8.

    • Search Google Scholar
    • Export Citation
  • Roulston, M. S., and L. A. Smith, 2002: Evaluating probabilistic forecasts using information theory. Mon. Wea. Rev., 130, 16531660, https://doi.org/10.1175/1520-0493(2002)130<1653:EPFUIT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Scheuerer, M., and T. M. Hamill, 2015: Variogram-based proper scoring rules for probabilistic forecasts of multivariate quantities. Mon. Wea. Rev., 143, 13211334, https://doi.org/10.1175/MWR-D-14-00269.1.

    • Search Google Scholar
    • Export Citation
  • Scott, D. W., 1979: On optimal and data-based histograms. Biometrika, 66, 605610, https://doi.org/10.1093/biomet/66.3.605.

  • Siegert, S., C. A. T. Ferro, D. B. Stephenson, and M. Leutbecher, 2019: The ensemble-adjusted ignorance score for forecasts issued as normal distributions. Quart. J. Roy. Meteor. Soc., 145, 129139, https://doi.org/10.1002/qj.3447.

    • Search Google Scholar
    • Export Citation
  • Singh, H., N. Misra, V. Hnizdo, A. Fedorowicz, and E. Demchuk, 2003: Nearest neighbor estimates of entropy. Amer. J. Math. Manage. Sci., 23, 301321, https://doi.org/10.1080/01966324.2003.10737616.

    • Search Google Scholar
    • Export Citation
  • Smith, L. A., and J. A. Hansen, 2004: Extending the limits of ensemble forecast verification with the minimum spanning tree. Mon. Wea. Rev., 132, 15221528, https://doi.org/10.1175/1520-0493(2004)132<1522:ETLOEF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Sugiyama, M., T. Suzuki, S. Nakajima, H. Kashima, P. von Bünau, and M. Kawanabe, 2008: Direct importance estimation for covariate shift adaptation. Ann. Inst. Stat. Math., 60, 699746, https://doi.org/10.1007/s10463-008-0197-x.

    • Search Google Scholar
    • Export Citation
  • Tödter, J., and B. Ahrens, 2012: Generalization of the ignorance score: Continuous ranked version and its decomposition. Mon. Wea. Rev., 140, 20052017, https://doi.org/10.1175/MWR-D-11-00266.1.

    • Search Google Scholar
    • Export Citation
  • Troin, M., R. Arsenault, A. W. Wood, F. Brissette, and J.-L. Martel, 2021: Generating ensemble streamflow forecasts: A review of methods and approaches over the past 40 years. Water Resour. Res., 57, e2020WR028392, https://doi.org/10.1029/2020WR028392.

    • Search Google Scholar
    • Export Citation
  • Vitart, F., and F. Molteni, 2010: Simulation of the Madden–Julian Oscillation and its teleconnections in the ECMWF forecast system. Quart. J. Roy. Meteor. Soc., 136, 842855, https://doi.org/10.1002/qj.623.

    • Search Google Scholar
    • Export Citation
  • Vitart, F., and Y. Takaya, 2021: Lagged ensembles in sub-seasonal predictions. Quart. J. Roy. Meteor. Soc., 147, 32273242, https://doi.org/10.1002/qj.4125.

    • Search Google Scholar
    • Export Citation
  • Vitart, F., and Coauthors, 2017: The Subseasonal to Seasonal (S2S) prediction project database. Bull. Amer. Meteor. Soc., 98, 163173, https://doi.org/10.1175/BAMS-D-16-0017.1.

    • Search Google Scholar
    • Export Citation
  • Wand, M. P., and M. C. Jones, 1995: Kernel Smoothing. Springer, 224 pp.

  • Weijs, S. V., R. van Nooijen, and N. van de Giesen, 2010: Kullback–Leibler divergence as a forecast skill score with classic reliability–resolution–uncertainty decomposition. Mon. Wea. Rev., 138, 33873399, https://doi.org/10.1175/2010MWR3229.1.

    • Search Google Scholar
    • Export Citation
  • Wheeler, M. C., and H. H. Hendon, 2004: An all-season real-time multivariate MJO index: Development of an index for monitoring and prediction. Mon. Wea. Rev., 132, 19171932, https://doi.org/10.1175/1520-0493(2004)132<1917:AARMMI>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2020: Regularized Dawid–Sebastiani score for multivariate ensemble forecasts. Quart. J. Roy. Meteor. Soc., 146, 24212431, https://doi.org/10.1002/qj.3800.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 446 446 5
Full Text Views 234 234 9
PDF Downloads 248 248 4