• Barnston, A. G., 1992: Correspondence among the correlation, RMSE, and Heidke forecast verification measures; refinement of the Heidke score. Wea. Forecasting, 7, 699709, doi:10.1175/1520-0434(1992)007<0699:CATCRA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Barnston, A. G., and J. L. Thomas, 1983: Rainfall measurement accuracy in FACE: A comparison of gage and radar rainfalls. J. Climate Appl. Meteor., 22, 20382052, doi:10.1175/1520-0450(1983)022<2038:RMAIFA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Carroll, R. J., D. Ruppert, L. A. Stefanski, and C. M. Crainiceanu, 2006: Measurement Error in Nonlinear Models: A Modern Perspective. 2nd ed. CRC Press, 485 pp.

  • Ebert, E. E., and J. L. McBride, 2000: Verification of precipitation in weather systems: Determination of systematic errors. J. Hydrol., 239, 179202, doi:10.1016/S0022-1694(00)00343-7.

    • Search Google Scholar
    • Export Citation
  • Entekhabi, D., R. H. Reichle, R. D. Koster, and W. T. Crow, 2010: Performance metrics for soil moisture retrievals and application requirements. J. Hydrometeor., 11, 832840, doi:10.1175/2010JHM1223.1.

    • Search Google Scholar
    • Export Citation
  • Fisher, S. R. A., 1958: Statistical Methods for Research Workers. 13th ed. Hafner Publishing Co., 356 pp.

  • Gong, W., H. Gupta, D. Yang, K. Sricharan, and A. Hero, 2013: Estimating epistemic and aleatory uncertainties during hydrologic modeling: An information theoretic approach. Water Resour. Res., 49, 22532273, doi:10.1002/wrcr.20161.

    • Search Google Scholar
    • Export Citation
  • Gupta, H. V., H. Kling, K. K. Yilmaz, and G. F. Martinez, 2009: Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling. J. Hydrol., 377, 8091, doi:10.1016/j.jhydrol.2009.08.003.

    • Search Google Scholar
    • Export Citation
  • Habib, E., W. F. Krajewski, and G. J. Ciach, 2001: Estimation of rainfall interstation correlation. J. Hydrometeor., 2, 621629, doi:10.1175/1525-7541(2001)002<0621:EORIC>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Jaynes, E. T., 2003: Probability Theory: The Logic of Science. Cambridge University Press, 762 pp.

  • Livezey, R. E., J. D. Hoopingarner, and J. Huang, 1995: Verification of official monthly mean 700-hPa height forecasts: An update. Wea. Forecasting, 10, 512527, doi:10.1175/1520-0434(1995)010<0512:VOOMMH>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1988: Skill scores based on the mean square error and their relationships to the correlation coefficient. Mon. Wea. Rev., 116, 24172424, doi:10.1175/1520-0493(1988)116<2417:SSBOTM>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1995: The coefficients of correlation and determination as measures of performance in forecast verification. Wea. Forecasting, 10, 681688, doi:10.1175/1520-0434(1995)010<0681:TCOCAD>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., and R. L. Winkler, 1987: A general framework for forecast verification. Mon. Wea. Rev., 115, 13301338, doi:10.1175/1520-0493(1987)115<1330:AGFFFV>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., and E. S. Epstein, 1989: Skill scores and correlation coefficients in model verification. Mon. Wea. Rev., 117, 572582, doi:10.1175/1520-0493(1989)117<0572:SSACCI>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., B. G. Brown, and Y.-S. Chen, 1989: Diagnostic verification of temperature forecasts. Wea. Forecasting, 4, 485501, doi:10.1175/1520-0434(1989)004<0485:DVOTF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Nearing, G., and H. Gupta, 2015: The quantity and quality of information in hydrologic models. Water Resour. Res., 51, 524538, doi:10.1002/2014WR015895.

    • Search Google Scholar
    • Export Citation
  • Papoulis, A., 1984: Probability, Random Variables and Stochastic Processes. 2nd ed. McGraw-Hill Inc., 575 pp.

  • Stewart, T. R., 1990: A decomposition of the correlation coefficient and its use in analyzing forecasting skill. Wea. Forecasting, 5, 661666, doi:10.1175/1520-0434(1990)005<0661:ADOTCC>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Tang, L., Y. Tian, F. Yan, and E. Habib, 2015: An improved procedure for the validation of satellite-based precipitation estimates. Atmos. Res., 163, 6173, doi:10.1016/j.atmosres.2014.12.016.

    • Search Google Scholar
    • Export Citation
  • Taylor, K. E., 2001: Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res., 106, 71837192, doi:10.1029/2000JD900719.

    • Search Google Scholar
    • Export Citation
  • Tian, Y., C. D. Peters-Lidard, and J. B. Eylander, 2010: Real-time bias reduction for satellite-based precipitation estimates. J. Hydrometeor., 11, 12751285, doi:10.1175/2010JHM1246.1.

    • Search Google Scholar
    • Export Citation
  • Tian, Y., G. J. Huffman, R. F. Adler, L. Tang, M. Sapiano, V. Maggioni, and H. Wu, 2013: Modeling errors in daily precipitation measurements: Additive or multiplicative? Geophys. Res. Lett., 40, 20602065, doi:10.1002/grl.50320.

    • Search Google Scholar
    • Export Citation
  • Vrugt, J., and M. Sadegh, 2013: Towards diagnostic model calibration and evaluation: Approximate Bayesian computation. Water Resour. Res., 49, 43354345, doi:10.1002/wrcr.20354.

    • Search Google Scholar
    • Export Citation
  • Wan, Z., Y. Zhang, Q. Zhang, and Z.-L. Li, 2004: Quality assessment and validation of the MODIS global land surface temperature. Int. J. Remote Sens., 25, 261274, doi:10.1080/0143116031000116417.

    • Search Google Scholar
    • Export Citation
  • Weijs, S., G. Schoups, and N. Giesen, 2010: Why hydrologic predictions should be evaluated using information theory. Hydrol. Earth Syst. Sci., 14, 25452558, doi:10.5194/hess-14-2545-2010.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2011: Statistical Methods in the Atmospheric Sciences. Academic Press, 676 pp.

  • Willmott, C. J., 1981: On the validation of models. Phys. Geogr., 2, 184194.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 800 531 26
PDF Downloads 752 494 43

Performance Metrics, Error Modeling, and Uncertainty Quantification

View More View Less
  • 1 Hydrological Sciences Laboratory, NASA Goddard Space Flight Center, Greenbelt, and Earth System Science Interdisciplinary Center, University of Maryland, College Park, College Park, Maryland
  • | 2 Hydrological Sciences Laboratory, NASA Goddard Space Flight Center, Greenbelt, and Science Applications International Corporation, Beltsville, Maryland
  • | 3 Hydrological Sciences Laboratory, NASA Goddard Space Flight Center, Greenbelt, Maryland
  • | 4 Hydrological Sciences Laboratory, NASA Goddard Space Flight Center, Greenbelt, and Earth System Science Interdisciplinary Center, University of Maryland, College Park, College Park, Maryland
  • | 5 Earth System Science Interdisciplinary Center, University of Maryland, College Park, College Park, Maryland
Restricted access

Abstract

A common set of statistical metrics has been used to summarize the performance of models or measurements—the most widely used ones being bias, mean square error, and linear correlation coefficient. They assume linear, additive, Gaussian errors, and they are interdependent, incomplete, and incapable of directly quantifying uncertainty. The authors demonstrate that these metrics can be directly derived from the parameters of the simple linear error model. Since a correct error model captures the full error information, it is argued that the specification of a parametric error model should be an alternative to the metrics-based approach. The error-modeling methodology is applicable to both linear and nonlinear errors, while the metrics are only meaningful for linear errors. In addition, the error model expresses the error structure more naturally, and directly quantifies uncertainty. This argument is further explained by highlighting the intrinsic connections between the performance metrics, the error model, and the joint distribution between the data and the reference.

Corresponding author address: Yudong Tian, NASA Goddard Space Flight Center, Mail Code 617, Greenbelt, MD 20771-5808. E-mail: yudong.tian@nasa.gov

Abstract

A common set of statistical metrics has been used to summarize the performance of models or measurements—the most widely used ones being bias, mean square error, and linear correlation coefficient. They assume linear, additive, Gaussian errors, and they are interdependent, incomplete, and incapable of directly quantifying uncertainty. The authors demonstrate that these metrics can be directly derived from the parameters of the simple linear error model. Since a correct error model captures the full error information, it is argued that the specification of a parametric error model should be an alternative to the metrics-based approach. The error-modeling methodology is applicable to both linear and nonlinear errors, while the metrics are only meaningful for linear errors. In addition, the error model expresses the error structure more naturally, and directly quantifies uncertainty. This argument is further explained by highlighting the intrinsic connections between the performance metrics, the error model, and the joint distribution between the data and the reference.

Corresponding author address: Yudong Tian, NASA Goddard Space Flight Center, Mail Code 617, Greenbelt, MD 20771-5808. E-mail: yudong.tian@nasa.gov
Save