Detecting Dependence in the Sensitive Parameter Space of a Model Using Statistical Inference and Large Forecast Ensembles

J. G. McLay Naval Research Laboratory, Monterey, California

Search for other papers by J. G. McLay in
Current site
Google Scholar
PubMed
Close
and
M. Liu Naval Research Laboratory, Monterey, California

Search for other papers by M. Liu in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

This study looks for evidence of correlation among model physical parameters in the sensitive parameter space defined by those randomly sampled physical parameter vectors that induce the most notable response in some forecast metric. These “sensitive parameter vectors” are identified through an ensemble methodology. The correlation analysis is facilitated by two established techniques from statistical inference theory. The random parameter vectors are found to induce a considerable range of forecast responses in terms of five metrics, such as bias and variance. The metrics enable measurement not only of the biggest forecast response but also of the most beneficial forecast response (e.g., in terms of reduction of forecast error). For most metrics, multiple parameter pairs exhibit significantly more correlation than would be expected from random sampling processes. The correlations frequently involve parameters from two different physical routines. These inference results are independently supported by a Monte Carlo simulation. The results suggest that correlations among parameters must be taken into account in order to gain the most response from a model when carrying out parameter variation experiments. Also, they reinforce the idea that parameter estimation efforts need to be expanded so that they simultaneously estimate the joint distribution of parameters across multiple physical routines.

Corresponding author address: J. G. McLay, Naval Research Laboratory, Stop 2, 7 Grace Hopper Ave., Monterey, CA 93943. E-mail: justin.mclay@nrlmry.navy.mil

Abstract

This study looks for evidence of correlation among model physical parameters in the sensitive parameter space defined by those randomly sampled physical parameter vectors that induce the most notable response in some forecast metric. These “sensitive parameter vectors” are identified through an ensemble methodology. The correlation analysis is facilitated by two established techniques from statistical inference theory. The random parameter vectors are found to induce a considerable range of forecast responses in terms of five metrics, such as bias and variance. The metrics enable measurement not only of the biggest forecast response but also of the most beneficial forecast response (e.g., in terms of reduction of forecast error). For most metrics, multiple parameter pairs exhibit significantly more correlation than would be expected from random sampling processes. The correlations frequently involve parameters from two different physical routines. These inference results are independently supported by a Monte Carlo simulation. The results suggest that correlations among parameters must be taken into account in order to gain the most response from a model when carrying out parameter variation experiments. Also, they reinforce the idea that parameter estimation efforts need to be expanded so that they simultaneously estimate the joint distribution of parameters across multiple physical routines.

Corresponding author address: J. G. McLay, Naval Research Laboratory, Stop 2, 7 Grace Hopper Ave., Monterey, CA 93943. E-mail: justin.mclay@nrlmry.navy.mil
Save
  • Aksoy, A., F. Zhang, and J. W. Nielsen-Gammon, 2006: Ensemble-based simultaneous state and parameter estimation in a two-dimensional sea-breeze model. Mon. Wea. Rev., 134, 29512970, doi:10.1175/MWR3224.1.

    • Search Google Scholar
    • Export Citation
  • Alexanderian, A., J. Winokur, I. Sraj, A. Srinivasan, M. Iskandarani, W. C. Thacker, and O. M. Knio, 2012: Global sensitivity analysis in an ocean general circulation model: A sparse spectral projection approach. Comput. Geosci., 16, 757778, doi:10.1007/s10596-012-9286-2.

    • Search Google Scholar
    • Export Citation
  • Arakawa, A., and W. H. Schubert, 1974: Interaction of a cumulus cloud ensemble with the large-scale environment, Part I. J. Atmos. Sci., 31, 674701, doi:10.1175/1520-0469(1974)031<0674:IOACCE>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Benjamini, Y., and Y. Hochberg, 1995: Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Stat. Soc., 57B, 289300.

    • Search Google Scholar
    • Export Citation
  • Bentley, J., 1999: Programming Pearls. 2nd ed. Addison-Wesley, 256 pp.

  • Bowler, N. E., A. Arribas, K. R. Mylne, K. B. Robertson, and S. E. Beare, 2008: The MOGREPS short-range ensemble prediction system. Quart. J. Roy. Meteor. Soc., 134, 703722, doi:10.1002/qj.234.

    • Search Google Scholar
    • Export Citation
  • Bright, D. R., and S. L. Mullen, 2002: Short-range ensemble forecasts of precipitation during the Southwest monsoon. Wea. Forecasting, 17, 10801100, doi:10.1175/1520-0434(2002)017<1080:SREFOP>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Chua, B., L. Xu, T. Rosmond, and E. Zaron, 2009: Preconditioning representer-based variational data assimilation systems: application to NAVDAS-AR. Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications, S. K. Park and L. Xu, Eds., Springer-Verlag, 307–319.

  • Clough, S. A., M. W. Shephard, E. J. Mlawer, J. S. Delamere, M. J. Iacono, K. Cady-Pereira, S. Boukabara, and P. D. Brown, 2005: Atmospheric radiative transfer modeling: A summary of the AER codes. J. Quant. Spectrosc. Radiat. Transf., 91, 233244, doi:10.1016/j.jqsrt.2004.05.058.

    • Search Google Scholar
    • Export Citation
  • Efron, B., and R. J. Tibshirani, 1993: An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability, Vol. 57, Chapman and Hall, 456 pp.

  • Errico, R. M., and T. Vukicevic, 1992: Sensitivity analysis using an adjoint of the PSU–NCAR mesoscale model. Mon. Wea. Rev., 120, 16441660, doi:10.1175/1520-0493(1992)120<1644:SAUAAO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Evensen, G., 2006: Data Assimilation: The Ensemble Kalman Filter. Springer, 279 pp.

  • Gibbons, J. D., and S. Chakraborti, 2011: Nonparametric Statistical Inference. 5th ed. Chapman and Hall/CRC Press, 650 pp.

  • Golaz, J.-C., V. E. Larson, J. A. Hansen, D. P. Schanen, and B. M. Griffin, 2007: Elucidating model inadequacies in a cloud parameterization by use of an ensemble-based calibration framework. Mon. Wea. Rev., 135, 40774096, doi:10.1175/2007MWR2008.1.

    • Search Google Scholar
    • Export Citation
  • Hacker, J. P., C. Snyder, S.-Y. Ha, and M. Pocernich, 2011: Linear and nonlinear response to parameter variations in a mesoscale model. Tellus, 63A, 429444, doi:10.1111/j.1600-0870.2010.00505.x.

    • Search Google Scholar
    • Export Citation
  • Han, J., and H.-L. Pan, 2011: Revision of convection and vertical diffusion schemes in the NCEP Global Forecast System. Wea. Forecasting, 26, 520533, doi:10.1175/WAF-D-10-05038.1.

    • Search Google Scholar
    • Export Citation
  • Hogan, T. F., 2007: Land surface modeling in the Navy Operational Global Atmospheric Prediction System. Preprints, 22nd Conf. on Weather Analysis and Forecasting/18th Conf. on Numerical Weather Prediction, Park City, Utah, Amer. Meteor. Soc., 11B.1. [Available online at https://ams.confex.com/ams/pdfpapers/123403.pdf.]

  • Hu, X.-M., F. Zhang, and J. W. Nielsen-Gammon, 2010: Ensemble-based simultaneous state and parameter estimation for treatment of mesoscale model error: A real-data study. Geophys. Res. Lett., 37, L08802, doi:10.1029/2010GL043017.

    • Search Google Scholar
    • Export Citation
  • Jarvinen, H., M. Laine, A. Solonen, and H. Haario, 2012: Ensemble prediction and parameter estimation system: The concept. Quart. J. Roy. Meteor. Soc., 138, 281288, doi:10.1002/qj.923.

    • Search Google Scholar
    • Export Citation
  • Jung, Y., M. Xue, and G. Zhang, 2010: Simultaneous estimation of microphysical parameters and the atmospheric state using simulated polarimetric radar data and ensemble Kalman filter in the presence of an observation operator error. Mon. Wea. Rev., 138, 539562, doi:10.1175/2009MWR2748.1.

    • Search Google Scholar
    • Export Citation
  • Khade, V. M., J. A. Hansen, J. S. Reid, and D. L. Westphal, 2013: Ensemble filter based estimation of spatially distributed parameters in a mesoscale dust model: Experiments with simulated and real data. Atmos. Chem. Phys., 13, 34813500, doi:10.5194/acp-13-3481-2013.

    • Search Google Scholar
    • Export Citation
  • Laine, M., A. Solonen, H. Haario, and H. Jarvinen, 2012: Ensemble prediction and parameter estimation system: The method. Quart. J. Roy. Meteor. Soc., 138, 289297, doi:10.1002/qj.922.

    • Search Google Scholar
    • Export Citation
  • Li, X., M. Charron, L. Spacek, and G. Candille, 2008: A regional ensemble prediction system based on moist targeted singular vectors and stochastic parameter perturbations. Mon. Wea. Rev., 136, 443462, doi:10.1175/2007MWR2109.1.

    • Search Google Scholar
    • Export Citation
  • Lin, J. W. B., and J. D. Neelin, 2000: Influence of a stochastic moist convective parameterization on tropical climate variability. Geophys. Res. Lett., 27, 36913694, doi:10.1029/2000GL011964.

    • Search Google Scholar
    • Export Citation
  • Louis, J. F., 1979: A parametric model of vertical eddy fluxes in the atmosphere. Bound.-Layer Meteor., 17, 187202, doi:10.1007/BF00117978.

    • Search Google Scholar
    • Export Citation
  • Mckay, M. D., R. J. Beckman, and W. J. Conover, 2000: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 42, 5561, doi:10.1080/00401706.2000.10485979.

    • Search Google Scholar
    • Export Citation
  • Murphy, J. M., D. M. H. Sexton, D. N. Barnett, G. S. Jones, M. J. Webb, M. Collins, and D. A. Stainforth, 2004: Quantification of modelling uncertainties in a large ensemble of climate change simulations. Nature, 430, 768772, doi:10.1038/nature02771.

    • Search Google Scholar
    • Export Citation
  • Navon, I. M., 1998: Practical and theoretical aspects of adjoint parameter estimation and identifiability in meteorology and oceanography. Dyn. Atmos. Oceans, 27, 5579, doi:10.1016/S0377-0265(97)00032-8.

    • Search Google Scholar
    • Export Citation
  • Posselt, D. J., and T. Vukicevic, 2010: Robust characterization of model physics uncertainty for simulations of deep moist convection. Mon. Wea. Rev., 138, 15131535, doi:10.1175/2009MWR3094.1.

    • Search Google Scholar
    • Export Citation
  • Posselt, D. J., and C. H. Bishop, 2012: Nonlinear parameter estimation: Comparison of an ensemble Kalman smoother with a Markov chain Monte Carlo algorithm. Mon. Wea. Rev., 140, 19571974, doi:10.1175/MWR-D-11-00242.1.

    • Search Google Scholar
    • Export Citation
  • Posselt, D. J., D. Hodyss, and C. H. Bishop, 2013: Errors in ensemble Kalman smoother estimates of cloud microphysical parameters. Mon. Wea. Rev., 142, 16311654, doi:10.1175/MWR-D-13-00290.1.

    • Search Google Scholar
    • Export Citation
  • Rabier, F., E. Klinker, P. Courtier, and A. Hollingsworth, 1996: Sensitivity of forecast errors to initial conditions. Quart. J. Roy. Meteor. Soc., 122, 121150, doi:10.1002/qj.49712252906.

    • Search Google Scholar
    • Export Citation
  • Reynolds, C. A., J. A. Ridout, and J. G. McLay, 2011: Examination of parameter variations in the U.S. Navy global ensemble. Tellus, 63A, 841857, doi:10.1111/j.1600-0870.2011.00532.x.

    • Search Google Scholar
    • Export Citation
  • Ritchie, H., 1991: Application of the semi-Lagrangian method to a multilevel spectral primitive equation model. Quart. J. Roy. Meteor. Soc., 117, 91106, doi:10.1002/qj.49711749705.

    • Search Google Scholar
    • Export Citation
  • Rosmond, T., and L. Xu, 2006: Development of NAVDAS-AR: Non-linear formulation and outer loop tests. Tellus, 58A, 4558, doi:10.1111/j.1600-0870.2006.00148.x.

    • Search Google Scholar
    • Export Citation
  • Rosolem, R., H. V. Gupta, W. J. Shuttleworth, X. Zeng, and L. G. Goncalves de Goncalves, 2012: A fully multiple-criteria implementation of the Sobol method for parameter sensitivity analysis. J. Geophys. Res., 117, D07103, doi:10.1029/2011JD016355.

    • Search Google Scholar
    • Export Citation
  • Sobol, I. M., 1993: Sensitivity estimates for nonlinear mathematical models. Math. Model. Comput. Exp., 1, 407414.

  • Sobol, I. M., 2001: Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math. Comput. Simul., 55, 271280, doi:10.1016/S0378-4754(00)00270-6.

    • Search Google Scholar
    • Export Citation
  • Sraj, I., and Coauthors, 2013: Bayesian inference of drag parameters using Fanapi AXBT data. Mon. Wea. Rev., 141, 23472367, doi:10.1175/MWR-D-12-00228.1.

    • Search Google Scholar
    • Export Citation
  • Stensrud, D. J., J.-W. Bao, and T. T. Warner, 2000: Using initial condition and model physics perturbations in short-range ensemble simulations of mesoscale convective systems. Mon. Wea. Rev., 128, 20772107, doi:10.1175/1520-0493(2000)128<2077:UICAMP>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Tamminen, J., and E. Kyrola, 2001: Bayesian solution for nonlinear and non-Gaussian inverse problems by Markov chain Monte Carlo method. J. Geophys. Res., 106, 14 37714 390, doi:10.1029/2001JD900007.

    • Search Google Scholar
    • Export Citation
  • Tong, M., and M. Xue, 2008: Simultaneous estimation of microphysical parameters and atmospheric state with simulated radar data and ensemble square root Kalman filter. Part I: Sensitivity analysis and parameter identifiability. Mon. Wea. Rev., 136, 16301648, doi:10.1175/2007MWR2070.1.

    • Search Google Scholar
    • Export Citation
  • Torn, R. D., and G. J. Hakim, 2008: Ensemble-based sensitivity analysis. Mon. Wea. Rev., 136, 663677, doi:10.1175/2007MWR2132.1.

  • Van Lier-Walqui, M., T. Vukicevic, and D. J. Posselt, 2012: Quantification of cloud microphysical parameterization uncertainty using radar reflectivity. Mon. Wea. Rev., 140, 34423466, doi:10.1175/MWR-D-11-00216.1.

    • Search Google Scholar
    • Export Citation
  • Webster, S., A. R. Brown, D. R. Cameron, and C. P. Jones, 2003: Improvements to the representation of orography in the Met Office Unified Model. Quart. J. Roy. Meteor. Soc., 129B, 19892010, doi:10.1256/qj.02.133.

    • Search Google Scholar
    • Export Citation
  • Wilcoxon, F., 1945: Individual comparisons by ranking methods. Biom. Bull., 1, 8083, doi:10.2307/3001968.

  • Wilks, D. S., 2006a: On “field significance” and the false discovery rate. J. Appl. Meteor. Climatol., 45, 11811189, doi:10.1175/JAM2404.1.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2006b: Statistical Methods in the Atmospheric Sciences. 2nd ed. Academic Press, 627 pp.

  • Williamson, D., M. Goldstein, L. Allison, A. Blaker, P. Challenor, L. Jackson, and K. Yamazaki, 2013: History matching for exploring and reducing climate model parameter space using observations and a large perturbed physics ensemble. Climate Dyn., 41, 17031729, doi:10.1007/s00382-013-1896-4.

    • Search Google Scholar
    • Export Citation
  • Xu, L., T. Rosmond, and R. Daley, 2005: Development of NAVDAS-AR: Formulation and initial tests of the linear problem. Tellus, 57A, 546559, doi:10.1111/j.1600-0870.2005.00123.x.

    • Search Google Scholar
    • Export Citation
  • Zhao, Q. Y., and F. H. Carr, 1997: A prognostic cloud scheme for operational NWP models. Mon. Wea. Rev., 125, 19311953, doi:10.1175/1520-0493(1997)125<1931:APCSFO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 989 881 198
PDF Downloads 61 21 1