• Adlerman, E. J., and K. K. Droegemeier, 2002: The sensitivity of numerically simulated cyclic mesocyclogenesis to variations in model physical and computational parameters. Mon. Wea. Rev., 130, 26712691, https://doi.org/10.1175/1520-0493(2002)130<2671:TSONSC>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ancell, B., and G. Hakim, 2007: Comparing adjoint- and ensemble-sensitivity analysis with applications to observation targeting. Mon. Wea. Rev., 135, 41174134, https://doi.org/10.1175/2007MWR1904.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Benjamini, Y., and Y. Hochberg, 1995: Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Stat. Soc., 57B, 289300.

    • Search Google Scholar
    • Export Citation
  • Bowman, K. P., J. Sacks, and Y.-F. Chang, 1993: Design and analysis of numerical experiments. J. Atmos. Sci., 50, 12671278, https://doi.org/10.1175/1520-0469(1993)050<1267:DAAONE>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Boyle, J. S., S. A. Klein, D. D. Lucas, H.-Y. Ma, J. Tannahill, and S. Xie, 2015: The parametric sensitivity of CAM5’s MJO. J. Geophys. Res. Atmos., 120, 14241444, https://doi.org/10.1002/2014JD022507.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bretz, F., T. Hothorn, and P. Westfall, 2010: Multiple Comparisons Using R. Chapman and Hall/CRC, 208 pp.

  • Cacuci, D. G., M. Ionescu-Bujor, and I. M. Navon, 2005: Sensitivity and Uncertainty Analysis, Volume II: Applications to Large-Scale Systems. Chapman and Hall/CRC, 368 pp.

    • Crossref
    • Export Citation
  • Cioppa, T., and T. Lucas, 2007: Efficient nearly orthogonal and space-filling Latin hypercubes. Technometrics, 49, 4555, https://doi.org/10.1198/004017006000000453.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cressie, N. A. C., 1993: Statistics for Spatial Data. John Wiley and Sons, 900 pp.

    • Crossref
    • Export Citation
  • Crook, N. A., 1996: Sensitivity of moist convection forced by boundary layer processes to low-level thermodynamic fields. Mon. Wea. Rev., 124, 17671785, https://doi.org/10.1175/1520-0493(1996)124<1767:SOMCFB>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Daescu, D. N., and R. H. Langland, 2013: Error covariance sensitivity and impact estimation with adjoint 4D-Var: Theoretical aspects and first applications to NAVDAS-AR. Quart. J. Roy. Meteor. Soc., 139, 226241, https://doi.org/10.1002/qj.1943.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Davis, C. A., and K. Emanuel, 1991: Potential vorticity diagnostics of cyclogenesis. Mon. Wea. Rev., 119, 19291953, https://doi.org/10.1175/1520-0493(1991)119<1929:PVDOC>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • DelSole, T., and X. Yang, 2011: Field significance of regression patterns. J. Climate, 24, 50945107, https://doi.org/10.1175/2011JCLI4105.1.

  • Dmitrienko, A., F. Bretz, P. H. Westfall, J. Troendle, B. L. Wiens, A. C. Tamhane, and J. C. Hsu, 2009: Multiple testing methodology. Multiple Testing Problems in Pharmaceutical Statistics, A. Dmitrienko, A. Tamhane, and F. Bretz, Eds., Chapman and Hall, 3598.

    • Crossref
    • Export Citation
  • Douglas, E. M., R. M. Vogel, and C. N. Kroll, 2000: Trends in floods and low flows in the United States: Impact of spatial correlation. J. Hydrol., 240, 90105, https://doi.org/10.1016/S0022-1694(00)00336-X.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, Q., and Coauthors, 2006: Model Parameter Estimation Experiment (MOPEX): An overview of science strategy and major results from the second and third workshops. J. Hydrol., 320, 317, https://doi.org/10.1016/j.jhydrol.2005.07.031.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Elmore, K. L., M. E. Baldwin, and D. M. Schultz, 2006: Field significance revisited: Spatial bias errors in forecasts as applied to the Eta Model. Mon. Wea. Rev., 134, 519531, https://doi.org/10.1175/MWR3077.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fang, K.-T., R. Li, and A. Sudjianto, 2006: Design and Modeling for Computer Experiments. Chapman & Hall/CRC, 290 pp.

    • Crossref
    • Export Citation
  • Fox, J., M. Friendly, and S. Weisberg, 2013: Hypothesis tests for multivariate linear models using the car package. The R J., 5, 3952.

  • Gallus, W. A., Jr., 1999: Eta simulations of three extreme precipitation events: Sensitivity to resolution and convective parameterization. Wea. Forecasting, 14, 405426, https://doi.org/10.1175/1520-0434(1999)014<0405:ESOTEP>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gombos, D., and J. A. Hansen, 2008: Potential vorticity regression and its relationship to dynamical piecewise inversion. Mon. Wea. Rev., 136, 26682682, https://doi.org/10.1175/2007MWR2165.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gómez-Navarro, J. J., C. C. Raible, and S. Dierer, 2015: Sensitivity of the WRF Model to PBL parametrisations and nesting techniques: Evaluation of wind storms over complex terrain. Geosci. Model Dev., 8, 33493363, https://doi.org/10.5194/gmd-8-3349-2015.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hacker, J. P., C. Snyder, S.-Y. Ha, and M. Pocernich, 2011: Linear and non-linear response to parameter variations in a mesoscale model. Tellus, 63, 429444, https://doi.org/10.1111/j.1600-0870.2010.00505.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hodur, R. M., 1997: The Naval Research Laboratory’s Coupled Ocean/Atmosphere Mesoscale Prediction System (COAMPS). Mon. Wea. Rev., 125, 14141430, https://doi.org/10.1175/1520-0493(1997)125<1414:TNRLSC>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Holt, T. R., J. A. Cummings, C. H. Bishop, J. D. Doyle, X. Hong, S. Chen, and Y. Jin, 2011: Development and testing of a coupled ocean–atmosphere mesoscale ensemble prediction system. Ocean Dyn., 61, 19371954, https://doi.org/10.1007/s10236-011-0449-9.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hourdin, F., and Coauthors, 2017: The art and science of climate model tuning. Bull. Amer. Meteor. Soc., 98, 589602, https://doi.org/10.1175/BAMS-D-15-00135.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Houston, A. L., and D. Niyogi, 2007: The sensitivity of convective initiation to the lapse rate of the active cloud-bearing layer. Mon. Wea. Rev., 135, 30133032, https://doi.org/10.1175/MWR3449.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Järvinen, H., M. Laine, A. Solonen, and H. Haario, 2012: Ensemble prediction and parameter estimation system: The concept. Quart. J. Roy. Meteor. Soc., 138, 281288, https://doi.org/10.1002/qj.923.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kain, J. S., and J. M. Fritsch, 1993: Convective parameterization for mesoscale models: The Kain–Fritsch scheme. The Representation of Cumulus Convection in Numerical Models, Meteor. Monogr., No. 46, Amer. Meteor. Soc., 165–170.

    • Crossref
    • Export Citation
  • Kucherenko, S., D. Albrecht, and A. Saltelli, 2015: Exploring multi-dimensional spaces: A comparison of Latin hypercube and quasi Monte Carlo sampling techniques. Cornell University Rep., 30 pp., https://arxiv.org/abs/1505.02350.

  • Laine, M., A. Solonen, H. Haario, and H. Järvinen, 2012: Ensemble prediction and parameter estimation system: The method. Quart. J. Roy. Meteor. Soc., 138, 289297, https://doi.org/10.1002/qj.922.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Li, X., N. Sudarsanam, and D. D. Frey, 2006: Regularities in data from factorial experiments. Complexity, 11, 3245, https://doi.org/10.1002/cplx.20123.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Marzban, C., 2013: Variance-based sensitivity analysis: An illustration on the Lorenz ’63 model. Mon. Wea. Rev., 141, 40694079, https://doi.org/10.1175/MWR-D-13-00032.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Marzban, C., S. Sandgathe, J. D. Doyle, and N. C. Lederer, 2014: Variance-based sensitivity analysis: Preliminary results in COAMPS. Mon. Wea. Rev., 142, 20282042, https://doi.org/10.1175/MWR-D-13-00195.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • McKay, M. D., R. J. Beckman, and W. J. Conover, 1979: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21, 239245, https://doi.org/10.2307/1268522.

    • Search Google Scholar
    • Export Citation
  • Mölders, N., M. Laube, and G. Kramm, 1995: On the parameterization of ice microphysics in a mesoscale α weather forecast model. Atmos. Res., 38, 207235, https://doi.org/10.1016/0169-8095(94)00094-T.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Montgomery, D. C., 2009: Design and Analysis of Experiments. 7th ed. John Wiley & Sons, 656 pp.

  • Noble, W. S., 2009: How does multiple testing correction work? Nat. Biotechnol., 27, 11351137, https://doi.org/10.1038/nbt1209-1135.

  • Ollinaho, P., H. Järvinen, P. Bauer, M. Laine, P. Bechtold, J. Susiluoto, and H. Haario, 2014: Optimization of NWP model closure parameters using total energy norm of forecast error as a target. Geosci. Model Dev., 7, 18891900, https://doi.org/10.5194/gmd-7-1889-2014.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Qian, Y., and Coauthors, 2015: Parametric sensitivity analysis of precipitation at global and local scales in the Community Atmosphere Model CAM5. J. Adv. Model. Earth Syst., 7, 382411, https://doi.org/10.1002/2014MS000354.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Robock, A., and Coauthors, 2003: Evaluation of the North American Land Data Assimilation System over the southern Great Plains during warm seasons. J. Geophys. Res., 108, 8846, https://doi.org/10.1029/2002JD003245.

    • Search Google Scholar
    • Export Citation
  • Roebber, P. J., 1989: The role of surface heat and moisture fluxes associated with large-scale ocean current meanders in maritime cyclogenesis. Mon. Wea. Rev., 117, 16761694, https://doi.org/10.1175/1520-0493(1989)117<1676:TROSHA>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Roebber, P. J., and L. F. Bosart, 1998: The sensitivity of precipitation to circulation details. Part I: An analysis of regional analogs. Mon. Wea. Rev., 126, 437455, https://doi.org/10.1175/1520-0493(1998)126<0437:TSOPTC>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rosenblatt, J., 2013: A practitioner’s guide to multiple hypothesis testing error rates. Cornell University Rep., 32 pp., https://arxiv.org/abs/1304.4920.

  • Rutledge, S. A., and P. Hobbs, 1983: The mesoscale and microscale structure and organization of clouds and precipitation in midlatitude cyclones. VIII: A model for the “seeder-feeder” process in warm-frontal rainbands. J. Atmos. Sci., 40, 11851206, https://doi.org/10.1175/1520-0469(1983)040<1185:TMAMSA>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sacks, J., W. J. Welch, T. J. Mitchell, and H. P. Wynn, 1989: Design and analysis of computer experiments. Stat. Sci., 4, 409423, https://doi.org/10.1214/ss/1177012413.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Saltelli, A., S. Tarantola, F. Campolongo, and M. Ratto, 2004: Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models. John Wiley & Sons, 232 pp.

  • Saltelli, A., P. Annoni, I. Azzini, F. Campolongo, M. Ratto, and S. Tarantola, 2010: Variance based sensitivity analysis of model output: Design and estimator for the total sensitivity index. Comput. Phys. Commun., 181, 259270, https://doi.org/10.1016/j.cpc.2009.09.018.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Santner, T. J., B. J. Williams, and W. I. Notz, 2003: The Design and Analysis of Computer Experiments. Springer, 283 pp.

    • Crossref
    • Export Citation
  • Schumann, M. R., and P. J. Roebber, 2010: The influence of upper-tropospheric potential vorticity on convective morphology. Mon. Wea. Rev., 138, 463474, https://doi.org/10.1175/2009MWR3091.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Stensrud, D. J., 2007: Parameterization Schemes: Keys to Understanding Numerical Weather Models. Cambridge University Press, 459 pp.

    • Crossref
    • Export Citation
  • Torn, R. D., and G. Hakim, 2008: Ensemble-based sensitivity analysis. Mon. Wea. Rev., 136, 663677, https://doi.org/10.1175/2007MWR2132.1.

  • Voudouri, A., P. Khain, I. Carmona, O. Bellprat, F. Grazzini, E. Avgoustoglou, and J. M. Bettems, and P. Kaufmann, 2017: Objective calibration of numerical weather prediction models. Atmos. Res., 190, 128140, https://doi.org/10.1016/j.atmosres.2017.02.007.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Warner, T. T., 2011: Numerical Weather and Climate Prediction. Cambridge University Press, 526 pp.

  • Weisman, M. L., and Coauthors, 2015: The Mesoscale Predictability Experiment (MPEX). Bull. Amer. Meteor. Soc., 96, 21272149, https://doi.org/10.1175/BAMS-D-13-00281.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 1997: Resampling hypothesis tests for autocorrelated fields. J. Climate, 10, 6582, https://doi.org/10.1175/1520-0442(1997)010<0065:RHTFAF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2006: On “field significance” and the false discovery rate. J. Appl. Meteor. Climatol., 45, 11811189, https://doi.org/10.1175/JAM2404.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2011: Statistical Methods in the Atmospheric Sciences. 3rd ed. International Geophysics Series, Vol. 100, Academic Press, 704 pp.

  • Zhao, J., and C. Tiede, 2011: Using a variance-based sensitivity analysis for analyzing the relation between measurements and unknown parameters of a physical model. Nonlinear Processes Geophys., 18, 269276, https://doi.org/10.5194/npg-18-269-2011.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 31 31 3
PDF Downloads 21 21 2

Sensitivity Analysis of the Spatial Structure of Forecasts in Mesoscale Models: Continuous Model Parameters

View More View Less
  • 1 Applied Physics Laboratory, and Department of Statistics, University of Washington, Seattle, Washington
  • 2 Department of Statistics, University of Washington, Seattle, Washington
  • 3 Applied Physics Laboratory, University of Washington, Seattle, Washington
  • 4 Naval Research Laboratory, Monterey, California
  • 5 The Boeing Company, Applied Mathematics, Seattle, Washington
© Get Permissions
Restricted access

Abstract

A methodology is proposed for examining the effect of model parameters (assumed to be continuous) on the spatial structure of forecasts. The methodology involves several statistical methods of sampling and inference to assure the sensitivity results are statistically sound. Specifically, Latin hypercube sampling is employed to vary the model parameters, and multivariate multiple regression is used to account for spatial correlations in assessing the sensitivities. The end product is a geographic “map” of p values for each model parameter, allowing one to display and examine the spatial structure of the sensitivity. As an illustration, the effect of 11 model parameters in a mesoscale model on forecasts of convective and grid-scale precipitation, surface air temperature, and water vapor is studied. A number of spatial patterns in sensitivity are found. For example, a parameter that controls the fraction of available convective clouds and precipitation fed back to the grid scale influences precipitation forecasts mostly over the southeastern region of the domain; another parameter that modifies the surface fluxes distinguishes between precipitation forecasts over land and over water. The sensitivity of surface air temperature and water vapor forecasts also has distinct spatial patterns, with the specific pattern depending on the model parameter. Among the 11 parameters examined, there is one (an autoconversion factor in the microphysics) that appears to have no influence in any region and on any of the forecast quantities.

© 2018 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Caren Marzban, marzban@stat.washington.edu

Abstract

A methodology is proposed for examining the effect of model parameters (assumed to be continuous) on the spatial structure of forecasts. The methodology involves several statistical methods of sampling and inference to assure the sensitivity results are statistically sound. Specifically, Latin hypercube sampling is employed to vary the model parameters, and multivariate multiple regression is used to account for spatial correlations in assessing the sensitivities. The end product is a geographic “map” of p values for each model parameter, allowing one to display and examine the spatial structure of the sensitivity. As an illustration, the effect of 11 model parameters in a mesoscale model on forecasts of convective and grid-scale precipitation, surface air temperature, and water vapor is studied. A number of spatial patterns in sensitivity are found. For example, a parameter that controls the fraction of available convective clouds and precipitation fed back to the grid scale influences precipitation forecasts mostly over the southeastern region of the domain; another parameter that modifies the surface fluxes distinguishes between precipitation forecasts over land and over water. The sensitivity of surface air temperature and water vapor forecasts also has distinct spatial patterns, with the specific pattern depending on the model parameter. Among the 11 parameters examined, there is one (an autoconversion factor in the microphysics) that appears to have no influence in any region and on any of the forecast quantities.

© 2018 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Caren Marzban, marzban@stat.washington.edu
Save