Editorial Type:
Article
Article Type:
Research Article

A Strategy for Verifying Near-Convection-Resolving Model Forecasts at Observing Sites

Marion P. Mittermaier Numerical Modelling, Weather Science, Met Office, Exeter, United Kingdom

Search for other papers by Marion P. Mittermaier in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Apr 2014
DOI:
https://doi.org/10.1175/WAF-D-12-00075.1
Page(s):
185–204
Restricted access

Abstract

Routine verification of deterministic numerical weather prediction (NWP) forecasts from the convection-permitting 4-km (UK4) and near-convection-resolving 1.5-km (UKV) configurations of the Met Office Unified Model (MetUM) has shown that it is hard to consistently demonstrate an improvement in skill from the higher-resolution model, even though subjective comparison suggests that it performs better. In this paper the use of conventional metrics and precise matching (through extracting the nearest grid point to an observing site) of the forecast to conventional synoptic observations in space and time is replaced with the use of inherently probabilistic metrics such as the Brier score, ranked probability, and continuous ranked probability scores applied to neighborhoods of forecast grid points. Three neighborhood sizes were used: ~4, ~12, and ~25 km, which match the sizes of the grid elements currently used operationally. Six surface variables were considered: 2-m temperature, 10-m wind speed, total cloud amount (TCA), cloud-base height (CBH), visibility, and hourly precipitation. Any neighborhood has a positive impact on skill, either in reducing the skill deficit or enhancing the skillfulness over and above the single grid point. This is true for all variables. An optimal neighborhood appears to depend on the variable and threshold. Adopting this probabilistic approach enables easy comparison to future near-convection-resolving ensemble prediction systems (EPS) and also enables the optimization of postprocessing to maximize the skill of forecast products.

Corresponding author address: Dr. Marion Mittermaier, Met Office, FitzRoy Road, Exeter EX1 3PB, United Kingdom. E-mail: marion.mittermaier@metoffice.gov.uk

Abstract

Routine verification of deterministic numerical weather prediction (NWP) forecasts from the convection-permitting 4-km (UK4) and near-convection-resolving 1.5-km (UKV) configurations of the Met Office Unified Model (MetUM) has shown that it is hard to consistently demonstrate an improvement in skill from the higher-resolution model, even though subjective comparison suggests that it performs better. In this paper the use of conventional metrics and precise matching (through extracting the nearest grid point to an observing site) of the forecast to conventional synoptic observations in space and time is replaced with the use of inherently probabilistic metrics such as the Brier score, ranked probability, and continuous ranked probability scores applied to neighborhoods of forecast grid points. Three neighborhood sizes were used: ~4, ~12, and ~25 km, which match the sizes of the grid elements currently used operationally. Six surface variables were considered: 2-m temperature, 10-m wind speed, total cloud amount (TCA), cloud-base height (CBH), visibility, and hourly precipitation. Any neighborhood has a positive impact on skill, either in reducing the skill deficit or enhancing the skillfulness over and above the single grid point. This is true for all variables. An optimal neighborhood appears to depend on the variable and threshold. Adopting this probabilistic approach enables easy comparison to future near-convection-resolving ensemble prediction systems (EPS) and also enables the optimization of postprocessing to maximize the skill of forecast products.

Corresponding author address: Dr. Marion Mittermaier, Met Office, FitzRoy Road, Exeter EX1 3PB, United Kingdom. E-mail: marion.mittermaier@metoffice.gov.uk
Save
Cover Weather and Forecasting
Weather and Forecasting

  • Atger, F., 2001: Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlinear Processes Geophys., 8, 401–417.

    • Search Google Scholar
    • Export Citation

  • Ben Bouallégue, Z., Theis S. , and Gebhardt C. , 2011: From verification results to probabilistic products: Spatial techniques applied to ensemble forecasting. Abstracts, Fifth Int. Verifcation Methods Workshop, Melbourne, VIC, Australia, CAWCR Tech. Rep. 46, 4.

  • Brier, G., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78, 1–3.

  • Bryan, G., Wyngaard J. , and Fritsch J. , 2003: Resolution requirements for the simulation of deep moist convection. Mon. Wea. Rev., 131, 2394–2416.

    • Search Google Scholar
    • Export Citation

  • Clark, P. A., Harcourt S. A. , Macpherson B. , Mathison C. T. , Cusack S. , and Naylor M. , 2008: Prediction of visibility and aerosol within the operational Met Office Unified Model. I: Model formulation and variational assimilation. Quart. J. Roy. Meteor. Soc., 134, 1801–1816.

    • Search Google Scholar
    • Export Citation

  • Crocker, R., and Mittermaier M. P. , 2013: Exploratory use of a satellite cloud mask to verify NWP models. Meteor. Appl., 20, 197–205.

    • Search Google Scholar
    • Export Citation

  • Cullen, M. P. J., and Brown A. R. , 2009: Large eddy simulation of the atmosphere on various scales. Philos. Trans. Roy. Soc. London,367A, 2947–2956, doi:10.1098/rsta.2008.0268.

  • Damrath, U., 2004: Verification against precipitation observations of a high density network - what did we learn? Int. Verification Methods Workshop, Montreal, QC, Canada, WMO. [Available online at http://www.cawcr.gov.au/projects/verification/#Damrath_2004.]

  • Davies, T., Cullen M. , Malcolm A. , Mawson M. , Staniforth A. , White A. , and Wood N. , 2005: A new dynamical core for the Met Office’s global and regional modelling of the atmosphere. Quart. J. Roy. Meteor. Soc., 131, 1759–1782.

    • Search Google Scholar
    • Export Citation

  • Ebert, E., 2008: Fuzzy verification of high-resolution gridded forecasts: A review and proposed framework. Meteor. Appl., 15, 51–64.

    • Search Google Scholar
    • Export Citation

  • Edwards, J. M., and Slingo A. , 1996: Studies with a flexible new radiation code. I: Choosing a configuration for a large scale model. Quart. J. Roy. Meteor. Soc., 122, 689–719.

    • Search Google Scholar
    • Export Citation

  • Epstein, E., 1969: A scoring system for probability forecasts of ranked probabilities. J. Appl. Meteor., 8, 985–987.

  • Geleyn, J.-F., and Hollingsworth A. , 1979: An economical analytical method for the computation of the interaction between scattering and line absorption of radiation. Beitr. Phys. Atmos., 52, 1–16.

    • Search Google Scholar
    • Export Citation

  • Gilleland, E., Ahijevych D. , Brown B. , and Ebert E. , 2009: Intercomparison of spatial forecast verification methods. Wea. Forecasting, 24, 1416–1430.

    • Search Google Scholar
    • Export Citation

  • Gregory, D., and Rowntree P. , 1990: A mass flux convection scheme with representation of cloud ensemble characteristics and stability-dependent closure. Mon. Wea. Rev., 118, 1483–1506.

    • Search Google Scholar
    • Export Citation

  • Hamill, T., 1999: Hypothesis tests for evaluating numerical precipitation forecasts. Wea. Forecasting, 14, 155–167.

  • Hennessey, J., 1977: Some aspects of wind power statistics. J. Appl. Meteor., 16, 119–128.

  • Hersbach, H., 2000: Decomposition of the continuous ranked probability score for ensemble prediction systems. Wea. Forecasting, 15, 559–570.

    • Search Google Scholar
    • Export Citation

  • Hewitt, H. T., Copsey D. , Culverwell I. D. , Harris C. M. , Hill R. S. R. , Keen A. B. , McLaren A. J. , and Hunke E. C. , 2011: Design and implementation of the infrastructure of HadGEM3: The next-generation Met Office climate modelling system. Geosci. Model Dev., 4, 223–253.

    • Search Google Scholar
    • Export Citation

  • Hogan, R., O’Connor E. , and Illingworth A. , 2009: Verification of cloud-fraction forecasts. Quart. J. Roy. Meteor. Soc., 135, 1494–1511.

    • Search Google Scholar
    • Export Citation

  • Hohenegger, C., and Schär C. , 2007: Atmospheric predictability at synoptic versus cloud-resolving scales. Bull. Amer. Meteor. Soc., 88, 1783–1793.

    • Search Google Scholar
    • Export Citation

  • Illingworth, A. J., and Coauthors, 2007: Cloudnet—Continuous evaluation of cloud profiles in seven operational models using ground-based observations. Bull. Amer. Meteor. Soc., 88, 883–898.

    • Search Google Scholar
    • Export Citation

  • Johnson, A., and Wang X. , 2012: Verification and calibration of neighborhood and object-based probabilistic precipitation forecasts from a multimodel convection-allowing ensemble. Mon. Wea. Rev., 140, 3054–3077.

    • Search Google Scholar
    • Export Citation

  • Jolliffe, I. T., and Stephenson D. B. , Eds., 2003: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. John Wiley and Sons, 240 pp.

    • Search Google Scholar
    • Export Citation

  • Lean, H., Clark P. , Dixon M. , Roberts N. , Fitch A. , Forbes R. , and Halliwell C. , 2008: Characteristics of high-resolution versions of the Met Office Unified Model for forecasting convection over the United Kingdom. Mon. Wea. Rev., 136, 3408–3424.

    • Search Google Scholar
    • Export Citation

  • Lock, A., Brown A. , Bush M. , Martin G. , and Smith R. , 2000: A new boundary layer mixing scheme. Part I: Scheme description and single-column model tests. Mon. Wea. Rev., 128, 3187–3199.

    • Search Google Scholar
    • Export Citation

  • Lorenz, E., 1969: Atmospheric predictability as revealed by naturally occurring analogues. J. Atmos. Sci., 26, 636–646.

  • Mass, C., Ovens D. , Westrick K. , and Colle B. , 2002: Does increasing horizontal resolution produce more skillful forecasts? The results of two years of real-time numerical weather prediction over the Pacific Northwest. Bull. Amer. Meteor. Soc., 83, 407–430.

    • Search Google Scholar
    • Export Citation

  • Matheson, J., and Winkler R. , 1976: Scoring rules for continuous probability distributions. Manage. Sci., 22, 1087–1095.

  • Mittermaier, M., 2008: The potential impact of using persistence as a reference forecast on perceived forecast skill. Wea. Forecasting, 23, 1022–1031.

    • Search Google Scholar
    • Export Citation

  • Mittermaier, M., 2012: A critical assessment of surface cloud observations and their use for verifying cloud forecasts. Quart. Jour. Roy. Meteor. Soc.,138, 1794–1807, doi:10.1002/qj.1918.

  • Mittermaier, M., and Bullock R. , 2013: Using MODE to explore the spatial and temporal characteristics of cloud cover forecasts from high-resolution NWP models. Meteor. Appl., 20, 187–196.

    • Search Google Scholar
    • Export Citation

  • Mittermaier, M., Roberts N. , and Thompson S. , 2013: A long-term assessment of precipitation forecast skill using the fractions skill score. Meteor. Appl.,20, 176–186, doi:10.1002/met.296.

  • Murphy, A., 1970: The ranked probability score and the probability score: A comparison. Mon. Wea. Rev., 98, 917–924.

  • Murphy, A., 1973: A new vector partition of the probability score. J. Appl. Meteor., 12, 595–600.

  • Räisänen, P., 1998: Effective longwave cloud fraction and maximum-random overlap of clouds: A problem and a solution. Mon. Wea. Rev., 126, 3336–3340.

    • Search Google Scholar
    • Export Citation

  • Roberts, N., 2003: Precipitation diagnostics for a high resolution forecasting system: Stage 3 report from the Storm Scale Modelling project. Met Office Tech. Rep. 423, 45 pp.

  • Roberts, N., and Lean H. , 2008: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev., 136, 78–97.

    • Search Google Scholar
    • Export Citation

  • Schwartz, C., Kain J. , Weiss S. , Xue D. B. M. , Kong F. , Thomas K. , Levit J. , and Coniglio M. , 2009: Next-day convection-allowing WRF model guidance: A second look at 2-km versus 4-km grid spacing. Mon. Wea. Rev., 137, 3351–3372.

    • Search Google Scholar
    • Export Citation

  • Smith, R. N. B., 1990: A scheme for predicting layer clouds and their water content in a general circulation model. Quart. J. Roy. Meteor. Soc., 116, 435–460.

    • Search Google Scholar
    • Export Citation

  • Smith, S., Vosper S. , and Field P. , 2014: Sensitivity of orographic precipitation enhancement to horizontal resolution in the operational Met Office weather forecasts. Meteor. Appl.,doi:10.1002/met.1352, in press.

  • Theis, S., Hense A. , and Damrath U. , 2005: Probabilistic precipitation forecasts from a deterministic model: A pragmatic approach. Meteor. Appl., 12, 257–268.

    • Search Google Scholar
    • Export Citation

  • Tustison, B., Harris D. , and Foufoula-Georgiou E. , 2001: Scale issues in verification of precipitation forecasts. J. Geophys. Res., 106 (D11), 11 775–11 784.

    • Search Google Scholar
    • Export Citation

  • Wilks, D., 2006: Statistical Methods in Atmospheric Sciences. 2nd ed. Academic Press, 627 pp.

  • World Meteorological Organization, 2008: Guide to meteorological instruments and methods of observation. 7th ed. WMO-8. [Available online at http://www.wmo.int/pages/prog/gcos/documents/gruanmanuals/CIMO/CIMO_Guide-7th_Edition-2008.pdf.]

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 2082 1265 87
PDF Downloads 949 232 16