Object-Based Metrics for Forecast Verification of Convective Development with Geostationary Satellite Data

Martin Rempel Leibniz Institute for Tropospheric Research, Leipzig, Germany

Search for other papers by Martin Rempel in
Current site
Google Scholar
PubMed
Close
,
Fabian Senf Leibniz Institute for Tropospheric Research, Leipzig, Germany

Search for other papers by Fabian Senf in
Current site
Google Scholar
PubMed
Close
, and
Hartwig Deneke Leibniz Institute for Tropospheric Research, Leipzig, Germany

Search for other papers by Hartwig Deneke in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Object-based metrics are adapted and applied to geostationary satellite observations with the evaluation of cloud forecasts in convective situations as the goal. Forecasts of the convection-permitting German-focused Consortium for Small-Scale Modeling (COSMO-DE) numerical model are transformed into synthetic observations using the RTTOV radiative transfer model, and contrasted with the corresponding real observations. Threshold-based segmentation techniques are applied to the fields for object identification. The statistical properties of the traditional measures cold cloud cover and average brightness temperature amplitude are contrasted to object-based metrics of spatial aggregation and object structure. Based on 59 case days from the summer half-years between 2012 and 2014, a variance decomposition technique is applied to the time series of the metrics to identify deficits in day-to-day, diurnal, and weather-regime-related variability of cold cloud characteristics in the forecasts. Furthermore, sensitivities of the considered metrics are discussed, which result from uncertainties in the satellite forward operator and from the choice of parameters in the object identification techniques.

Denotes content that is immediately available upon publication as open access.

Current affiliation: Deutscher Wetterdienst, Offenbach, Germany.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Martin Rempel, martin.rempel@dwd.de

Abstract

Object-based metrics are adapted and applied to geostationary satellite observations with the evaluation of cloud forecasts in convective situations as the goal. Forecasts of the convection-permitting German-focused Consortium for Small-Scale Modeling (COSMO-DE) numerical model are transformed into synthetic observations using the RTTOV radiative transfer model, and contrasted with the corresponding real observations. Threshold-based segmentation techniques are applied to the fields for object identification. The statistical properties of the traditional measures cold cloud cover and average brightness temperature amplitude are contrasted to object-based metrics of spatial aggregation and object structure. Based on 59 case days from the summer half-years between 2012 and 2014, a variance decomposition technique is applied to the time series of the metrics to identify deficits in day-to-day, diurnal, and weather-regime-related variability of cold cloud characteristics in the forecasts. Furthermore, sensitivities of the considered metrics are discussed, which result from uncertainties in the satellite forward operator and from the choice of parameters in the object identification techniques.

Denotes content that is immediately available upon publication as open access.

Current affiliation: Deutscher Wetterdienst, Offenbach, Germany.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Martin Rempel, martin.rempel@dwd.de
Save
  • Akkermans, T., and Coauthors, 2012: Regime-dependent evaluation of accumulated precipitation in COSMO. Theor. Appl. Climatol., 108, 3952, doi:10.1007/s00704-011-0502-0.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Baldauf, M., A. Seifert, J. Förstner, D. Majewski, M. Raschendorfer, and T. Reinhardt, 2011: Operational convective-scale numerical weather prediction with the COSMO model: Description and sensitivities. Mon. Wea. Rev., 139, 38873905, doi:10.1175/MWR-D-10-05013.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Baldwin, M. E., and J. S. Kain, 2006: Sensitivity of several performance measures to displacement error, bias, and event frequency. Wea. Forecasting, 21, 636648, doi:10.1175/WAF933.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Baur, F., 1963: Grosswetterkunde und langfristige Witterungsvorhersage. Akademische Verlagsgesellschaft, 91 pp.

  • Bikos, D., and Coauthors, 2012: Synthetic satellite imagery for real-time high-resolution model evaluation. Wea. Forecasting, 27, 784795, doi:10.1175/WAF-D-11-00130.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Böhme, T., and Coauthors, 2011: Long-term evaluation of COSMO forecasting using combined observational data of the GOP period. Meteor. Z., 20, 119132, doi:10.1127/0941-2948/2011/0225.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brown, M. B., and A. B. Forsythe, 1974: Robust tests for the equality of variances. J. Amer. Stat. Assoc., 69, 364367, doi:10.1080/01621459.1974.10482955.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Casati, B., and Coauthors, 2008: Forecast verification: Current status and future directions. Meteor. Appl., 15, 318, doi:10.1002/met.52.

  • Crocker, R., and M. Mittermaier, 2013: Exploratory use of a satellite cloud mask to verify NWP models. Meteor. Appl., 20, 197205, doi:10.1002/met.1384.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Davis, C. A., B. G. Brown, and R. Bullock, 2006a: Object-based verification of precipitation forecasts. Part I: Methodology and application to mesoscale rain areas. Mon. Wea. Rev., 134, 17721784, doi:10.1175/MWR3145.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Davis, C. A., B. G. Brown, and R. Bullock, 2006b: Object-based verification of precipitation forecasts. Part II: Application to convective rain systems. Mon. Wea. Rev., 134, 17851795, doi:10.1175/MWR3146.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Davis, C. A., B. G. Brown, R. Bullock, and J. Halley-Gotway, 2009: The Method for Object-Based Diagnostic Evaluation (MODE) applied to numerical forecasts from the 2005 NSSL/SPC Spring Program. Wea. Forecasting, 24, 12521267, doi:10.1175/2009WAF2222241.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Derrien, M., and H. Le Gléau, 2005: MSG/SEVIRI cloud mask and type from SAFNWC. Int. J. Remote Sens., 26, 47074732, doi:10.1080/01431160500166128.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Dotzek, N., P. Groenemeijer, B. Feuerstein, and A. M. Holzer, 2009: Overview of ESSL’s severe convective storms research using the European Severe Weather Database ESWD. Atmos. Res., 93, 575586, doi:10.1016/j.atmosres.2008.10.020.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ebert, E. E., 2008: Fuzzy verification of high-resolution gridded forecasts: A review and proposed framework. Meteor. Appl., 15, 5164, doi:10.1002/met.25.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ebert, E. E., and J. L. McBride, 2000: Verification of precipitation in weather systems: Determination of systematic errors. J. Hydrol., 239, 179202, doi:10.1016/S0022-1694(00)00343-7.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ebert, E. E., and W. A. Gallus Jr., 2009: Toward better understanding of the contiguous rain area (CRA) method for spatial forecast verification. Wea. Forecasting, 24, 14011415, doi:10.1175/2009WAF2222252.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Eikenberg, S., C. Köhler, A. Seifert, and S. Crewell, 2015: How microphysical choices affect simulated infrared brightness temperatures. Atmos. Res., 156, 6779, doi:10.1016/j.atmosres.2014.12.010.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Feidas, H., and A. Giannakos, 2012: Classifying convective and stratiform rain using multispectral infrared Meteosat Second Generation satellite data. Theor. Appl. Climatol., 108, 613630, doi:10.1007/s00704-011-0557-y.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gilleland, E., D. Ahijevych, B. G. Brown, B. Casati, and E. E. Ebert, 2009: Intercomparison of spatial forecast verification methods. Wea. Forecasting, 24, 14161430, doi:10.1175/2009WAF2222269.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hanley, K., D. Kirshbaum, N. Roberts, and G. Leoncini, 2013: Sensitivities of a squall line over central Europe in a convective-scale ensemble. Mon. Wea. Rev., 141, 112133, doi:10.1175/MWR-D-12-00013.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jankov, I., and Coauthors, 2011: An evaluation of five ARW-WRF microphysics schemes using synthetic GOES imagery for an atmospheric river event affecting the California coast. J. Hydrometeor., 12, 618633, doi:10.1175/2010JHM1282.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kann, A., I. Meirold-Mautner, F. Schmid, G. Kirchengast, J. Fuchsberger, V. Meyer, L. Tüchler, and B. Bica, 2015: Evaluation of high-resolution precipitation analyses using a dense station network. Hydrol. Earth Syst. Sci., 19, 15471559, doi:10.5194/hess-19-1547-2015.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Keller, M., O. Fuhrer, J. Schmidli, M. Stengel, R. Stöckli, and C. Schär, 2016: Evaluation of convection-resolving models using satellite data: The diurnal cycle of summer convection over the Alps. Meteor. Z., 25, 165179, doi:10.1127/metz/2015/0715.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kidd, C., E. Dawkins, and G. Huffman, 2013: Comparison of precipitation derived from the ECMWF operational forecast model and satellite precipitation datasets. J. Hydrometeor., 14, 14631482, doi:10.1175/JHM-D-12-0182.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lindstedt, D., P. Lind, E. Kjellström, and C. Jones, 2015: A new regional climate model operating at the meso-gamma scale: performance over Europe. Tellus, 67A, 24138, doi:10.3402/tellusa.v67.24138.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Machado, L. A. T., and J.-P. Chaboureau, 2015: Effect of turbulence parameterization on assessment of cloud organization. Mon. Wea. Rev., 143, 32463262, doi:10.1175/MWR-D-14-00393.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Morcrette, J.-J., 1991: Evaluation of model-generated cloudiness: Satellite-observed and model-generated diurnal variability of brightness temperature. Mon. Wea. Rev., 119, 12051224, doi:10.1175/1520-0493(1991)119<1205:EOMGCS>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281293, doi:10.1175/1520-0434(1993)008<0281:WIAGFA>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Negri, R. G., L. A. T. Machado, S. English, and M. Forsythe, 2014: Combining a cloud-resolving model with satellite for cloud process model simulation validation. J. Appl. Meteor. Climatol., 53, 521533, doi:10.1175/JAMC-D-12-0178.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Pfeifer, M., and Coauthors, 2010: Validating precipitation forecasts using remote sensor synergy: A case study approach. Meteor. Z., 19, 601617, doi:10.1127/0941-2948/2010/0487.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Roca, R., and V. Ramanathan, 2000: Scale dependence of monsoonal convective systems over the Indian Ocean. J. Climate, 13, 12861298, doi:10.1175/1520-0442(2000)013<1286:SDOMCS>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rossa, A., P. Nurmi, and E. Ebert, 2008: Overview of methods for the verification of quantitative precipitation forecasts. Precipitation: Advances in Measurement, Estimation and Prediction, S. C. Michaelides, Ed., Springer, 419–452.

    • Crossref
    • Export Citation
  • Saunders, R., M. Matricardi, and P. Brunel, 1999: An improved fast radiative transfer model for assimilation of satellite radiance observations. Quart. J. Roy. Meteor. Soc., 125, 14071425, doi:10.1002/qj.1999.49712555615.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Schmetz, J., P. Pili, S. Tjemkes, D. Just, J. Kerkmann, S. Rota, and A. Ratier, 2002: An introduction to Meteosat Second Generation (MSG). Bull. Amer. Meteor. Soc., 83, 977992, doi:10.1175/1520-0477(2002)083<0977:AITMSG>2.3.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Schneider, S., Y. Wang, W. Wagner, and J.-F. Mahfouf, 2014: Impact of ASCAT soil moisture assimilation on regional precipitation forecasts: A case study for Austria. Mon. Wea. Rev., 142, 15251541, doi:10.1175/MWR-D-12-00311.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Senf, F., and H. Deneke, 2017: Uncertainties in synthetic Meteosat SEVIRI infrared brightness temperatures in the presence of cirrus clouds and implications for evaluation of cloud microphysics. Atmos. Res., 183, 113129, doi:10.1016/j.atmosres.2016.08.012.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tobin, I., S. Bony, and R. Roca, 2012: Observational evidence for relationships between the degree of aggregation of deep convection, water vapor, surface fluxes, and radiation. J. Climate, 25, 68856904, doi:10.1175/JCLI-D-11-00258.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Weniger, M., and P. Friederichs, 2016: Using the SAL technique for spatial verification of cloud processes: A sensitivity analysis. J. Appl. Meteor. Climatol., 55, 20912108, doi:10.1175/JAMC-D-15-0311.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wernli, H., M. Paulat, M. Hagen, and C. Frei, 2008: SAL—A novel quality measure for the verification of quantitative precipitation forecasts. Mon. Wea. Rev., 136, 44704487, doi:10.1175/2008MWR2415.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2011: Statistical Methods in the Atmospheric Sciences. 3rd ed. Elsevier, 676 pp.

    • Crossref
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1392 843 60
PDF Downloads 463 70 2