Regional Cloud Forecast Verification Using Standard, Spatial, and Object-Oriented Methods

H. Christophersen aNaval Research Laboratory, Monterey, California

Search for other papers by H. Christophersen in
Current site
Google Scholar
PubMed
Close
,
J. Nachamkin aNaval Research Laboratory, Monterey, California

Search for other papers by J. Nachamkin in
Current site
Google Scholar
PubMed
Close
, and
W. Davis bDeVine, Monterey, California

Search for other papers by W. Davis in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

This study assesses the accuracy of the Coupled Ocean–Atmosphere Mesoscale Prediction System (COAMPS) forecasts for clouds within stable and unstable environments (thereafter refers as “stable” and “unstable” clouds). This evaluation is conducted by comparing these forecasts against satellite retrievals through a combination of traditional, spatial, and object-based methods. To facilitate this assessment, the Model Evaluation Tools (MET) community tool is employed. The findings underscore the significance of fine-tuning the MET parameters to achieve a more accurate representation of the features under scrutiny. The study’s results reveal that when employing traditional pointwise statistics (e.g., frequency bias and equitable threat score), there is consistency in the results whether calculated from Method for Object-Based Diagnostic Evaluation (MODE)-based objects or derived from the complete fields. Furthermore, the object-based statistics offer valuable insights, indicating that COAMPS generally predicts cloud object locations accurately, though the spread of these predicted locations tends to increase with time. It tends to overpredict the object area for unstable clouds while underpredicting it for stable clouds over time. These results are in alignment with the traditional pointwise bias scores for the entire grid. Overall, the spatial metrics provided by the object-based verification methods emerge as crucial and practical tools for the validation of cloud forecasts.

Significance Statement

As the general Navy meteorological and oceanographic (METOC) community engages in collaboration with the broader scientific community, our goal is to harness community tools like MET for the systematic evaluation of weather forecasts, with a specific focus on variables crucial to the Navy. Clouds, given their significant impact on visibility, hold particular importance in our investigations. Cloud forecasts pose unique challenges, primarily attributable to the intricate physics governing cloud development and the complexity of representing these processes within numerical models. Cloud observations are also constrained by limitations, arising from both top-down satellite measurements and bottom-up ground-based measurements. This study illustrates that, with a comprehensive understanding of community tools, cloud forecasts can be consistently verified. This verification encompasses traditional evaluation methods, measuring general qualities such as bias and root-mean-squared error, as well as newer techniques like spatial and object-based methods designed to account for displacement errors.

© 2024 American Meteorological Society. This published article is licensed under the terms of the default AMS reuse license. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Hui Christophersen, hui.christophersen@nrlmry.navy.mil

Abstract

This study assesses the accuracy of the Coupled Ocean–Atmosphere Mesoscale Prediction System (COAMPS) forecasts for clouds within stable and unstable environments (thereafter refers as “stable” and “unstable” clouds). This evaluation is conducted by comparing these forecasts against satellite retrievals through a combination of traditional, spatial, and object-based methods. To facilitate this assessment, the Model Evaluation Tools (MET) community tool is employed. The findings underscore the significance of fine-tuning the MET parameters to achieve a more accurate representation of the features under scrutiny. The study’s results reveal that when employing traditional pointwise statistics (e.g., frequency bias and equitable threat score), there is consistency in the results whether calculated from Method for Object-Based Diagnostic Evaluation (MODE)-based objects or derived from the complete fields. Furthermore, the object-based statistics offer valuable insights, indicating that COAMPS generally predicts cloud object locations accurately, though the spread of these predicted locations tends to increase with time. It tends to overpredict the object area for unstable clouds while underpredicting it for stable clouds over time. These results are in alignment with the traditional pointwise bias scores for the entire grid. Overall, the spatial metrics provided by the object-based verification methods emerge as crucial and practical tools for the validation of cloud forecasts.

Significance Statement

As the general Navy meteorological and oceanographic (METOC) community engages in collaboration with the broader scientific community, our goal is to harness community tools like MET for the systematic evaluation of weather forecasts, with a specific focus on variables crucial to the Navy. Clouds, given their significant impact on visibility, hold particular importance in our investigations. Cloud forecasts pose unique challenges, primarily attributable to the intricate physics governing cloud development and the complexity of representing these processes within numerical models. Cloud observations are also constrained by limitations, arising from both top-down satellite measurements and bottom-up ground-based measurements. This study illustrates that, with a comprehensive understanding of community tools, cloud forecasts can be consistently verified. This verification encompasses traditional evaluation methods, measuring general qualities such as bias and root-mean-squared error, as well as newer techniques like spatial and object-based methods designed to account for displacement errors.

© 2024 American Meteorological Society. This published article is licensed under the terms of the default AMS reuse license. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Hui Christophersen, hui.christophersen@nrlmry.navy.mil
Save
  • Alexander, S. P., and A. Protat, 2018: Cloud properties observed from the surface and by satellite at the northern edge of the Southern Ocean. J. Geophys. Res. Atmos., 123, 443456, https://doi.org/10.1002/2017JD026552.

    • Search Google Scholar
    • Export Citation
  • Bani Shahabadi, M. B., Y. Huang, L. Garand, S. Heilliette, and P. Yang, 2016: Validation of a weather forecast model at radiance level against satellite observations allowing quantification of temperature, humidity, and cloud-related biases. J. Adv. Model. Earth Syst., 8, 14531467, https://doi.org/10.1002/2016MS000751.

    • Search Google Scholar
    • Export Citation
  • Bodas-Salcedo, A., and Coauthors, 2011: COSP: Satellite simulation software for model assessment. Bull. Amer. Meteor. Soc., 92, 10231043, https://doi.org/10.1175/2011BAMS2856.1.

    • Search Google Scholar
    • Export Citation
  • Cai, H., and R. E. Dumais Jr., 2015: Object-based evaluation of a numerical weather prediction model’s performance through forecast storm characteristic analysis. Wea. Forecasting, 30, 14511468, https://doi.org/10.1175/WAF-D-15-0008.1.

    • Search Google Scholar
    • Export Citation
  • Chen, S., and Coauthors, 2003: COAMPS version 3 model description-general theory and equations. Naval Research Laboratory Tech. Note NRL/PU/7500-03448, 143 pp.

  • Daley, R., and E. Barker, 2001: NAVDAS Source Book 2001: NRL atmospheric variational data assimilation system. Naval Research Laboratory Tech. Note NRL/PU/7530-01-441, 163 pp.

  • Davis, C. A., B. G. Brown, and R. Bullock, 2006: Object-based verification of precipitation forecasts. Part II: Application to convective rain systems. Mon. Wea. Rev., 134, 17851795, https://doi.org/10.1175/MWR3146.1.

    • Search Google Scholar
    • Export Citation
  • Davis, C. A., B. G. Brown, R. Bullock, and J. Halley-Gotway, 2009: The Method for Object-based Diagnostic Evaluation (MODE) applied to numerical forecasts from the 2005 NSSL/SPC Spring Program. Wea. Forecasting, 24, 12521267, https://doi.org/10.1175/2009WAF2222241.1.

    • Search Google Scholar
    • Export Citation
  • Ebert, E. E., and J. L. McBride, 2000: Verification of precipitation in weather systems: Determination of systematic errors. J. Hydrol., 239, 179202, https://doi.org/10.1016/S0022-1694(00)00343-7.

    • Search Google Scholar
    • Export Citation
  • Griffin, S. M., and Coauthors, 2021: Evaluating the impact of planetary boundary layer, land surface model, and microphysics parameterization schemes on cold cloud objects in simulated GOES-16 brightness temperatures. J. Geophys. Res. Atmos., 126, e2021JD034709, https://doi.org/10.1029/2021JD034709.

    • Search Google Scholar
    • Export Citation
  • Huffman, G. J., and Coauthors, 2019: NASA Global Precipitation Measurement (GPM) Integrated Multi-satellitE Retrievals for GPM (IMERG). NASA Algorithm Theoretical Basis Doc., version 06, NASA, 38 pp., https://gpm.nasa.gov/sites/default/files/document_files/IMERG_ATBD_V06.pdf.

  • Johnson, A., and X. Wang, 2013: Object-based evaluation of a storm-scale ensemble during the 2009 NOAA Hazardous Weather Testbed Spring Experiment. Mon. Wea. Rev., 141, 10791098, https://doi.org/10.1175/MWR-D-12-00140.1.

    • Search Google Scholar
    • Export Citation
  • Johnson, B. T., C. Dang, P. Stegmann, Q. Liu, I. Moradi, and T. Auligne, 2023: The Community Radiative Transfer Model (CRTM): Community-focused collaborative model development accelerating research to operations. Bull. Amer. Meteor. Soc., 104, E1817E1830, https://doi.org/10.1175/BAMS-D-22-0015.1.

    • Search Google Scholar
    • Export Citation
  • Lack, S. A., G. L. Limpert, and N. I. Fox, 2010: An object-oriented multiscale verification scheme. Wea. Forecasting, 25, 7992, https://doi.org/10.1175/2009WAF2222245.1.

    • Search Google Scholar
    • Export Citation
  • McErlich, C., A. McDonald, A. Schuddeboom, and I. Silber, 2021: Comparing satellite- and ground-based observations of cloud occurrence over high southern latitudes. J. Geophys. Res. Atmos., 126, e2020JD033607, https://doi.org/10.1029/2020JD033607.

    • Search Google Scholar
    • Export Citation
  • Minnis, P., and Coauthors, 2021: CERES MODIS cloud product retrievals for edition—4. Part I: Algorithm changes. IEEE Trans. Geosci. Remote Sens., 59, 27442780, https://doi.org/10.1109/TGRS.2020.3008866.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281293, https://doi.org/10.1175/1520-0434(1993)008<0281:WIAGFA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Nachamkin, J. E., 2004: Mesoscale verification using meteorological composites. Mon. Wea. Rev., 132, 941955, https://doi.org/10.1175/1520-0493(2004)132<0941:MVUMC>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Nachamkin, J. E., A. Bienkowski, R. Bankert, K. Pattipati, D. Sidoti, M. Surratt, J. Gull, and C. Nguyen, 2022: Classification and evaluation of stable and unstable cloud forecasts. Mon. Wea. Rev., 150, 8198, https://doi.org/10.1175/MWR-D-21-0056.1.

    • Search Google Scholar
    • Export Citation
  • Peel, S., and L. J. Wilson, 2008: Modeling the distribution of precipitation forecasts from the Canadian Ensemble Prediction System using kernel density estimation. Wea. Forecasting, 23, 575595, https://doi.org/10.1175/2007WAF2007023.1.

    • Search Google Scholar
    • Export Citation
  • Roberts, N. M., and H. W. Lean, 2008: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev., 136, 7897, https://doi.org/10.1175/2007MWR2123.1.

    • Search Google Scholar
    • Export Citation
  • Roebber, P. J., 2009: Visualizing multiple measures of forecast quality. Wea. Forecasting, 24, 601608, https://doi.org/10.1175/2008WAF2222159.1.

    • Search Google Scholar
    • Export Citation
  • Rutledge, S. A., and P. V. Hobbs, 1983: The mesoscale and microscale structure and organization of clouds and precipitation in midlatitude cyclones. VIII: A model for the “seeder-feeder” process in warm-frontal rainbands. J. Atmos. Sci., 40, 11851206, https://doi.org/10.1175/1520-0469(1983)040%3C1185:TMAMSA%3E2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Rutledge, S. A., and P. V. Hobbs, 1984: The mesoscale and microscale structure and organization of clouds and precipitation in midlatitude cyclones. XII: A diagnostic modeling study of precipitation development in narrow cold-frontal rainbands. J. Atmos. Sci., 40, 11851206, https://doi.org/10.1175/1520-0469(1984)041%3C2949:TMAMSA%3E2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Saunders, R., and Coauthors, 2020: RTTOV-13 science and validation report. NWPSAF-MO-TV-046, 106 pp., https://nwp-saf.eumetsat.int/site/download/documentation/rtm/docs_rttov13/rttov13_svr.pdf.

  • Skamarock, W. C., 2004: Evaluating mesoscale NWP models using kinetic energy spectra. Mon. Wea. Rev., 132, 30193032, https://doi.org/10.1175/MWR2830.1.

    • Search Google Scholar
    • Export Citation
  • Skinner, P. S., and Coauthors, 2018: Object-based verification of a prototype warn-on-forecast system. Wea. Forecasting, 33, 12251250, https://doi.org/10.1175/WAF-D-18-0020.1.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences: An Introduction. International Geophysics Series, Vol. 59, Elsevier, 467 pp.

  • Wolff, J. K., M. Harrold, T. Fowler, J. H. Gotway, L. Nance, and B. G. Brown, 2014: Beyond the basics: Evaluating model-based precipitation forecasts using traditional, spatial, and object-based methods. Wea. Forecasting, 29, 14511472, https://doi.org/10.1175/WAF-D-13-00135.1.

    • Search Google Scholar
    • Export Citation
  • Zhang, Y., S. A. Klein, J. Boyle, and G. G. Mace, 2010: Evaluation of tropical cloud and precipitation statistics of community atmosphere model version 3 using CloudSat and CALIPSO data. J. Geophys. Res., 115, D12205, https://doi.org/10.1029/2009JD012006.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 180 180 17
Full Text Views 105 105 2
PDF Downloads 128 128 3