Proactive QC: A Fully Flow-Dependent Quality Control Scheme Based on EFSO

Daisuke Hotta University of Maryland, College Park, College Park, Maryland, and Japan Meteorological Agency, Tokyo, Japan

Search for other papers by Daisuke Hotta in
Current site
Google Scholar
PubMed
Close
,
Tse-Chun Chen University of Maryland, College Park, College Park, Maryland

Search for other papers by Tse-Chun Chen in
Current site
Google Scholar
PubMed
Close
,
Eugenia Kalnay University of Maryland, College Park, College Park, Maryland

Search for other papers by Eugenia Kalnay in
Current site
Google Scholar
PubMed
Close
,
Yoichiro Ota Japan Meteorological Agency, Tokyo, Japan

Search for other papers by Yoichiro Ota in
Current site
Google Scholar
PubMed
Close
, and
Takemasa Miyoshi University of Maryland, College Park, College Park, Maryland, and RIKEN Advanced Institute for Computational Science, Kobe, Japan

Search for other papers by Takemasa Miyoshi in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Despite dramatic improvements over the last decades, operational NWP forecasts still occasionally suffer from abrupt drops in their forecast skill. Such forecast skill “dropouts” may occur even in a perfect NWP system because of the stochastic nature of NWP but can also result from flaws in the NWP system. Recent studies have shown that dropouts occur due not to a model’s deficiencies but to misspecified initial conditions, suggesting that they could be mitigated by improving the quality control (QC) system so that the observation-minus-background (O-B) innovations that would degrade a forecast can be detected and rejected. The ensemble forecast sensitivity to observations (EFSO) technique enables for the quantification of how much each observation has improved or degraded the forecast. A recent study has shown that 24-h EFSO can detect detrimental O-B innovations that caused regional forecast skill dropouts and that the forecast can be improved by not assimilating them. Inspired by that success, a new QC method is proposed, termed proactive QC (PQC), that detects detrimental innovations 6 h after the analysis using EFSO and then repeats the analysis and forecast without using them. PQC is implemented and tested on a lower-resolution version of NCEP’s operational global NWP system. It is shown that EFSO is insensitive to the choice of verification and lead time (24 or 6 h) and that PQC likely improves the analysis, as attested to by forecast improvements of up to 5 days and beyond. Strategies for reducing the computational costs and further optimizing the observation rejection criteria are also discussed.

Denotes content that is immediately available upon publication as open access.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Daisuke Hotta, dhotta@mri-jma.go.jp

This article is included in the Sixth WMO Data Assimilation Symposium Special Collection.

Abstract

Despite dramatic improvements over the last decades, operational NWP forecasts still occasionally suffer from abrupt drops in their forecast skill. Such forecast skill “dropouts” may occur even in a perfect NWP system because of the stochastic nature of NWP but can also result from flaws in the NWP system. Recent studies have shown that dropouts occur due not to a model’s deficiencies but to misspecified initial conditions, suggesting that they could be mitigated by improving the quality control (QC) system so that the observation-minus-background (O-B) innovations that would degrade a forecast can be detected and rejected. The ensemble forecast sensitivity to observations (EFSO) technique enables for the quantification of how much each observation has improved or degraded the forecast. A recent study has shown that 24-h EFSO can detect detrimental O-B innovations that caused regional forecast skill dropouts and that the forecast can be improved by not assimilating them. Inspired by that success, a new QC method is proposed, termed proactive QC (PQC), that detects detrimental innovations 6 h after the analysis using EFSO and then repeats the analysis and forecast without using them. PQC is implemented and tested on a lower-resolution version of NCEP’s operational global NWP system. It is shown that EFSO is insensitive to the choice of verification and lead time (24 or 6 h) and that PQC likely improves the analysis, as attested to by forecast improvements of up to 5 days and beyond. Strategies for reducing the computational costs and further optimizing the observation rejection criteria are also discussed.

Denotes content that is immediately available upon publication as open access.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Daisuke Hotta, dhotta@mri-jma.go.jp

This article is included in the Sixth WMO Data Assimilation Symposium Special Collection.

Save
  • Alpert, J. C., D. L. Carlis, B. A. Ballish, and V. K. Kumar, 2009: Using pseudo RAOB observations to study GFS skill score dropouts. 23rd Conf. on Weather Analysis and Forecasting/19th Conf. on Numerical Weather Prediction, Omaha, NE, Amer. Meteor. Soc., 5A.6. [Available online at https://ams.confex.com/ams/23WAF19NWP/techprogram/paper_154268.htm.]

  • Anderson, E., and H. Järvinen, 1999: Variational quality control. Quart. J. Roy. Meteor. Soc., 125, 697722, doi:10.1002/qj.49712555416.

  • Bishop, C. H., and D. Hodyss, 2009a: Ensemble covariances adaptively localized with ECO‐RAP. Part 1: Tests on simple error models. Tellus, 61A, 8496, doi:10.1111/j.1600-0870.2008.00371.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., and D. Hodyss, 2009b: Ensemble covariances adaptively localized with ECO‐RAP. Part 2: A strategy for the atmosphere. Tellus, 61A, 97111, doi:10.1111/j.1600-0870.2008.00372.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Boukabara, S., and Coauthors, 2016: S4: An O2R/R2O infrastructure for optimizing satellite data utilization in NOAA numerical modeling systems: A step toward bridging the gap between research and operations. Bull. Amer. Meteor. Soc., 97, 23592378, doi:10.1175/BAMS-D-14-00188.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Buehner, M., 2005: Ensemble-derived stationary and flow-dependent background-error covariances: Evaluation in a quasi-operational NWP setting. Quart. J. Roy. Meteor. Soc., 131, 10131043, doi:10.1256/qj.04.15.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cardinali, C., 2009: Monitoring the observation impact on the short-range forecast. Quart. J. Roy. Meteor. Soc., 135, 239250, doi:10.1002/qj.366.

  • Carrassi, A., A. Trevisan, and F. Uboldi, 2007: Adaptive observations and assimilation in the unstable subspace by breeding on the data-assimilation system. Tellus, 59A, 101113, doi:10.1111/j.1600-0870.2006.00210.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Clayton, A., A. Lorenc, and D. Barker, 2013: Operational implementation of a hybrid ensemble/4D-Var global data assimilation system at the Met Office. Quart. J. Roy. Meteor. Soc., 139, 14451461, doi:10.1002/qj.2054.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Daescu, D. N., 2009: On the deterministic observation impact guidance: A geometrical perspective. Mon. Wea. Rev., 137, 35673574, doi:10.1175/2009MWR2954.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ehrendorfer, M., R. M. Errico, and K. D. Raeder, 1999: Singular-vector perturbation growth in a primitive equation model with moist physics. J. Atmos. Sci., 56, 16271648, doi:10.1175/1520-0469(1999)056<1627:SVPGIA>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gaspari, G., and S. E. Cohn, 1999: Construction of correlation functions in two and three dimensions. Quart. J. Roy. Meteor. Soc., 125, 723757, doi:10.1002/qj.49712555417.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gasperoni, N. A., and X. Wang, 2015: Adaptive localization for the ensemble-based observation impact estimate using regression confidence factors. Mon. Wea. Rev., 143, 19812000, doi:10.1175/MWR-D-14-00272.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Geer, A., 2016: Significance of changes in medium-range forecast scores. Tellus, 68A, 30 229, doi:10.3402/tellusa.v68.30229.

  • Gelaro, R., and Y. Zhu, 2009: Examination of observation impacts derived from observing system experiments (OSEs) and adjoint models. Tellus, 61A, 179193, doi:10.1111/j.1600-0870.2008.00388.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gelaro, R., R. H. Langland, S. Pellerin, and R. Todling, 2010: The THORPEX Observation Impact Intercomparison Experiment. Mon. Wea. Rev., 138, 40094025, doi:10.1175/2010MWR3393.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Holdaway, D., R. Errico, R. Gelaro, and J. G. Kim, 2014: Inclusion of linearized moist physics in NASA’s Goddard Earth Observing System data assimilation tools. Mon. Wea. Rev., 142, 414433, doi:10.1175/MWR-D-13-00193.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hunt, B. R., E. J. Kostelich, and I. Szunyogh, 2007: Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter. Physica D, 230, 112126, doi:10.1016/j.physd.2006.11.008.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ingleby, N. B., and A. C. Lorenc, 1993: Bayesian quality control using multivariate normal distribution. Quart. J. Roy. Meteor. Soc., 119, 11951225, doi:10.1002/qj.49711951316.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Isaksen, L., M. Fisher, E. Andersson, and J. Barkmeijer, 2005: The structure and realism of sensitivity perturbations and their interpretation as ‘Key Analysis Errors.’ Quart. J. Roy. Meteor. Soc., 131, 30533078, doi:10.1256/qj.04.99.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ishibashi, T., 2010: Optimization of error covariance matrices and estimation of observation data impact in the JMA global 4D-Var system. CASJSC WGNE Research Activities in Atmospheric and Oceanic Modelling, World Climate Research Programme Rep. 40, 1–11.

  • Kalnay, E., 2003: Atmospheric Modeling, Data Assimilation and Predictability. Cambridge University Press, 341 pp.

    • Crossref
    • Export Citation
  • Kalnay, E., Y. Ota, T. Miyoshi, and J. Liu, 2012: A simpler formulation of forecast sensitivity to observations: application to ensemble Kalman filters. Tellus, 64A, 18 462, doi:10.3402/tellusa.v64i0.18462.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Keyser, D., 2011: Observational data processing at NCEP. NOAA/NCEP/NWS/Environmental Modeling Center. [Available online at http://www.emc.ncep.noaa.gov/mmb/data_processing/data_processing/.]

  • Keyser, D., 2013: PREPBUFR processing at NCEP. NOAA/NCEP/NWS/Environmental Modeling Center. [Available online at http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/document.htm.]

  • Kleist, D. T., 2012: An evaluation of hybrid variational-ensemble data assimilation for the NCEP GFS. Ph.D. dissertation, University of Maryland, College Park, 149 pp. [Available online at http://hdl.handle.net/1903/13135.]

  • Kleist, D. T., and M. C. Morgan, 2005: Application of adjoint-derived forecast sensitivities to the 24–25 January 2000 U.S. east coast snowstorm. Mon. Wea. Rev., 133, 31483175, doi:10.1175/MWR3023.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kleist, D. T., and K. Ide, 2015a: An OSSE-based evaluation of hybrid variational–ensemble data assimilation for the NCEP GFS. Part I: System description and 3D-hybrid results. Mon. Wea. Rev., 143, 433451, doi:10.1175/MWR-D-13-00351.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kleist, D. T., and K. Ide, 2015b: An OSSE-based evaluation of hybrid variational–ensemble data assimilation for the NCEP GFS. Part II: 4DEnVar and hybrid variants. Mon. Wea. Rev., 143, 452470, doi:10.1175/MWR-D-13-00350.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kumar, K., J. C. Alpert, D. L. Carlis, and B. A. Ballish, 2009: Investigation of NCEP GFS model forecast skill “dropout” characteristics using the EBI index. 23rd Conf. on Weather Analysis and Forecasting/19th Conf. on Numerical Weather Prediction, Omaha, NE, Amer. Meteor. Soc., 13A.1. [Available online at https://ams.confex.com/ams/23WAF19NWP/techprogram/paper_154282.htm.]

  • Langland, R. H., and N. L. Baker, 2004: Estimation of observation impact using the NRL atmospheric variational data assimilation adjoint system. Tellus, 56A, 189201, doi:10.3402/tellusa.v56i3.14413.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Li, H., J. Liu, and E. Kalnay, 2010: Correction of “Estimating observation impact without adjoint model in an ensemble Kalman filter.” Quart. J. Roy. Meteor. Soc., 136, 16521654, doi:10.1002/qj.658.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lien, G.-Y., 2014: Ensemble assimilation of global large-scale precipitation. Ph.D. dissertation, University of Maryland, College Park, 165 pp. [Available online at http://hdl.handle.net/1903/15274.]

  • Liu, J., and E. Kalnay, 2008: Estimating observation impact without adjoint model in an ensemble Kalman filter. Quart. J. Roy. Meteor. Soc., 134, 13271335, doi:10.1002/qj.280.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, J., E. Kalnay, T. Miyoshi, and C. Cardinali, 2009: Analysis sensitivity calculation in an ensemble Kalman filter. Quart. J. Roy. Meteor. Soc., 135, 18421851, doi:10.1002/qj.511.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., 1981: A global three-dimensional multivariate statistical interpolation scheme. Mon. Wea. Rev., 109, 701721, doi:10.1175/1520-0493(1981)109<0701:AGTDMS>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., and R. T. Marriott, 2014: Forecast sensitivity to observations in the Met Office global numerical weather prediction system. Quart. J. Roy. Meteor. Soc., 140, 209223, doi:10.1002/qj.2122.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Onogi, K., 1998: A data quality control method using forecasted horizontal gradient and tendency in a NWP system: Dynamic QC. J. Meteor. Soc. Japan, 76, 497516, doi:10.2151/jmsj1965.76.4_497.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ota, Y., J. C. Derber, T. Miyoshi, and E. Kalnay, 2013: Ensemble-based observation impact estimates using the NCEP GFS. Tellus, 65A, 20 038, doi:10.3402/tellusa.v65i0.20038.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Pires, C., R. Vautard, and O. Talagrand, 1996: On extending the limits of variational assimilation in nonlinear chaotic systems. Tellus, 48A, 96121, doi:10.3402/tellusa.v48i1.11634.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rodwell, M. J., and Coauthors, 2013: Characteristics of occasional poor medium-range weather forecasts for Europe. Bull. Amer. Meteor. Soc., 94, 13931405, doi:10.1175/BAMS-D-12-00099.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Simmons, A., 2011: From observations to service delivery: Challenges and opportunities. WMO Bulletin, Vol. 60, No. 2, WMO, Geneva, Switzerland, 96–107. [Available online at https://public.wmo.int/en/bulletin/observations-service-delivery-challenges-and-opportunities.]

  • Tavolato, C., and L. Isaksen, 2015: On the use of a Huber norm for observation quality control in the ECMWF 4D-Var. Quart. J. Roy. Meteor. Soc., 141, 15141526, doi:10.1002/qj.2440.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Trevisan, A., and F. Uboldi, 2004: Assimilation of standard and targeted observations within the unstable subspace of the observation–analysis–forecast cycle system. J. Atmos. Sci., 61, 103113, doi:10.1175/1520-0469(2004)061<0103:AOSATO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Trevisan, A., M. D’Isidoro, and O. Talagrand, 2010: Four-dimensional variational assimilation in the unstable subspace and the optimal subspace dimension. Quart. J. Roy. Meteor. Soc., 136, 487496, doi:10.1002/qj.571.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Uboldi, F., and A. Trevisan, 2006: Detecting unstable structures and controlling error growth by assimilation of standard and adaptive observations in a primitive equation ocean model. Nonlinear Processes Geophys., 13, 6781, doi:10.5194/npg-13-67-2006.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, X., D. Parrish, D. Kleist, and J. Whitaker, 2013: GSI 3DVar-based ensemble-variational hybrid data assimilation for NCEP Global Forecast System: Single-resolution experiments. Mon. Wea. Rev., 141, 40984117, doi:10.1175/MWR-D-12-00141.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S., and T. M. Hamill, 2002: Ensemble data assimilation without perturbed observations. Mon. Wea. Rev., 130, 19131924, doi:10.1175/1520-0493(2002)130<1913:EDAWPO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 8473 5984 414
PDF Downloads 668 159 8