• Anderson, E., , and H. Järvinen, 1999: Variational quality control. Quart. J. Roy. Meteor. Soc.,125, 697722, doi:10.1002/qj.49712555416.

    • Search Google Scholar
    • Export Citation
  • Auligné, T., 2014: Multivariate minimum residual method for cloud retrieval. Part I: Theoretical aspects and simulated observation experiments. Mon. Wea. Rev.,142, 43834398, doi:10.1175/MWR-D-13-00172.1.

  • Auligné, T., , and A. P. McNally, 2007: Interaction between bias correction and quality control. Quart. J. Roy. Meteor. Soc.,133, 643653, doi:10.1002/qj.57.

    • Search Google Scholar
    • Export Citation
  • Auligné, T., , A. P. McNally, , and D. P. Dee, 2007: Adaptive bias correction for satellite data in a numerical weather prediction system. Quart. J. Roy. Meteor. Soc., 133, 631642, doi:10.1002/qj.56.

    • Search Google Scholar
    • Export Citation
  • Chevallier, F., , P. Lopez, , M. Janiskova, , and E. Moreau, 2004: The capability of 4D-Var systems to assimilate cloud-affected satellite infrared radiances. Quart. J. Roy. Meteor. Soc.,130, 917932, doi:10.1256/qj.03.113.

    • Search Google Scholar
    • Export Citation
  • Dahoui, M., , L. Lavanant, , F. Rabier, , and T. Auligné, 2005: Use of MODIS imager to help deal with AIRS cloudy radiances. Quart. J. Roy. Meteor. Soc.,131, 25592579, doi:10.1256/qj.04.90.

    • Search Google Scholar
    • Export Citation
  • Gauthier, P., , C. Chouinard, , and B. Brasnett, 2002: Quality control: Methodology and applications. Data Assimilation for the Earth System, R. Swinbank, V. Shutyaev, and W. A. Lahoz, Eds., Springer, 177187.

  • Gautier, C., , Y. Shiren, , and M. D. Hofstadter, 2003: AIRS/Vis near IR instrument. IEEE Trans. Geosci. Remote Sens., 41, 330342, doi:10.1109/TGRS.2002.808242.

    • Search Google Scholar
    • Export Citation
  • Han, Y., , P. van Delst, , Q. Liu, , F. Weng, , B. Yan, , R. Treadon, , and J. Derber, 2006: JCSDA Community Radiative Transfer Model (CRTM), version 1. NOAA Tech Rep. NESDIS 122, National Oceanic and Atmospheric Administration, National Environmental Satellite, Data, and Information Service, 33 pp. [Available online at ftp://ftp.emc.ncep.noaa.gov/jcsda/CRTM/CRTM_v1-NOAA_Tech_Report_NESDIS122.pdf.]

  • Hu, M., , S. Weygandt, , M. Xue, , and S. Benjamin, 2006: Development and testing of a new cloud analysis package using radar, satellite, and surface cloud observations within GSI for initializing Rapid Refresh. 18th Conf. on Numerical Weather Prediction, Park City, UT, Amer. Meteor. Soc., P2.5. [Available online at https://ams.confex.com/ams/pdfpapers/124496.pdf.]

  • Ingleby, N. B., , and A. C. Lorenc, 1993: Bayesian quality control using multivariate normal distributions. Quart. J. Roy. Meteor. Soc.,119, 11951225, doi:10.1002/qj.49711951316.

    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., 1986: Analysis methods for numerical weather prediction. Quart. J. Roy. Meteor. Soc.,112, 11771194, doi:10.1002/qj.49711247414.

    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., , and O. Hammon, 1988: Objective quality control of observations using Bayesian methods. Theory, and a practical implementation. Quart. J. Roy. Meteor. Soc.,114, 515543, doi:10.1002/qj.49711448012.

    • Search Google Scholar
    • Export Citation
  • McNally, A. P., 2002: A note on the occurrence of cloud in meteorologically sensitive areas and the implications for advanced infrared sounders. Quart. J. Roy. Meteor. Soc., 128, 25512556, doi:10.1256/qj.01.206.

    • Search Google Scholar
    • Export Citation
  • McNally, A. P., 2009: The direct assimilation of cloud-affected satellite infrared radiances in the ECMWF 4D-Var. Quart. J. Roy. Meteor. Soc., 135, 12141229, doi:10.1002/qj.426.

    • Search Google Scholar
    • Export Citation
  • McNally, A. P., , and P. D. Watts, 2003: A cloud detection algorithm for high spectral resolution infrared sounders. Quart. J. Roy. Meteor. Soc., 129, 34113423, doi:10.1256/qj.02.208.

    • Search Google Scholar
    • Export Citation
  • Morrison, H., , J. A. Curry, , and V. I. Khvorostyanov, 2005: A new double-moment microphysics parameterization for application in cloud and climate models. Part I: Description. J. Atmos. Sci., 62, 16651677, doi:10.1175/JAS3446.1.

    • Search Google Scholar
    • Export Citation
  • Pangaud, T., , N. Fourrie, , V. Guidard, , M. Dahoui, , and F. Rabier, 2009: Assimilation of AIRS radiances affected by mid- to low-level clouds. Mon. Wea. Rev., 137, 42764292, doi:10.1175/2009MWR3020.1.

    • Search Google Scholar
    • Export Citation
  • Pavelin, E. G., , S. J. English, , and J. R. Eyre, 2008: The assimilation of cloud-affected infrared satellite radiances for numerical weather prediction. Quart. J. Roy. Meteor. Soc., 134, 737749, doi:10.1002/qj.243.

    • Search Google Scholar
    • Export Citation
  • Skamarock, W. C., , J. B. Klemp, , J. Dudhia, , D. O. Gill, , D. M. Barker, , W. Wang, , and J. G. Powers, 2005: A description of the advanced research WRF version 2. NCAR Tech. Note NCAR/TN-468+STR, 88 pp. [Available online at http://www.mmm.ucar.edu/wrf/users/docs/arw_v2.pdf.]

  • Wu, M.-L. C., 1987: A method for remote sensing the emissivity, fractional cloud cover and cloud top temperature of high-level, thin clouds. J. Climate Appl. Meteor., 26, 225233, doi:10.1175/1520-0450(1987)026<0225:AMFRST>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • View in gallery

    Difference in cloud-top pressure estimations between MMR and MW03. The histogram is calculated on a population of 101 430 AIRS observation points.

  • View in gallery

    Cloud-top pressure retrieved with (top) MW03 and (bottom) MMR for 101 430 AIRS observation points.

  • View in gallery

    Cloud-top pressure retrieved with (top) MW03 and (bottom) MMR for AIRS observations over the North Atlantic.

  • View in gallery

    Statistics of observation-minus-background departures before bias correction for observations declared clear by MW03 (black) and MMR (gray) over a population of 101 430 AIRS observation points: (top) bias, (middle) standard deviation, and (bottom) data count.

  • View in gallery

    Histogram of observation-minus-background departures before bias correction for (top) MW03 and (bottom) MMR for AIRS channel 1545 (sensitive to midtroposphere moisture).

  • View in gallery

    As in Fig. 5, but for AIRS window channel 787 (around 11 μm).

  • View in gallery

    Total cloud cover in the AIRS field as observed by the VIS/NIR imager. Only populations reported as clear by either the (top) MW03 or (bottom) MMR scheme are represented. Points with a fraction of cloud of 1.0 are removed because of an ambiguity in the observation dataset with nighttime unusable visible observations.

  • View in gallery

    SEVIRI channel 4 image (around 3.9 μm) at 0300 UTC 10 May 2006 over the North Atlantic and AIRS observation points retained by the MMR scheme for (a) channel 145 (mostly sensitive around 100 hPa), (b) channel 221 (mostly sensitive around 600 hPa), and (c) channel 787 (mostly sensitive near the surface).

  • View in gallery

    SEVIRI channel 4 image (around 3.9 μm) at 0000 UTC 10 May 2006 over South Africa and AIRS observation points retained by the MMR scheme for (a) channel 145 (mostly sensitive around 100 hPa), (b) channel 221 (mostly sensitive around 600 hPa), and (c) channel 787 (mostly sensitive near the surface).

  • View in gallery

    (left) Mean and (right) standard deviation of the a priori (blue circles) and a posteriori (green stars) probabilities of cloud contamination AIRS channels in the 15-μm band. The channels are ranked vertically according to the pressure of the peak of their weighting function determined for a standard atmosphere.

  • View in gallery

    (a) Percentage of total of observations rejected by the variational algorithm (gray), and the relative impact (in %) on the standard deviation of observation-minus-background departures (black circles). We consider the AIRS observations assimilated over 12 h in the Southern Hemisphere (about 8000 observation points) using the MMR cloud detection scheme. (b) AIRS channel wavelengths.

  • View in gallery

    (a) Bias (dashed gray curve) and standard deviation (solid gray curve) observation-minus-background departures for all observations. Also shown are the bias (blue) and standard deviation (green) of departures between the MMR-simulated observations and the background. (b) Standard deviation of the departures between observations and the MMR-simulated observations for all observations (black) and only clear observations (gray).

  • View in gallery

    Maps of brightness temperatures for AIRS window channel 787 (11 μm) from (a) observations and (b) simulation with the MMR observation operator using the retrieved profiles of the cloud fraction. Scatterplots of observations vs (c) clear-sky calculation from the model background, and (d) calculation using the MMR observation operator with retrieved cloud fractions.

  • View in gallery

    Maps of (a) retrieved cloud fraction (in %) integrated over the vertical column and (b) GOES Imager channel 4 irradiance (in W m−2), both valid at 2000 UTC 6 Jun 2009.

  • View in gallery

    Vertical cross section of cloud fraction retrieved with MMR (isocontours every 1%) and model background relative humidity (green shading every 10% above 40%). The cross section is defined along the 40°N line across the domain in Fig. 14.

  • View in gallery

    Scatterplots of observations vs (a) all-sky calculation from the model background and (b) the MMR observation operator using the retrieved profiles of the cloud fraction.

  • View in gallery

    Taylor diagrams of the model background (red) and analysis (blue) fit to observations, with each point corresponding to a specific AIRS channel selected for the assimilation. The reference “REF” represents the (bias corrected) observations. Normalized standard deviation and correlation are shown along the abscissa and angle, respectively. The background corresponds to (a) the model equivalent calculated with the all-sky CRTM and (b) the model equivalent calculated with the MMR observation operator.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 17 17 2
PDF Downloads 13 13 2

Multivariate Minimum Residual Method for Cloud Retrieval. Part II: Real Observations Experiments

View More View Less
  • 1 European Centre for Medium-Range Weather Forecasts, Reading, United Kingdom, and National Center for Atmospheric Research, Boulder, Colorado
© Get Permissions
Full access

Abstract

In of this two-part paper, the multivariate minimum residual (MMR) scheme was introduced to retrieve profiles of cloud fraction from satellite infrared radiances and identify clear observations. In this paper it is now validated with real observations from the Atmospheric Infrared Sounder (AIRS) instrument. This new method is compared with the cloud detection scheme presented earlier by McNally and Watts and operational at the European Centre for Medium-Range Weather Forecasts (ECMWF). Cloud-top pressures derived from both algorithms are comparable, with some differences at the edges of the synoptic cloud systems. The population of channels considered as clear is less contaminated with residual cloud for the MMR scheme. Further procedures, based on the formulation of the variational quality control, can be applied during the variational analysis to reduce the weight of observations that have a high chance of being contaminated by cloud. Finally, the MMR scheme can be used as a preprocessing step to improve the assimilation of cloud-affected infrared radiances.

Corresponding author address: Thomas Auligné, Mesoscale and Microscale Meteorology Division, National Center for Atmospheric Research, P.O. Box 3000, Boulder, CO 80307-3000. E-mail: auligne@ucar.edu

Abstract

In of this two-part paper, the multivariate minimum residual (MMR) scheme was introduced to retrieve profiles of cloud fraction from satellite infrared radiances and identify clear observations. In this paper it is now validated with real observations from the Atmospheric Infrared Sounder (AIRS) instrument. This new method is compared with the cloud detection scheme presented earlier by McNally and Watts and operational at the European Centre for Medium-Range Weather Forecasts (ECMWF). Cloud-top pressures derived from both algorithms are comparable, with some differences at the edges of the synoptic cloud systems. The population of channels considered as clear is less contaminated with residual cloud for the MMR scheme. Further procedures, based on the formulation of the variational quality control, can be applied during the variational analysis to reduce the weight of observations that have a high chance of being contaminated by cloud. Finally, the MMR scheme can be used as a preprocessing step to improve the assimilation of cloud-affected infrared radiances.

Corresponding author address: Thomas Auligné, Mesoscale and Microscale Meteorology Division, National Center for Atmospheric Research, P.O. Box 3000, Boulder, CO 80307-3000. E-mail: auligne@ucar.edu

1. Introduction

This paper is the second part of a two-part paper. In Auligné (2014, hereafter Part I), we introduced the multivariate minimum residual (MMR) method to retrieve profiles of cloud fraction from satellite infrared radiances at every field of view. The scheme consists of comparing the observations with their model equivalent computed for clear sky with a radiative transfer model. We use an additional array of radiances corresponding to the simulated radiances with an opaque black cloud located at each model vertical level. All this information is provided to a variational minimization, which builds the optimal profile of cloud fractions to fit the observations.

In Part I, the ability of the MMR scheme to identify clear channels is studied with simulated data for the Atmospheric Infrared Sounder (AIRS) instrument. We use for comparison the cloud detection scheme from McNally and Watts (2003, hereafter MW03), which is used operationally at the European Centre for Medium-Range Weather Forecasts (ECMWF). The two schemes behave comparatively in terms of cloud detection. In addition, the MMR scheme shows some skill in retrieving multilayer cloud information.

In this paper, we will expand the study to real observations from the AIRS instrument and will validate the results with independent satellite data. The second objective is to outline the potential use of the MMR scheme in the assimilation of cloud-affected infrared radiances. The paper is organized as follows. Section 2 compares the cloud-top pressure derived from both the MMR and MW03 schemes. The cloud residual in the channels selected by the cloud detection scheme is analyzed in section 3. A preliminary validation of the cloud detection is provided in section 4 using Spinning Enhanced Visible and Infrared Imager (SEVIRI) observations. Section 5 proposes an extension of the scheme to reduce the cloud residual, based on the formulation of the variational quality control (VarQC; Ingleby and Lorenc 1993; Andersson and Jarvinen 1999). Finally, section 6 offers perspectives for the use of MMR in the assimilation of cloud-affected radiances and section 7 concludes this work.

2. Retrieved cloud-top pressure

We now want to study the behavior of the cloud detection scheme in an environment similar to a real analysis for NWP forecasting. We consider the AIRS observations in a time window between 2100 UTC 9 May and 0900 UTC 10 May 2006 and restricted to data over the oceans. This period involves a variety of cloud structures that have been observed by Meteosat SEVIRI, including synoptic perturbations over the eastern North Atlantic and the western Indian Oceans, mid- to-low-level clouds off the coast of South Africa, and convective clouds off the coast of Morocco. The study population includes 101 430 observations and provides almost global coverage. The two schemes, MW03 and MMR, are provided a set of observation-minus-first-guess departures calculated after bias correction with the first guess originating from a 12-h forecast using the Integrated Forecast System (IFS) model (with a T159 spectral truncation and 91 vertical levels). The same bias correction is applied for both experiments and it is calculated via the variational bias correction (VarBC; Auligné et al. 2007).

The cloud-top pressure Pcld is a diagnostic that is estimated as follows:

  • For the MW03 scheme, we determine the lowest clear channel. We then define Pcld as the maximum pressure that a black cloud should have for its radiative impact to affect the channel by at least 1%.
  • For the MMR scheme, clear and cloudy simulated radiances are compared for every channel. Channels for which the difference exceeds a certain threshold (e.g., 1% of the clear radiance) are declared cloudy. Then, Pcld is determined from the lowest clear channel, as for MW03.

The difference in the estimates between the two cloud detection schemes is shown in Fig. 1. For a majority of the observations, both schemes position the top of the cloud at the same altitude. However, they differ by ±400 hPa for a nonnegligible number of points, but there is no significant bias between the two methods.

Fig. 1.
Fig. 1.

Difference in cloud-top pressure estimations between MMR and MW03. The histogram is calculated on a population of 101 430 AIRS observation points.

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

Figure 2 shows that the geographical distribution of clouds is estimated similarly by both schemes at synoptic scales. In particular, the positions of the intertropical convergence zone (ITCZ) and of large cloud systems are comparable. The cloud altitudes are however significantly different in the polar regions. In general, the distinctions appear along the edges of the cloud systems: MMR presents a less abrupt transition between the center of systems with a high cloud top and the areas with clear or low-cloud conditions. The discrete nature of the cloud-top pressure visible in the histograms is due to the estimation method. The histograms show that MMR produces a more homogeneous vertical distribution of clouds than does MW03, which particularly estimates few low clouds between 800 and 900 hPa. This property of MW03 can be explained by the purpose for which the scheme was designed: to identify cloud-contaminated channels. Because of the potentially large negative impact of residual cloud contamination on the analysis, the scheme is intentionally conservative in the attribution of the cloud height for difficult cases, such as low clouds.

Fig. 2.
Fig. 2.

Cloud-top pressure retrieved with (top) MW03 and (bottom) MMR for 101 430 AIRS observation points.

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

Differences across subsynoptic scales are visible, as shown in Fig. 3. Again, the major differences appear at the edges of convective and synoptic systems rather than at the center. Some small-scale structures are represented differently, for example, for the occlusion zone for the perturbation off the coasts of Europe.

Fig. 3.
Fig. 3.

Cloud-top pressure retrieved with (top) MW03 and (bottom) MMR for AIRS observations over the North Atlantic.

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

3. Cloud residual in clear population

We will now consider the characteristics of the clear population, that is, the population that the cloud detection scheme considered acceptable for the analysis. We consider here the operational implementation of MW03 at ECMWF prior to the implementation of the so-called cross-band option, which would rely solely on the 15-μm band to perform cloud detection for the whole spectrum. Figure 4 represents the bias (top) and standard deviation (middle) for departures before bias correction, for observations in the Southern Hemisphere declared clear by the MW03 and MMR schemes, respectively. The number of clear observations is shown in the bottom panel of Fig. 4. It is important to consider statistics before bias correction, because the bias correction is adjusted adaptively to correct departures selected by the MW03 scheme. For the CO2 longwave (i.e., the first 300 channels) and shortwave (channels above 1850) spectral bands, the two schemes are relatively comparable. We note however that MMR unlike MW03 does not exclude any observation in the upper stratosphere. For the spectral band sensitive to moisture (channels between 1400 and 1800), MMR selects twice as many observations as does MW03 with a nearly identical bias and a slightly higher standard deviation. For window channels (between about 400 and 1400) MMR selected slightly more data than MW03, resulting in an increased standard deviation but with the bias reduced. Populations of clear observations for channel 1545, sensitive to moisture in the midtroposphere, are shown in Fig. 5. The two schemes produce distributions with warm tails, because the impact of a cloud usually results in negative departures and we therefore reject more observations on the negative side of the distribution than on the positive side. We note, however, that the population selected by MMR is more symmetrical. Auligné and McNally (2007) showed that asymmetric populations can trigger a detrimental feedback loop between cloud detection and correction bias. We can therefore expect that this interaction will be less pronounced with the MMR scheme. Figure 6 describes the populations selected by the two schemes for channel 787 sensitive to the surface. For the negative part of the departure distribution, the two histograms are quite alike. However, MW03 truncates most of the positive differences while MMR keeps a large number of them. Indeed, the MW03 algorithm includes a quality control (QC) process comparable to a boxcar window applied to the bias-corrected departures. For this channel the bias correction is negative and the QC threshold is tight, which results in discarding some positive (potentially good) departures.

Fig. 4.
Fig. 4.

Statistics of observation-minus-background departures before bias correction for observations declared clear by MW03 (black) and MMR (gray) over a population of 101 430 AIRS observation points: (top) bias, (middle) standard deviation, and (bottom) data count.

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

Fig. 5.
Fig. 5.

Histogram of observation-minus-background departures before bias correction for (top) MW03 and (bottom) MMR for AIRS channel 1545 (sensitive to midtroposphere moisture).

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

Fig. 6.
Fig. 6.

As in Fig. 5, but for AIRS window channel 787 (around 11 μm).

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

To validate cloud detection for window channels, an independent data source is introduced: the AIRS visible/near infrared (VIS/NIR) imager, which is collocated with the AIRS sounder (Gautier et al. 2003). This instrument provides information on the fraction of each AIRS pixel covered by a cloud. As the imager operates in the visible and near-infrared band, this information is only available during daytime. Figure 7 represents the cloud fraction observed by the VIS/NIR imager for observation points where the cloud detection scheme considers channel 787 clear. The data are limited to pixels with a cloud fraction strictly less than 1, due to an ambiguity in the VIS/NIR dataset between nighttime and fully covered observations, but this limitation has minor impact on the results since the two schemes are very comparable for fully covered fields of view. The imager indicates less cloud contamination in the clear population of window channels for MMR compared to MW03. It may be noted that both schemes are contaminated in the same cloudy regions.

Fig. 7.
Fig. 7.

Total cloud cover in the AIRS field as observed by the VIS/NIR imager. Only populations reported as clear by either the (top) MW03 or (bottom) MMR scheme are represented. Points with a fraction of cloud of 1.0 are removed because of an ambiguity in the observation dataset with nighttime unusable visible observations.

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

4. Comparison with SEVIRI observations

MW03 like MMR allows information to be retained from channels whose weighting functions peak above the cloud. This is a significant advantage when considering that nearly 90% of the fields of view are cloud affected and especially within the sensitive areas where the forecast error growth is largest (McNally 2002). As a visual validation, Figs. 8 and 9 present a superposition of channels selected by the MMR scheme with imaging from SEVIRI for the spectral band at 3.9 μm (channel 4). Three channels are shown: channel 145 (at 14.5 μm, peaking around 100 hPa), channel 221 (at 13.5 μm, peaking around 600 hPa), and channel 787 (at 11.0 μm, peaking near the surface). Two geographic areas are chosen around the North Atlantic and South Africa to cover a wide range of cloudy situations (clear zones, low clouds, synoptic disturbance, shallow convection, deep convection). High-peaking water vapor channel 145 is almost always preserved, except in the vicinity of tropical convective clouds near Madagascar. As expected, the lower in the atmosphere a channel peaks, the fewer the number of pixels that are retained by the scheme. Considering spatiotemporal discrepancies between the two instruments, we see a very good fit between clear areas for SEVIRI and clear pixels in the MMR scheme. In the Atlantic area, MMR correctly rejects the observations close to the low pressure system and retains the pixels at the rear of the perturbation. The observations off the Canary Islands seem to match actual clear areas in the middle of convective clouds. One can note an underrepresentation of window channel 787 over the continents even for areas obviously clear (e.g., over South Africa). This large number of rejections by the MMR scheme for low-peaking channels is most likely due to the sensitivity of the algorithm to the surface temperature and emissivity, which can be poorly represented by the model over the continents.

Fig. 8.
Fig. 8.

SEVIRI channel 4 image (around 3.9 μm) at 0300 UTC 10 May 2006 over the North Atlantic and AIRS observation points retained by the MMR scheme for (a) channel 145 (mostly sensitive around 100 hPa), (b) channel 221 (mostly sensitive around 600 hPa), and (c) channel 787 (mostly sensitive near the surface).

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

Fig. 9.
Fig. 9.

SEVIRI channel 4 image (around 3.9 μm) at 0000 UTC 10 May 2006 over South Africa and AIRS observation points retained by the MMR scheme for (a) channel 145 (mostly sensitive around 100 hPa), (b) channel 221 (mostly sensitive around 600 hPa), and (c) channel 787 (mostly sensitive near the surface).

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

5. Variational cloud detection

a. Variational quality control

The VarQC scheme is a quality control methodology implemented directly within the variational analysis. It takes into account the non-Gaussian distribution of gross errors in the observations. While observation errors associated with the accuracy of measurements or representativeness are random in nature and can be described by normal distributions, gross errors are independent of the probability of observed values. As described in Lorenc and Hammon (1988), the gross errors can be assumed to be equiprobable on a range of values. Instead of making the assumption of Gaussian statistics, the probability distribution of observation error is defined as the sum of two components: a Gaussian distribution representing the correct observations and a flat distribution corresponding to the gross errors. It can be written in the following manner:
e1
where is the a priori probability of error for the observation with index I, and N and F are, respectively, the Gaussian and flat distributions given by
e2
e3
where is the observation operator for the radiance and the flat distribution is defined on an interval expressed as a multiple of the standard deviation of the observation error σo (using the factor determined empirically).
Recall the definition of the cost function corresponding to the observations in the variational analysis, which allows estimating the most likely state of the atmosphere p such as
e4
The classical contribution for an observation is written in the following manner:
e5
This cost function is modified to take into account the non-Gaussian distribution of gross errors in the observations, resulting in a decrease in the weight given to observations departing substantially from the current state of the analysis. Substituting Eqs. (1)(3) into Eq. (4), we obtain after rearrangement an expression of the cost function modified by the variational quality control and of its gradient as a function of the classic cost function :
e6
e7
where we have
e8
e9
This method can also be explained using the formalism of Bayes (Lorenc 1986). First, an a priori estimate of the probability of gross error is attributed to each observation from empirical data. Then, during the variational assimilation, the a posteriori estimate is calculated based on the state of the analysis (Ingleby and Lorenc 1993). VarQC uses the calculated weight by the quality control 1 − P(G) to balance the gradient of the cost function with respect to observations. Thus, the data that are wrong with near certainty P(G) receive a quasi-zero weighting in the analysis. Observations are never rejected irrevocably and they can see their influence increase again during the analysis if the surrounding data confirm them. Thus, observations for which the weight was strongly decreased are often those that conflict with the surrounding data and we can therefore consider VarQC as a kind of “buddy check.” The formalism of VarQC and its implementation are described in Anderson and Järvinen (1999) and Gauthier et al. (2002). In practice, most analyses use a conjugate gradient minimizer and they do not tolerate modification to the cost function. The a posteriori estimate of the probability of gross error is only recalculated for each outer loop of the incremental variational assimilation.

b. Application to cloud detection

We saw in the first part of this paper that the theoretical accuracy of the MMR method is better for high clouds than for low clouds, because of a greater “contrast” between the clear and cloudy radiances for the entire spectrum measured by the AIRS instrument. However, in practice significantly reduced performance can be expected for the detection of high clouds because of the following reasons:

  • The thicker the cloud, the more difficult it gets to model it by a stack of thin clouds without overlap between them. It is therefore possible that the simple model adopted in the MMR scheme to represent the clouds may not be realistic for clouds with high vertical extension.
  • The hypothesis of an effective emissivity (i.e., the cloud emissivity multiplied by the cloud fraction) independent from the frequency over a wide spectral band is generally acceptable in the infrared but becomes inaccurate for semitransparent ice clouds (Wu 1987).
  • Semitransparent clouds (mainly cirrus) are particularly difficult to identify due to their reduced radiative impact. The MMR method uses a threshold on the differences between clear and cloudy radiances. It is possible that some semitransparent clouds do not reach this threshold, and therefore they are not detected.

Generally, the radiative effect of errors in the observations and the model are neglected compared to the radiative signature of a cloud, which can lead to inaccuracies in the cloud altitude estimation. We propose to reduce the impact on the analysis of detection errors from the MMR scheme. For this, we perform a variational adjustment of the probability of cloud contamination for each frequency. In Part I of this paper, we saw that we can interpret the cloud fraction Xk calculated by MMR for a model level k as an estimate of the probability that the actual cloud is in the class of thin black clouds at level k. At each observation point, we define the probability that the channel υ is cloud affected by the following equation:
e10
where Nυ represents the ensemble of all model levels for which the radiative impact of thin black cloud on channel υ is greater than a threshold determined arbitrarily (fixed to 1% of the clear-sky radiance). We consider the sum Pυ as the probability of cloud contamination calculated by MMR for channel υ.

The clouds that were not detected by the MMR scheme can degrade the quality of the analysis. The impact of a residual cloud contamination on first-guess departures resembles that of the gross error, but only for negative departures. Therefore, the population of departures may be decomposed into a clear population with a Gaussian distribution and a cloudy residual with a flat distribution on the negative side. We choose to adopt the formalism of VarQC to correct the residual cloud contamination. For each observation point, the a priori probability of cloud contamination for channel υ is equal to Pυ calculated by the MMR scheme and is then rescaled by a constant (fixed empirically to 0.3).

During the 4DVAR analysis, the algorithm adjusts the probability according to Bayes’s formalism to give more influence to radiances supported by surrounding data and to remove radiances that the analysis cannot reproduce accurately. Unlike the MW03 and MMR schemes, the various AIRS observation points are no longer treated independently, and we can consider the variational algorithm as a kind of buddy check, comparing observations to surrounding ones. As for VarQC, correlations between channels are neglected. This assumption is questionable because it is likely that a misdetected cloud may produce similar effects on adjacent channels; however, the specification of error correlations would be complex. The a posteriori probability of cloud contamination is produced by the VarQC algorithm. Figure 10 represents the statistics of a priori and a posteriori probabilities of cloud contamination for AIRS channels in the 15-μm temperature band. The average a priori probability increases with altitude in the troposphere, demonstrating the practical difficulty of accurately detecting high clouds with MMR. The corresponding standard deviation is relatively constant in the troposphere. Above 100 hPa, a residual cloud contamination in the observations is unlikely. For most channels, the variational algorithm decreases the mean and standard deviation of the probability of cloud contamination, since the analysis was able to effectively assimilate observations. However, the a posteriori probability stays important for channels peaking between 100 and 300 hPa, and it has a high variability. Observations “rejected” by the variational algorithm essentially correspond to points in the polar regions or along cloud systems detected by MMR (not shown). We define as “rejected” by the algorithm, the observations for which the likelihood of residual contamination is increased. It can be seen in Fig. 11 for a small proportion of rejected observations (usually less than 1%) that the variational algorithm reduces the standard deviations in the first-guess departure by about 5%. In practice, the observations are not really rejected from the assimilation but their confidence is decreased, which results in a smaller influence in the analysis.

Fig. 10.
Fig. 10.

(left) Mean and (right) standard deviation of the a priori (blue circles) and a posteriori (green stars) probabilities of cloud contamination AIRS channels in the 15-μm band. The channels are ranked vertically according to the pressure of the peak of their weighting function determined for a standard atmosphere.

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

Fig. 11.
Fig. 11.

(a) Percentage of total of observations rejected by the variational algorithm (gray), and the relative impact (in %) on the standard deviation of observation-minus-background departures (black circles). We consider the AIRS observations assimilated over 12 h in the Southern Hemisphere (about 8000 observation points) using the MMR cloud detection scheme. (b) AIRS channel wavelengths.

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

6. Perspectives for the assimilation all-sky radiances

a. Discrimination of clear radiances

The first application of the MMR scheme is to detect channels that are affected by clouds in order to identify a population of cloud-free channels, which can be assimilated directly. In preliminary assimilation experiments (not shown) performed at low resolution T159 over a period of 3 weeks with clear radiances, the MMR scheme showed analyses comparable to MW03 over continental areas controlled by the radiosondes. For oceanic and polar regions, differences of the order of 0.5 K appear locally between analyzes without any specific structure. The root-mean-square error of analysis increments are comparable between the two experiments (not shown). The impact of the new cloud detection on the quality of the forecast is broadly neutral. Further study is required to confirm these results.

b. Simplified observation operator

Even though it relies on a basic modeling of clouds, the MMR scheme still manages to reproduce cloud radiative effects for AIRS observations, as shown in the top panel of Fig. 12. We note that the observations and their simulated counterparts calculated by MMR show similar first-guess departures for all tropospheric channels. If we substitute for cloud-affected observations the clear-sky radiative transfer model (RTM) calculation by the cloudy-sky calculation from MMR, we obtain departures represented (in black) in the bottom panel of Fig. 12. These departures are slightly larger than those calculated for clear observations (in gray), especially for temperature channels sensitive to lower levels, but the variability of clear observations is probably underestimated due to an overly stringent quality control procedure. Departures using the MMR cloudy observation operator show statistics that are comparable to the departures when using cloud-cleared radiances provided by the National Oceanic and Atmospheric Administration/National Environmental Satellite, Data, and Information Service (NOAA/NESDIS; not shown).

Fig. 12.
Fig. 12.

(a) Bias (dashed gray curve) and standard deviation (solid gray curve) observation-minus-background departures for all observations. Also shown are the bias (blue) and standard deviation (green) of departures between the MMR-simulated observations and the background. (b) Standard deviation of the departures between observations and the MMR-simulated observations for all observations (black) and only clear observations (gray).

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

A different experiment is run at the convection-allowing horizontal resolution of 4 km over the central region of the United States. The goal is to demonstrate the robustness of the cloud retrieval algorithm for a different domain, resolution, surface type (land versus ocean) NWP model, and radiative transfer model. The algorithm was simply ported to the new system without the need for any specific tuning. The background is a short-term forecast using the Weather Research and Forecasting Model (WRF; Skamarock et al. 2005). We use the Community Radiative Transfer Model (CRTM; Han et al. 2006) to compute brightness temperatures for the AIRS instrument. Figure 13 shows the observations and model equivalent using the simplified MMR observation operator for AIRS window channel 787, which has a maximum sensitivity to clouds. The ability of the simplified MMR observation operator to model the cloud radiative effect is remarkable and very similar to the performance with synthetic data shown in the Part I. One can notice a small residual for warm observations, which actually corresponds to clear observations. For these data, the surface emissivity and skin temperature are poorly modeled over land, resulting in discrepancies between the modeled and observed data. The fact that the MMR scheme does not create artificial clouds to compensate for surface deficiencies is a good property.

Fig. 13.
Fig. 13.

Maps of brightness temperatures for AIRS window channel 787 (11 μm) from (a) observations and (b) simulation with the MMR observation operator using the retrieved profiles of the cloud fraction. Scatterplots of observations vs (c) clear-sky calculation from the model background, and (d) calculation using the MMR observation operator with retrieved cloud fractions.

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

Besides its simplicity and low computational cost, the major advantage of this approach for the cloudy radiances is in the linearity of the observation operator. Indeed, the MMR method calculates the radiative transfer only for clear sky (i.e., without cloud emission, diffusion, or scattering) and simply relocates the surface at different altitudes. The resulting radiances are combined linearly.

Although this is beyond the scope of this paper, it would be particularly interesting to study the impact of assimilating cloud-affected radiances using this simplified observation operator based on MMR. This work would complement the experiments of Pavelin et al. (2008), Pangaud et al. (2009), and McNally (2009), which are focused on single-layer clouds, and could expand their methodology to multilayer clouds. Therefore, there is a chance to relax the current selection of cloud scenes, which is limited to the small fraction of opaque homogeneous clouds. We could examine the types of clouds (stratiform or convective) and altitudes for which clouds are adequately represented. One might also consider the optical thickness of clouds for which we obtain useful information for the analysis.

c. Improved first guess for the assimilation of all-sky radiances

A fundamentally different approach for the assimilation of cloud-affected radiances consists of updating the three-dimensional cloud variables (e.g., cloud liquid water, ice water mixing ratios) using the cloudy radiative transfer. Within this context, Chevallier et al. (2004) and Dahoui et al. (2005) found strong nonlinearities for AIRS channels sensitive to clouds, which currently represent an obstacle to their assimilation. These nonlinearities can originate in the radiative transfer, the horizontal interpolations, and the model moist physics. The quality of the model first-guess conditions the amplitude of the analysis updates. The better the first guess, the smaller the required region of linearity and the more likely the analysis will be able to handle nonlinearities.

Hu et al. (2006) use an empirical method to relate a retrieved cloud envelope to mixing ratios of cloud liquid water and ice. We follow the same methodology to transform the cloud fraction retrieved by MMR into microphysical variables. In this paper, the cloud envelope is defined as the three-dimensional regions where the MMR cloud fraction is greater than a threshold of 1%. As shown in Fig. 14, the location of the clouds retrieved by MMR correlates quite well with the image from the Geostationary Operational Environmental Satellite (GOES) Imager. The vertical cross section in Fig. 15 shows a cloud envelope, which is physically realistic and corresponds to areas of saturation in the model background. We recognize that this validation is rather subjective and further work will be required to properly verify the retrieval of vertical clouds. Our goal here is merely to show that neighboring retrievals are consistent vertically with one another and that retrieved clouds occur in areas where the model would have predicted significant saturation.

Fig. 14.
Fig. 14.

Maps of (a) retrieved cloud fraction (in %) integrated over the vertical column and (b) GOES Imager channel 4 irradiance (in W m−2), both valid at 2000 UTC 6 Jun 2009.

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

Fig. 15.
Fig. 15.

Vertical cross section of cloud fraction retrieved with MMR (isocontours every 1%) and model background relative humidity (green shading every 10% above 40%). The cross section is defined along the 40°N line across the domain in Fig. 14.

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

We use the empirical relationship defined for stratiform clouds in Hu et al. (2006) to estimate the cloud condensate as a predefined fraction of the autoconversion mixing ratio. However, the radiative transfer requires cloud liquid water and ice water content, so we need to define a partitioning scheme to separate water phases within the cloud condensate. We use the partitioning scheme from the Global Forecast System (GFS) model:
e11
where qc is the cloud condensate, ql the cloud liquid water, and qi the cloud ice mixing ratio, and is a coefficient defined as
e12
with T00 and Ti two scalar thresholds that can be tuned for particular regions.

The retrieved cloud liquid water and ice mixing ratios are then input into the CRTM to compute cloudy brightness temperatures. The fit to observations is shown in Fig. 16b for AIRS channel 787 and in Fig. 17b (in red) for all channels used in the analysis. Figures 16a and 17a correspond to an experiment without MMR, where the background cloud liquid water and ice simply originate from the WRF short-term forecast [with the microphysical scheme from Morrison et al. (2005)]. Although the proposed procedure combines the MMR retrieval of cloud fraction with a relatively crude empirical transformation into liquid and ice mixing ratios, it still compares favorably (Fig. 17b) to our best background estimate of clouds (i.e., a short-term numerical forecast at cloud-allowing resolution). We conclude that MMR can be used to improve the first guesses of cloud liquid and ice mixing ratios for a data assimilation. The blue dots in Fig. 17 represent the analyses after assimilating cloud-affected radiances in a variational system and updating cloud liquid and ice mixing ratios. We notice that the improved first guess from MMR results in an analysis, which fits the observations more closely. However, this does not necessarily mean that subsequent forecasts will be improved. Future work will be dedicated to addressing this aspect.

Fig. 16.
Fig. 16.

Scatterplots of observations vs (a) all-sky calculation from the model background and (b) the MMR observation operator using the retrieved profiles of the cloud fraction.

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

Fig. 17.
Fig. 17.

Taylor diagrams of the model background (red) and analysis (blue) fit to observations, with each point corresponding to a specific AIRS channel selected for the assimilation. The reference “REF” represents the (bias corrected) observations. Normalized standard deviation and correlation are shown along the abscissa and angle, respectively. The background corresponds to (a) the model equivalent calculated with the all-sky CRTM and (b) the model equivalent calculated with the MMR observation operator.

Citation: Monthly Weather Review 142, 12; 10.1175/MWR-D-13-00173.1

7. Conclusions

In the first part of the paper, a new cloud retrieval scheme has been introduced and studied with synthetic observations. In this second part, we validated the scheme with real observations from the AIRS instrument. For the identification of cloud-free channels within each field of view, the MMR scheme provides comparable results to the MW03 scheme used operationally at the European Centre for Medium-Range Weather Forecasts (ECMWF) and they both identify cloud structures correctly at the synoptic scale, with differences particularly along the edges of cloud systems. Statistics of the clear populations provided to the analysis diverge for the water vapor spectral band where MMR retains more data. This is also the case for window channels where MMR is less restricted to the observations being close to the model first guess. A first validation with the VIS/NIR imager and SEVIRI channel 4 indicates an acceptable level of performance for the MMR cloud detection over the oceans. The impact of the surface temperature and emissivity over the continents deserves to be investigated in future studies.

High clouds are less well detected by the algorithm, but the probability of residual contamination can be estimated for each channel. The adjustment in the 4DVAR algorithm of that probability, following the VarQC formalism, identifies the observations that disagree most with all of the information contained in the analysis. These observations are considered contaminated by clouds and they see their influence reduced in the analysis.

The second interest of the MMR scheme is in its potential use to assimilate cloud-affected radiances. The profiles of cloud fraction that are retrieved by MMR can be used to define a simplified observation operator, which only uses clear-sky radiative transfer calculation and is therefore much more linear than full cloudy radiative transfer. The use of such an operator and the impact of cloud-affected radiances on the analysis will be studied in future work.

Furthermore, the retrieved profiles of cloud fractions can be converted into microphysical variables, using a simple empirical scheme. Although this procedure is relatively crude, it still provides a first guess that is significantly better than a numerical forecast of cloud at high resolution. Therefore, MMR can be considered as an option to preprocess the model first guess and reach a better level of accuracy, which will reduce the range of the region of linearity required in the analysis. As a result, this procedure may partially mitigate the issue of strong nonlinearities for the assimilation of cloud-affected satellite infrared sensors.

Acknowledgments

The author would like to acknowledge Antony McNally for his participation in this work. Jean-Noel Thepaut is also thanked for his help and support. Hongli Wang and Ming Hu are acknowledged for their support in computing the Rapid Refresh cloud analysis, and Andy Jones for providing the GEOS Imager data. The help provided by Francois Vandenberghe in the elaboration of the manuscript is greatly appreciated. This work was partially funded by the Air Force Weather Agency.

REFERENCES

  • Anderson, E., , and H. Järvinen, 1999: Variational quality control. Quart. J. Roy. Meteor. Soc.,125, 697722, doi:10.1002/qj.49712555416.

    • Search Google Scholar
    • Export Citation
  • Auligné, T., 2014: Multivariate minimum residual method for cloud retrieval. Part I: Theoretical aspects and simulated observation experiments. Mon. Wea. Rev.,142, 43834398, doi:10.1175/MWR-D-13-00172.1.

  • Auligné, T., , and A. P. McNally, 2007: Interaction between bias correction and quality control. Quart. J. Roy. Meteor. Soc.,133, 643653, doi:10.1002/qj.57.

    • Search Google Scholar
    • Export Citation
  • Auligné, T., , A. P. McNally, , and D. P. Dee, 2007: Adaptive bias correction for satellite data in a numerical weather prediction system. Quart. J. Roy. Meteor. Soc., 133, 631642, doi:10.1002/qj.56.

    • Search Google Scholar
    • Export Citation
  • Chevallier, F., , P. Lopez, , M. Janiskova, , and E. Moreau, 2004: The capability of 4D-Var systems to assimilate cloud-affected satellite infrared radiances. Quart. J. Roy. Meteor. Soc.,130, 917932, doi:10.1256/qj.03.113.

    • Search Google Scholar
    • Export Citation
  • Dahoui, M., , L. Lavanant, , F. Rabier, , and T. Auligné, 2005: Use of MODIS imager to help deal with AIRS cloudy radiances. Quart. J. Roy. Meteor. Soc.,131, 25592579, doi:10.1256/qj.04.90.

    • Search Google Scholar
    • Export Citation
  • Gauthier, P., , C. Chouinard, , and B. Brasnett, 2002: Quality control: Methodology and applications. Data Assimilation for the Earth System, R. Swinbank, V. Shutyaev, and W. A. Lahoz, Eds., Springer, 177187.

  • Gautier, C., , Y. Shiren, , and M. D. Hofstadter, 2003: AIRS/Vis near IR instrument. IEEE Trans. Geosci. Remote Sens., 41, 330342, doi:10.1109/TGRS.2002.808242.

    • Search Google Scholar
    • Export Citation
  • Han, Y., , P. van Delst, , Q. Liu, , F. Weng, , B. Yan, , R. Treadon, , and J. Derber, 2006: JCSDA Community Radiative Transfer Model (CRTM), version 1. NOAA Tech Rep. NESDIS 122, National Oceanic and Atmospheric Administration, National Environmental Satellite, Data, and Information Service, 33 pp. [Available online at ftp://ftp.emc.ncep.noaa.gov/jcsda/CRTM/CRTM_v1-NOAA_Tech_Report_NESDIS122.pdf.]

  • Hu, M., , S. Weygandt, , M. Xue, , and S. Benjamin, 2006: Development and testing of a new cloud analysis package using radar, satellite, and surface cloud observations within GSI for initializing Rapid Refresh. 18th Conf. on Numerical Weather Prediction, Park City, UT, Amer. Meteor. Soc., P2.5. [Available online at https://ams.confex.com/ams/pdfpapers/124496.pdf.]

  • Ingleby, N. B., , and A. C. Lorenc, 1993: Bayesian quality control using multivariate normal distributions. Quart. J. Roy. Meteor. Soc.,119, 11951225, doi:10.1002/qj.49711951316.

    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., 1986: Analysis methods for numerical weather prediction. Quart. J. Roy. Meteor. Soc.,112, 11771194, doi:10.1002/qj.49711247414.

    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., , and O. Hammon, 1988: Objective quality control of observations using Bayesian methods. Theory, and a practical implementation. Quart. J. Roy. Meteor. Soc.,114, 515543, doi:10.1002/qj.49711448012.

    • Search Google Scholar
    • Export Citation
  • McNally, A. P., 2002: A note on the occurrence of cloud in meteorologically sensitive areas and the implications for advanced infrared sounders. Quart. J. Roy. Meteor. Soc., 128, 25512556, doi:10.1256/qj.01.206.

    • Search Google Scholar
    • Export Citation
  • McNally, A. P., 2009: The direct assimilation of cloud-affected satellite infrared radiances in the ECMWF 4D-Var. Quart. J. Roy. Meteor. Soc., 135, 12141229, doi:10.1002/qj.426.

    • Search Google Scholar
    • Export Citation
  • McNally, A. P., , and P. D. Watts, 2003: A cloud detection algorithm for high spectral resolution infrared sounders. Quart. J. Roy. Meteor. Soc., 129, 34113423, doi:10.1256/qj.02.208.

    • Search Google Scholar
    • Export Citation
  • Morrison, H., , J. A. Curry, , and V. I. Khvorostyanov, 2005: A new double-moment microphysics parameterization for application in cloud and climate models. Part I: Description. J. Atmos. Sci., 62, 16651677, doi:10.1175/JAS3446.1.

    • Search Google Scholar
    • Export Citation
  • Pangaud, T., , N. Fourrie, , V. Guidard, , M. Dahoui, , and F. Rabier, 2009: Assimilation of AIRS radiances affected by mid- to low-level clouds. Mon. Wea. Rev., 137, 42764292, doi:10.1175/2009MWR3020.1.

    • Search Google Scholar
    • Export Citation
  • Pavelin, E. G., , S. J. English, , and J. R. Eyre, 2008: The assimilation of cloud-affected infrared satellite radiances for numerical weather prediction. Quart. J. Roy. Meteor. Soc., 134, 737749, doi:10.1002/qj.243.

    • Search Google Scholar
    • Export Citation
  • Skamarock, W. C., , J. B. Klemp, , J. Dudhia, , D. O. Gill, , D. M. Barker, , W. Wang, , and J. G. Powers, 2005: A description of the advanced research WRF version 2. NCAR Tech. Note NCAR/TN-468+STR, 88 pp. [Available online at http://www.mmm.ucar.edu/wrf/users/docs/arw_v2.pdf.]

  • Wu, M.-L. C., 1987: A method for remote sensing the emissivity, fractional cloud cover and cloud top temperature of high-level, thin clouds. J. Climate Appl. Meteor., 26, 225233, doi:10.1175/1520-0450(1987)026<0225:AMFRST>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
Save