• Arking, A., 1991: The radiative effects of clouds and their impact on climate. Bull. Amer. Meteor. Soc., 72 , 795813.

  • Beesley, J. A., , C. S. Bretherton, , C. Jakob, , E. L. Andreas, , J. M. Intrieri, , and T. A. Uttal, 2000: A comparison of cloud and boundary layer variables in the ECMWF forecast model with observations at Surface Heat Budget of the Arctic Ocean (SHEBA) ice camp. J. Geophys. Res., 105 , 1233712349.

    • Search Google Scholar
    • Export Citation
  • Chen, W-N., , C-W. Chiang, , and J-B. Nee, 2002: Lidar ratio and depolarization ratio for cirrus clouds. Appl. Opt., 41 , 64706476.

  • Chepfer, H., , M. Chiriaco, , R. Vautard, , and J. Spinhirne, 2007: Evaluation of MM5 optically thin clouds over Europe in fall using ICESat lidar spaceborne observations. Mon. Wea. Rev., 135 , 27372753.

    • Search Google Scholar
    • Export Citation
  • Chiriaco, M., , R. Vautard, , H. Chepfer, , M. Haeffelin, , J. Dudhia, , Y. Wanherdrick, , Y. Morille, , and A. Protat, 2006: The ability of MM5 to simulate ice clouds: Systematic comparison between simulated and measured fluxes and lidar/radar profiles at the SIRTA atmospheric observatory. Mon. Wea. Rev., 134 , 897918.

    • Search Google Scholar
    • Export Citation
  • Delanoë, J., , and R. J. Hogan, 2008: A variational scheme for retrieving ice cloud properties from combined radar, lidar, and infrared radiometer. J. Geophys. Res., 113 .D07204, doi:10.1029/2007JD009000.

    • Search Google Scholar
    • Export Citation
  • Foot, J. S., 1988: Some observations of the optical properties of clouds. II: Cirrus. Quart. J. Roy. Meteor. Soc., 114 , 145164.

  • Gettelman, A., , P. M. de F. Forster, , M. Fujiwara, , Q. Fu, , H. Vömel, , L. K. Gohar, , C. Johanson, , and M. Ammerman, 2004: Radiation balance of the tropical tropopause layer. J. Geophys. Res., 109 .D07103, doi:10.1029/2003JD004190.

    • Search Google Scholar
    • Export Citation
  • Haynes, J. M., , R. T. Marchand, , Z. Luo, , A. Bodas-Salcedo, , and G. L. Stephens, 2007: A multi-purpose radar simulation package: Quickbeam. Bull. Amer. Meteor. Soc., 88 , 17231727.

    • Search Google Scholar
    • Export Citation
  • Hogan, R. J., 2006: Fast, approximate calculation of multiply scattered lidar returns. Appl. Opt., 45 , 59845992.

  • Hogan, R. J., , and A. Battaglia, 2008: Fast lidar and radar multiple-scattering models. Part II: Wide-angle scattering using the time-dependent two-stream approximation. J. Atmos. Sci., in press.

    • Search Google Scholar
    • Export Citation
  • Hogan, R. J., , C. Jakob, , and A. J. Illingworth, 2001: Comparison of ECMWF winter-season cloud fraction with radar-derived values. J. Appl. Meteor., 40 , 513525.

    • Search Google Scholar
    • Export Citation
  • Hogan, R. J., , D. P. Donovan, , C. Tinel, , M. E. Brooks, , D. Bouniol, , A. J. Illingworth, , and J. P. V. Poiares Baptista, 2006: Independent evaluation of the ability of spacebourne radar and lidar to retrieve the microphysical and radiative properties of ice clouds. J. Atmos. Oceanic Technol., 23 , 211227.

    • Search Google Scholar
    • Export Citation
  • Illingworth, A. J., and Coauthors, 2007: Cloudnet—Continuous evaluation of cloud profiles in seven operational models using ground-based observations. Bull. Amer. Meteor. Soc., 88 , 883898.

    • Search Google Scholar
    • Export Citation
  • Jakob, C., 1994: The impact of the new cloud scheme on ECMWF’s intergrated forecasting system (IFS). Proc. ECMWF/GEWEX Workshop on Modelling, Validation and Assimilation of Clouds, Reading, United Kingdom, ECMWF, 277–294.

    • Search Google Scholar
    • Export Citation
  • Jakob, C., 1999: Cloud cover in the ECMWF reanalysis. J. Climate, 12 , 947959.

  • Mace, G. G., , C. Jakob, , and K. P. Moran, 1998: Validation of hydrometeor occurrence predicted by the ECMWF model using millimeter wave radar data. Geophys. Res. Lett., 25 , 16451648.

    • Search Google Scholar
    • Export Citation
  • Mace, G. C., , M. Deng, , B. Soden, , and E. Zipser, 2006: Association of tropical cirrus in the 10–15-km layer with deep convective sources: An observational study combining millimeter radar data and satellite-derived trajectories. J. Atmos. Sci., 63 , 480503.

    • Search Google Scholar
    • Export Citation
  • Martin, G. M., , D. W. Johnson, , and A. Spice, 1994: The measurement and parameterization of effective radius of droplets in warm stratocumulus clouds. J. Atmos. Sci., 51 , 18231842.

    • Search Google Scholar
    • Export Citation
  • McCormick, M. P., and Coauthors, 1993: Scientific investigations planned for the Lidar In-Space Technology Experiment (LITE). Bull. Amer. Meteor. Soc., 74 , 205214.

    • Search Google Scholar
    • Export Citation
  • Miller, S. D., , G. L. Stephens, , and A. C. M. Beljaars, 1999: A validation survey of the ECMWF prognostic cloud scheme using LITE. Geophys. Res. Lett., 26 , 14171420.

    • Search Google Scholar
    • Export Citation
  • Morcrette, J-J., , and C. Jakob, 2000: The response of the ECMWF model to changes in the cloud overlap assumption. Mon. Wea. Rev., 128 , 17071732.

    • Search Google Scholar
    • Export Citation
  • O’Connor, E. J., , R. J. Hogan, , and A. J. Illingworth, 2005: Retrieving stratocumulus drizzle parameters using Doppler radar and lidar. J. Appl. Meteor., 44 , 1427.

    • Search Google Scholar
    • Export Citation
  • Ou, S-C., , and K-N. Liou, 1995: Ice microphysics and climatic temperature feedback. Atmos. Res., 35 , 127138.

  • Palm, S. P., , A. Benedetti, , and J. D. Spinhirne, 2005: Validation of ECMWF global forecast model parameters using GLAS atmospheric channel measurements. Geophys. Res. Lett., 32 .L22S09, doi:10.1029/2005GL023535.

    • Search Google Scholar
    • Export Citation
  • Pinnick, R. G., , S. G. Jennings, , P. Chýlek, , C. Ham, , and W. T. J. Grandy, 1983: Backscatter and extinction in water clouds. J. Geophys. Res., 88 , 67876796.

    • Search Google Scholar
    • Export Citation
  • Platt, C. M. R., 1973: Lidar and radiometric observations of cirrus clouds. J. Atmos. Sci., 30 , 11911204.

  • Platt, C. M. R., , D. M. Winker, , M. A. Vaughan, , and S. D. Miller, 1999: Backscatter-to-extinction ratios in the top layers of tropical mesoscale convective systems and in isolated cirrus from LITE observations. J. Appl. Meteor., 38 , 13301345.

    • Search Google Scholar
    • Export Citation
  • Quante, M., 2004: The role of clouds in the climate system. J. Phys. IV France, 121 , 6186.

  • Solomon, S., , D. Qin, , M. Manning, , M. Marquis, , K. Averyt, , M. M. B. Tignor, , H. L. Miller Jr., , and Z. Chen, 2007: Climate Change 2007: The Physical Sciences Basis. Cambridge University Press, 996 pp.

    • Search Google Scholar
    • Export Citation
  • Stephens, G. L., , S. C. Tsay, , P. W. Stackhouse Jr., , and P. Flatau, 1990: The relevance of the microphysical and radiative properaties of cirrus clouds to climate and climate feedback. J. Atmos. Sci., 47 , 17421753.

    • Search Google Scholar
    • Export Citation
  • Stephenson, D. B., 2000: Use of the “odds ratio” for diagnosing forecast skill. Wea. Forecasting, 15 , 221232.

  • Tiedtke, M., 1993: Representation of clouds in large-scale models. Mon. Wea. Rev., 121 , 30403061.

  • Vaughan, M., , S. Young, , D. Winker, , K. Powwell, , A. Omar, , Z. Liu, , Y. Hu, , and C. Hostetler, 2004: Fully automated analysis of space-based lidar data: An overview of the CALIPSO retrieval algorithms and data products. Laser Radar Techniques for Atmospheric Sensing, U. N. Singh, Ed., International Society for Optical Engineering (SPIE Proceedings, Vol. 5575), 16–30.

  • Wetzel, M. A., , and G. T. Bates, 1995: Comparison of simulated cloud cover with satellite observations over the western United States. J. Climate, 8 , 296314.

    • Search Google Scholar
    • Export Citation
  • Winker, D. M., , and L. R. Poole, 1995: Monte-Carlo calculations of cloud returns for ground-based and space-based lidars. Appl. Phys. B, 60 , 341344.

    • Search Google Scholar
    • Export Citation
  • Zwally, H. J., and Coauthors, 2002: ICESat’s laser measurements of polar ice, atmosphere, ocean and land. J. Geodyn., 34 , 405445.

  • View in gallery

    Calculation of the backscatter that would be measured by ICESat in (a) a 4-km-thick ice cloud with an optical depth of 4, and (b) a 1-km-thick liquid cloud with an optical depth of 10. The black lines show the results of the Hogan and Battaglia (2008) multiple-scattering model using the lidar characteristics given in Table 1 with two values of effective radius. The difference between these two lines is predominantly due to the difference in scattering asymmetry factor, but is also affected by the difference in the width of the forward-scattered lobe in the phase function. The gray solid line shows the approximation due to Platt (1973) with η = 0.5. Also shown is the backscatter due to single-scattering alone (equivalent to Platt’s approximation with η = 1).

  • View in gallery

    (a) High-resolution simulation of extinction coefficient, α (m−1), using ECMWF variables and Eq. (2). (b) Corresponding simulation of backscatter (sr−1 m−1), using the lidar forward model. The data are taken at approximately 0118–0121 UTC 30 Sep 2003 and from longitudes −52° to −63° on the right-hand side. In both (a) and (b), the vertical black lines show the model horizontal grid resolution. The influence of the maximum-random overlap can be seen in each plot. The rapid change in α and β close to 4 km and between latitudes −78° and −74° is caused by a change in phase in the model from ice to liquid cloud.

  • View in gallery

    Demonstration of the processing of ECMWF model data at 0117–0129 UTC 30 Sep 2003, with longitudes from −27° to −74°. This corresponds to a longer time series of the same data presented in Fig. 2, the location of which is shown with black dashed lines. (a) Simulated lidar forward model backscatter (sr−1 m−1), corrected for instrument sensitivity, on the model grid. (b) ECMWF model cloud fraction in its raw state. (c) The ECMWF cloud fraction after the lidar forward model has been run. The corresponding observations are shown in Fig. 4. Note the horizontal grid resolution in Fig. 2 is 10 times as high as that in this figure to show subgrid-scale cloud variability. The vertical grid resolution is also much less in this figure. Therefore, not all cloud features match perfectly.

  • View in gallery

    Processing of ICESat lidar data for the same case as in Fig. 3. (a) Attenuated backscatter (m−1 sr−1) (b) after processing to remove noise and aerosol and (c) when averaged on to the model grid, and (d) cloud fraction calculated on the model grid.

  • View in gallery

    (a) Lat and height variation of mean ECMWF model cloud fraction for 15 days from 0000 UTC 30 Sep 2003 to 2359 UTC 14 Oct 2003 under the ICESat track. (b) Cloud fraction from the ECMWF model from the same time period as (a), but after the lidar forward model had been run and the cloud fraction resampled. (c) ICESat cloud fraction data for the same period as in (a) and (b). The assumed lidar ratio for the forward model is 20 sr.

  • View in gallery

    (top) Mean cloud fraction, (middle) mean frequency of cloud occurrence, and (bottom) mean amount when present for the tropics, midlatitudes, and polar regions in each hemisphere. The thick solid line indicates the ICESat lidar measurements, the thin solid line shows the results of the lidar forward model applied to the ECMWF model, and the dashed line shows the raw ECMWF model cloud fraction. The gray area shows the errors likely on the lidar forward model, determined by the method in section 2e. The lat bands to which each column are referring is given at the top of each column.

  • View in gallery

    Comparison of ICESat lidar and the lidar forward model applied to the ECMWF model, for the 15-day study period, where the only points selected are those above the land surface. (a) ICESat lidar mean cloud fraction, (b) ICESat lidar frequency of occurrence, (c) ICESat lidar amount when present, (d) lidar forward model mean cloud fraction, (e) forward model frequency of occurrence, and (f) forward model amount when present.

  • View in gallery

    As in Fig. 7, but for grid points over the sea only.

  • View in gallery

    (top) Mean backscatter and (bottom) backscatter when present taken from ICESat and the ECMWF model over the 15-day study period. The lat region of each profile is denoted at the top in the top row.

  • View in gallery

    (top) The latitude (grouped into bands of 30° width) and height variation of ETS and (bottom) the log of odds ratio for the ECMWF model, after the lidar forward model has been run.

  • View in gallery

    Global mean variation of cloud statistics and lidar simulations with changing effective radius in the lidar forward model: (a) mean cloud fraction, (b) frequency of occurrence, (c) amount when present, (d) mean attenuated backscatter, and (e) backscatter when present.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 105 105 18
PDF Downloads 25 25 8

Use of a Lidar Forward Model for Global Comparisons of Cloud Fraction between the ICESat Lidar and the ECMWF Model

View More View Less
  • 1 Department of Meteorology, University of Reading, Reading, United Kingdom
  • | 2 ECMWF, Shinfield Park, Reading, United Kingdom
© Get Permissions
Full access

Abstract

The performance of the European Centre for Medium-Range Weather Forecasts (ECMWF) model in simulating clouds is evaluated using observations by the Geoscience Laser Altimeter System lidar on the Ice, Cloud, and Land Elevation Satellite (ICESat). To account for lidar attenuation in the comparison, model variables are used to simulate the attenuated backscatter using a lidar forward model. This generates a new model cloud fraction that can then be fairly compared with the ICESat lidar. The lidar forward model and ICESat comparison is performed over 15 days (equivalent to 226 orbits of Earth, or roughly 9 million km) of data. The model is assessed by cloud fraction statistics, skill scores, and its ability to simulate lidar backscatter. The results show that the model generally simulates the occurrence and location of clouds well but overestimates the mean amount when present of the ice cloud by around 10%, particularly in the tropics. The skill of the model is slightly better over the land than over the sea. The model also has some problems representing the amount when present in tropical boundary layer cloud, particularly over land, where there is an underestimate by as much as 15%. Calculations of backscatter reveal that the ECMWF model predicts the lidar backscatter to within 5% on average, for a lidar ratio of 20 sr, apart from in thick ice clouds. Sensitivity tests show that realistic variations in extinction-to-backscatter ratio and effective radius affect the forward modeled mean cloud fraction by no more than 10%.

* Current affiliation: Met Office, Exeter, United Kingdom.

Corresponding author address: R. J. Hogan, Department of Meteorology, Earley Gate, P.O. Box 243, Reading RG6 6BB, United Kingdom. Email: r.j.hogan@reading.ac.uk

Abstract

The performance of the European Centre for Medium-Range Weather Forecasts (ECMWF) model in simulating clouds is evaluated using observations by the Geoscience Laser Altimeter System lidar on the Ice, Cloud, and Land Elevation Satellite (ICESat). To account for lidar attenuation in the comparison, model variables are used to simulate the attenuated backscatter using a lidar forward model. This generates a new model cloud fraction that can then be fairly compared with the ICESat lidar. The lidar forward model and ICESat comparison is performed over 15 days (equivalent to 226 orbits of Earth, or roughly 9 million km) of data. The model is assessed by cloud fraction statistics, skill scores, and its ability to simulate lidar backscatter. The results show that the model generally simulates the occurrence and location of clouds well but overestimates the mean amount when present of the ice cloud by around 10%, particularly in the tropics. The skill of the model is slightly better over the land than over the sea. The model also has some problems representing the amount when present in tropical boundary layer cloud, particularly over land, where there is an underestimate by as much as 15%. Calculations of backscatter reveal that the ECMWF model predicts the lidar backscatter to within 5% on average, for a lidar ratio of 20 sr, apart from in thick ice clouds. Sensitivity tests show that realistic variations in extinction-to-backscatter ratio and effective radius affect the forward modeled mean cloud fraction by no more than 10%.

* Current affiliation: Met Office, Exeter, United Kingdom.

Corresponding author address: R. J. Hogan, Department of Meteorology, Earley Gate, P.O. Box 243, Reading RG6 6BB, United Kingdom. Email: r.j.hogan@reading.ac.uk

1. Introduction

Clouds play a major role in the earth’s radiation budget and predictions of future climate (Stephens et al. 1990; Solomon et al. 2007; Quante 2004), yet both ice and liquid clouds are often poorly represented in general circulation models (Arking 1991), which is one of the major factors limiting the accuracy of future climate predictions. In numerical weather prediction, clouds are important for a number of reasons, including the model radiation scheme, surface temperature forecasts, visibility, aircraft icing forecasts, and their role in the formation of precipitation.

There have been a number of previous studies assessing the clouds within models. For example, Wetzel and Bates (1995) compared measurements from the Geostationary Operational Environmental Satellites (GOES) with the fifth-generation Pennsylvania State University–National Center for Atmospheric Research Mesoscale Model (MM5) model clouds and found a negative bias of up to 30% in model cloudiness. Because of its global domain, many comparison studies have been performed with the European Centre for Medium-Range Weather Forecasts (ECMWF) model. Jakob (1999) compared ECMWF reanalysis data with observations from the International Satellite Cloud Climatology Project (ISCCP) for the period July 1983–December 1990. The reanalysis used the prognostic ECMWF cloud scheme [Tiedtke (1993), modified by Jakob (1994)] and it was found that the ECMWF reanalysis (ERA) data tended to underestimate cloud cover in the extratropical oceans, the trade wind cumulus, the stratocumulus sheets off the west coast of subtropical continents and the summertime convective cloud over Eurasia.

Previous active ground-based studies have also assessed the performance of clouds in the ECMWF model. Mace et al. (1998) used a 35-GHz radar in Oklahoma to compare the frequency of cloud occurrence with the ECMWF model. They concluded that the ECMWF model layers were generally too deep and were predicted within the model too early (by an unspecified time). Hogan et al. (2001) used data from 35- and 94-GHz radars, and a lidar ceilometer located at Chilbolton, (southern England) United Kingdom, to evaluate the ECMWF model mean cloud fraction. Their results showed that the model tended to have too little cloud beneath 7 km and too much cloud above 7 km. However, once the model cloud fraction was modified to account for precipitating snow and to remove any thin ice cloud that cannot be detected by the radar, the comparisons were better, but the model still had a tendency to overestimate the amount of high ice cloud by a factor of 2. Illingworth et al. (2007) took this to a new level in the Cloudnet project, combining radar, lidar, and microwave radiometers from three sites in Europe to evaluate cloud fraction and water content in seven operational models (including ECMWF) over a period of several years.

The use of active instruments in space offers the potential of both global coverage and high vertical resolution. A lidar flew on board the space shuttle Discovery in September 1994 as part of the Lidar In-Space Technology Experiment (LITE; McCormick et al. 1993). Miller et al. (1999) used the 532-nm channel data from 15 LITE orbits to evaluate the ECMWF model cloud frequency of occurrence using statistical skill scores of hit rate, threat score, probability of detection, and false alarm rate. They concluded that the model generally tended to overestimate the total amount of cloud, but made no account for attenuation of the lidar signal through thick ice cloud or liquid water cloud.

Launched in 2003, the Ice, Cloud and Land Elevation Satellite (ICESat; Zwally et al. 2002) is a polar-orbiting satellite upon which is mounted the Geoscience Laser Altimeter Survey (GLAS) instrument. It consists of a nadir-pointing lidar that, although intended primarily to measure the height of the polar ice sheets, is also able to make measurements of ice cloud and thin liquid water clouds. These data can be used to evaluate the performance of general circulation models (GCMs) and, unlike LITE, are capable of showing the seasonal variation of global cloud profiles.

ICESat lidar data have been available long enough to be used to observe clouds and perform model comparison studies. Palm et al. (2005), took one orbit of ICESat data and compared it with cloud fraction from the ECMWF model. By making direct comparisons between cloud fraction and using skill scores from Miller et al. (1999), they deduced that the ECMWF model was capable of representing low clouds quite well but often produced too much high cloud, particularly evident from long (48 h) forecasts. Model skill scores also decreased with increasing forecast length. However, one must be very careful when comparing observations made by lidar instruments directly to clouds as there will be a loss of signal power (attenuation) as the beam passes through clouds, and often there will be a total extinction of the signal in liquid water clouds. Palm et al. (2005) estimated that this occurs around 10% of the time. However, to achieve a fair comparison between the lidar and model, a method of accounting for the attenuation of the signal must be found. In addition, we should be careful in comparing models directly with observations, as sometimes two different variables are being compared.

This study attempts to make a fair comparison between the ICESat lidar and the ECMWF model by following a similar approach to that of Chiriaco et al. (2006) and Chepfer et al. (2007), and using model variables to simulate the lidar signal rather than trying to use the lidar observations to retrieve model variables. This means that we account for the attenuation that occurs as the lidar passes through thick clouds. Simulation also has applications for data assimilation, as it allows model variables to be used to predict observations. In this case, it could form the basis of a cloud assimilation scheme within the model.

Section 2 describes the methodology used to derive the lidar signal from the ECMWF model variables. Processing of the ICESat lidar signal is necessary to remove noise and to place the data upon the ECMWF model grid; this is described in section 3. Once processing is complete, various statistical tests can be performed to judge the model’s performance, which are described in section 4, with the associated results. A summary is presented in section 5.

2. Processing of ECMWF model data

This section describes the lidar forward model. The ECMWF model versions used were IFS cycles 26r1 and 26r3. The model analyses were taken every 12 h with intervening 3-h forecasts. The grid spacing is 40 km in the horizontal.

a. Lidar equation

Following the method of Platt (1973), the lidar equation may be written as the following approximation:
i1520-0493-136-10-3742-e1
where β′(z) (sr−1 m−1) is the attenuated backscatter at altitude z (m) above the earth’s surface and zlid is the orbit altitude (m) of ICESat. The extinction-to-backscatter ratio or “lidar ratio” is denoted by s and multiple scattering is represented by the inclusion of the dimensionless multiple-scattering factor η. This will be explained further in section 2d. The visible extinction coefficient is α (m−1). At the ICESat 532-nm wavelength, there is only scattering by particles and molecules and hence α(z) inside the integral in (1) is due to scattering and not absorption. Collectively the α(z)/s term represents the unattenuated backscatter while the terms within the exponential represent the attenuation of the lidar signal.

The principle of the lidar forward model is to first use ECMWF model variables to predict the value of the visible extinction coefficient α at each vertical grid box within the model. If we can obtain an estimate of the lidar ratio s and the multiple-scattering factor η, then the attenuated backscatter (β′) can be predicted at each level of the model and then compared with ICESat data.

b. Converting model variables to the extinction coefficient

To calculate the value of the visible extinction coefficient α from model variables at each individual grid box, we use the following expression (after Foot 1988 and assuming a scattering efficiency of 2, which is a valid assumption for cloud particles observed at 532 nm):
i1520-0493-136-10-3742-e2
where IWC (kg m−3) is the model ice water content and LWC (kg m−3) is the model liquid water content at that particular grid box. The values ρl and ρi are the densities (kg m−3) of liquid water and solid ice, respectively. The calculation uses the ECMWF model parameterizations of the effective radius of ice rei (m), which is taken as a function of temperature following Ou and Liou (1995), but setting the constraint that rei can only vary between 30 and 60 μm. Liquid water effective radius rel (m) follows the parameterization of Martin et al. (1994), with the concentration of cloud condensation nuclei over the ocean held constant at 50 cm−1 and over the land held constant at 900 cm−1. Typical values of the effective radius produced by the Martin et al. (1994) parameterization are between 5 and 9 μm for continental cloud and between 8 and 15 μm for maritime cloud. Once the profiles of extinction coefficient have been calculated, the lidar forward model can be run but it is important that the forward model takes account of the cloud overlap scheme used within the ECMWF model. This is discussed further in section 2h.

c. Lidar ratio

One unknown variable within (1) is the value of the lidar ratio s. This takes the values of around 18 sr for liquid water (Pinnick et al. 1983; O’Connor et al. 2005) but can vary between 10 and 40 sr for ice particles (Platt et al. 1999; Chen et al. 2002). In this experiment, we shall take the value of the lidar ratio to be 20 sr, as it is close to the constant for liquid water, and it is easy to test the sensitivity to the maximum and minimum values by doubling and halving the value of s. We shall examine the sensitivity of changing the lidar ratio in ice between the extreme values on the final result incorporated in the error estimates.

d. Calculation of the multiple-scattering factor

An important consideration in (1) is the multiple-scattering factor η, introduced by Platt (1973). This can vary between 0.5 and 1 depending on the altitude of the satellite, the telescope field of view, the wavelength of the radiation, and the size of the particles the radiation encounters. A multiple-scattering factor of 1 is the single scattering limit, often appropriate for ground-based lidar, where all the scattered photons are lost except for those directly backscattered to the instrument. The value of 0.5 is the wide field-of-view limit, often more appropriate for spaceborne lidars, where narrowly forward scattered photons remain within the telescope field of view.

Figure 1 shows a comparison of the simple approximation of Platt (1973) using η = 0.5 with the sophisticated (and computationally slower) multiple-scattering model of Hogan and Battaglia (2008), for idealized ice and liquid clouds. This model combines the Hogan (2006) treatment of small-angle scattering with a new method to calculate the contribution from wide-angle scattering. It can be seen that for ice clouds the Platt approximation performs very well; this is because the large particles lead to a narrow forward-scattered lobe in the phase function, and virtually all these forward-scattered photons remain within the large field of view of the lidar. There is a small error at cloud base due to the neglect of the pulse-stretching effect, although the associated backscatter is below the sensitivity threshold of the lidar. In the case of liquid clouds, the pulse-stretching effect (Winker and Poole 1995) is significantly greater due to the higher optical depth. Even so, given that signals below 10−6 m−1 sr−1 at heights lower than 3 km tend to be rejected, the error in forward-modeled cloud fraction will be small.

Given these results, the use of Platt’s simplified representation of multiple scattering is adequate, and the value of η = 0.5 is used for all forward model calculations. Nonetheless, a sensitivity test to η is included in the error analysis in the next section. It should be noted that Platt’s representation is not very accurate when simulating the molecular return beneath clouds; in this paper we do not attempt to forward model the molecular return, and remove it from the observations before performing the comparison. However, any future lidar forward models attempting to simulate the molecular return subject to multiple scattering would need to use a more sophisticated model, such as that of Hogan and Battaglia (2008).

e. Lidar forward model error estimation

The results of (1) are sensitive to the values of the water contents (ice and liquid), the assumed effective radii of liquid and ice particles, the lidar ratio, and the multiple-scattering factor. As mentioned earlier, the water contents and effective radii are determined directly from the ECMWF model assumptions; as this study intends to compare this model with ICESat, should the values of effective radius and water content be wrong, it is a problem with the ECMWF model and not the lidar forward model. To determine the errors on the lidar forward model, only the variation of the lidar ratio and the multiple-scattering factor are examined, although a sensitivity to effective radius will be examined in section 4f.

To examine the sensitivity of the model results to changes in the lidar ratio s, three different values were used. In the control case, a midrange value of s of 20 sr was used. This is close to the value predicted for liquid water (18 sr; Pinnick et al. 1983; O’Connor et al. 2005). However, studies (Platt et al. 1999; Chen et al. 2002) have indicated that s can vary in ice between values of 10 and 40 sr, and due to the complex particle shapes, tend not to be simple functions of particle size as might be expected from Mie theory. Hence these values were used as the maximum extremes of the lidar ratio and were used in the calculation of the error bars.

To examine the sensitivity of the multiple-scattering factor, this was also varied. However, the values of the multiple-scattering factor are around the minimum value suggested by Platt (1973) and a value lower that 0.5 would be unphysical. So the multiple-scattering factor within these experiments was allowed to vary between 0.5 and 0.6.

Examining (1), it can be seen that a high value of the multiple-scattering factor causes the most attenuation and the smallest value of the attenuated backscatter and vice versa. The highest value of the lidar ratio causes the smallest value of attenuated backscatter and vice versa. Hence, to assess the errors on the lidar forward model, three experiments were performed on one day of ECMWF model data (10 October 2003, chosen as the ICESat latitude–altitude analysis showed clouds in many different regions of the globe and more so than any other day). The first experiment was run with lidar ratio of 20 sr and multiple-scattering factor of 0.5 (as the control experiment); the second had a value of lidar ratio of 40 sr and a multiple-scattering factor of 0.6; the third had a lidar ratio of 10 sr and a multiple-scattering factor of 0.5. The differences between the results were determined by percentage errors showing the difference between the three experiments and are included in the results presented in section 4. The estimated error (typically around 10%, but varying with latitude and height) for this one day contained a large data sample, which was assumed to be representative of the error over the whole 15-day study period.

f. Interpolation of model data on to the ICESat height grid

It should be noted that (1) can be inaccurate if discretized on too course a grid. This could potentially create sharp changes in ice and liquid water contents, leading to increased extinction of the signal at a single point in the forward modeled column. For this reason, the model water contents are interpolated on to the ICESat height grid, which also has the advantage of showing where total extinction would occur within a model grid box. However, in order to calculate extinction, it is important to show where the model sky is clear and where it is cloudy. This is because if the grid box is assumed to be completely full, attenuation will be much greater and the signal strength will be reduced. Hence, the raw model cloud fraction was interpolated using a nearest-neighbor interpolation method. This avoids vertical subgrid variability in the cloud fraction profile, which would be unfaithful to the modeled cloud overlap assumptions.

However, this higher resolution is only used to run the lidar forward model. In section 2i we shall see how the cloud fraction is recalculated on the model grid, allowing a direct comparison between the ICESat lidar and the ECMWF model before and after running the lidar forward model.

g. Selecting model grid points

The model grid points nearest (in time and space) to the ICESat track are extracted. The horizontal grid spacing of the ECMWF model is roughly 40 km. To analyze 1 day of ICESat data, the time and location (i.e., latitude and longitude) of the satellite are extracted from the orbit information. The ECMWF analyses are available at 0000 and 1200 UTC, whereas intervening forecasts are available at 3-h intervals.

The ICESat track points are arranged into nine time groups corresponding to the closest ECMWF data available. ICESat points between 0000 and 0130 UTC are associated with the 0000 model analysis, and the ICESat points between 0130 and 0430 UTC are associated with the 0300 UTC model forecast. This process continues in 3-h groups, until 2230 UTC. The points between 2230 and 0000 UTC are associated with the relevant 0000 UTC analysis of the following day.

Once the ICESat points have been grouped in time, ECMWF model profiles closest to the ICESat ground track are extracted, containing a vertical profile of the model variables cloud fraction, relative humidity, temperature, wind speed, ice, and liquid water contents.

h. Cloud overlap

An important factor in the lidar forward model calculations is the use of maximum-random overlap as currently implemented in the ECMWF model (Morcrette and Jakob 2000). To achieve maximum-random overlap, each model grid box is first divided into 10 subcolumns horizontally, with cloud fraction in each model grid box rounded to the nearest 10%. In preliminary experiments, 10 subcolumns were found to provide a good balance between computer run time and accuracy. The process is shown in Fig. 2. Once the details of the maximum-random overlap are known and the values of all the variables in (1) are known, the lidar forward model can be applied to calculate backscatter using the model variables interpolated on to the ICESat height grid.

i. Cloud fraction resampling

To compare the cloud fraction from the lidar forward model with cloud fraction from ICESat, the cloud fraction data are “resampled.” At this stage, the lidar forward model is on a high-resolution grid with 10 subcolumns to each model grid box in the horizontal and the ICESat height grid in the vertical. To resample the cloud fraction, the model grid is superimposed on to the forward model attenuated backscatter, and the fraction of points containing backscatter above the sensitivity threshold (minimum detectable backscatter of ICESat, shown in in Table 1) is calculated. Any points below the sensitivity threshold are ignored when subsequently calculating the cloud fraction.

Figure 2 shows detail of how the maximum-random overlap works for the lidar forward model. Between 7.5 and 10 km and −80° to −76°, a “ghost cloud1” is visible with low extinction coefficient in Fig. 2a, which corresponds to an ice water content of around 10−5 g m−3; only part of this cloud (that which is optically thick enough to be detected) is present in Fig. 2b. Between −78.5° and −74° and beneath 3 km, total extinction of the lidar signal takes place. This can be seen by high values of extinction coefficient in Fig. 2a, which do not translate into a backscatter signal in Fig. 2b.

Figure 3 shows the simulated attenuated lidar backscatter from the lidar forward model on the ECMWF model grid, along with the raw ECMWF mean cloud fraction and the lidar forward model cloud fraction. From the lidar forward model cloud fraction, it can be seen that the cloud fraction has been reduced both above around 9 km (where the model cloud would be too thin for the lidar to detect) and below around 6 km (where attenuation would mean that there would be no cloud observed).

3. Processing of lidar data

Processing of the ICESat lidar data is carried out to remove any return that is not cloud and to map the observations on to the ECMWF grid. The properties of the ICESat lidar as used in these experiments are shown in Table 1.

a. Rejection of unwanted (noncloud) signal

To clean molecular echoes and instrument noise from the data, the profiles are divided into 4-min periods (equivalent to 1680 km) and the mean backscatter above 15 km for this period is used to remove the molecular noise from the lower levels for the whole 4 min. It is assumed that at this level, there is no cloud and that all the signal received is instrument noise. The mean and standard deviation of the linear backscatter at these heights are obtained. This is used to set a noise threshold for the rest of the profile. The threshold at which a signal is accepted to be a cloud signal rather than noise is two standard deviations above the mean, chosen empirically. If a higher threshold is set, then the clouds start to be removed. With a lower threshold, obvious noise still remains within the data. The threshold of two standard deviations is sufficient to remove most noise down to ground level. A speckle filter is then used to remove any remaining isolated pixels. In addition, aerosol has to be removed from the data. Since the highest values of aerosol have similar values of backscatter as thin ice cloud, aerosol pixels would be above the noise threshold defined for the 4-min period. Aerosol pixels tend to exist as small groups of adjacent data points with backscatter high enough to allow them to exist after the single pixels have been removed. It is therefore important that we ensure only aerosol (and not cloud) is removed from the data.

To remove the aerosol, the lowest 3 km of the data are divided into a series of grid boxes. As aerosol tends to have relatively low backscatter values compared to liquid water cloud, all points within the box with low backscatter (values less than 5 × 10−5 m−1 sr−1) are located and the standard deviation of these points is calculated. Since aerosol has a low standard deviation and cloud a higher standard deviation (backscatter values increase sharply within clouds from the edges), the points with a very low standard deviation (less than 3 × 10−6 m−1 sr−1) are assumed to be aerosol, and removed.

Naturally, the choice of removal box size is important to the aerosol and low-altitude molecular removal process. If the grid boxes are too large, as much as 20% low-backscatter ice cloud will be removed, but if the grid boxes are too small then there will be insufficient data to provide a good sample of the variance of the low backscatter used to remove the aerosol. The best results occurred when grid boxes of 50 pixels horizontally by 20 pixels vertically were used, where one lidar pixel is 75 m in the vertical and 1.24 km in the horizontal. This gives 1000 data points in total in each grid box, which is a large enough sample for variance calculation.

Another obvious issue with the aerosol removal method is the removal of any low-backscatter cloud data that occurred in the same box as the aerosol that we were trying to remove. In some cases, the edges of the clouds were actually removed from grid boxes containing data. However, upon closer examination, this would only lead to the removal of a few points, which we estimate would equate to less than a 4% error in the cloud fraction calculated. A digital elevation map was used to remove the high backscatter return from the ground up to and including 75 m above the surface.

b. Averaging of ICESat backscatter data

To compare backscatter from the ECMWF model and backscatter from the ICESat lidar, ICESat profiles that correspond to the same ECMWF model grid box were averaged in the horizontal while maintaining the ICESat vertical resolution. Figure 4 shows the processing of ICESat lidar data to remove noise and to average on to the model grid to give mean backscatter and cloud fraction. The data correspond to the same time period as shown in Fig. 3.

The data were averaged on to the model grid selecting and averaging ICESat 5-Hz rays to the nearest ECMWF model grid point. Occasionally observed is apparent horizontal stretching and squashing of the original data as they are transformed on to the model horizontal grid. This phenomenon is due to the grouping of ICESat 5-Hz (1.2 km) rays to individual ECMWF model grid points. Some model grid points will be the average of many ICESat 5-Hz rays while some will only be the average of a few. However, any model grid point that corresponded to less than 5 ICESat 5-Hz rays (7 km of ground track) was rejected from the analysis. The largest boxes were 28 rays (35 km). The maximum distance between ICESat track and model grid points is 20 km. The maximum time difference between observations is 1.5 h. Cloud fraction was then calculated on the model grid by using the same method as used for the lidar forward model cloud fraction resampling, described in section 2i, with the number of ICESat profiles replacing the subcolumns used in the lidar forward model.

4. Results

Fifteen days of ICESat observations, from 30 September 2003 to 14 October 2003 inclusive, have been compared with the corresponding ECMWF model data after running the lidar forward model. This has generated a dataset where the length of the ICESat track is over 9 million km. This is equivalent to over 14 yr of observations at a midlatitude ground-based station, assuming a mean wind over the station of 20 m s−1. Although it is desirable to examine the seasonal variations in cloud climatology, this study is restricted to a short data period because of the availability of ICESat data at the time of processing that were of the same release version and of good quality. Data were specifically selected from a period just after a new laser was switched on, to avoid the problems caused by laser signal degradation.

a. The effect of running the lidar forward model on ECMWF clouds

As we saw in Fig. 3, there was a reduction of model cloud fraction due to the processing of the lidar forward model. Cloud fraction is reduced at high altitudes because of the backscatter produced being lower than the sensitivity of the ICESat lidar. Cloud fraction is also reduced at lower altitudes because of the attenuation of the lidar beam as it passes through thick cloud.

Figure 5 shows the effect of this processing on the ECMWF model mean cloud fraction. Comparing plots in Figs. 5a,b, the following points are noted:

  • There is a large reduction of model cloud fraction in the tropics, at heights above 10 km. This is also notable in the midlatitudes in the highest ice clouds. The cause of this reduction could be due to thin ice cloud falling beneath the ICESat sensitivity threshold.
  • In midlatitudes and high latitudes and at high altitudes, the cloud fraction taken from the lidar forward model is close to the original ECMWF cloud fraction.
  • At the lowest altitudes, mean cloud fraction is reduced from around 0.1–0.3 to around 0.05–0.1 due to attenuation of the lidar signal in the higher ice cloud. This is important for any comparisons between model and lidar that are made; comparing the ICESat-derived cloud fraction directly with the ECMWF model would wrongly indicate that the model had too much cloud.

b. Performance of the ECMWF model versus latitude and height

Figures 5b,c show that the ECMWF model has clouds in approximately the correct position at all latitudes, but in places the cloud amounts are substantially higher than with the ICESat observations. In particular, the mean cloud fraction in the ice cloud above the tropics and above the Poles is too large by up to 10%. Interestingly, the low cloud above the North Pole and above the Southern Ocean is up to 15% too large as well. It is uncertain exactly why this may occur, but it could be due to incorrect parameterizations of supercooled liquid water and ice within the model.

To examine the performance of the ECMWF model over several different latitude bands and different altitudes, each hemisphere was divided into three latitude regions. In each region, following the work of Hogan et al. (2001) the mean frequency of cloud occurrence and mean cloud amount when present was calculated for altitudes up to 15 km; above this level the comparison is not robust because of the assumption made earlier that all the signals received at ICESat are purely noise. Frequency of occurrence is the fraction of time that a cloud occurs within the model or observations at these latitudes. It is related to the dynamics of the model and humidity—the processes that actually determine whether a cloud is present or not. Amount when present is the mean cloud fraction over the grid boxes within the latitude band when a cloud is detected. It is more related to the performance of the cloud scheme. A cloud is deemed to be detected when the cloud fraction threshold exceeds a threshold value of 0.05.

Figure 6 can be used to show how the model performs over the 15-day period. By comparing the thick solid line (ICESat observations) with the thin solid line (processed ECMWF model), we note the following:

  • In the tropics, the model mean cloud fraction at altitudes above 10 km is too large by up to 0.1. Throughout the rest of the profile, the tropical mean cloud fraction is very low, but agrees with the ICESat observations.
  • Similarly, the tropical frequency of occurrence is too large by as much as 0.15 above 8 km and the amount when present in the tropics is too large by as much as 0.15 above 12 km. Beneath 7 km, the topical frequency of occurrence is too low, by as much as 0.1.
  • In the midlatitudes, a similar picture emerges. Mean cloud fraction is too high by up to 0.1 between the altitudes of 5 and 10 km, but agrees with the ICESat observations otherwise. The frequency of occurrence is too large by up to 0.05 between 3 and 12 km, but agrees with the observations at other altitudes. The amount when present is too large by up to 0.05 between 3 and 12 km. At other altitudes, it agrees with the observations, apart from in the Southern Hemisphere midlatitude band, where above 11 km, the amount when present is too low by as much as 0.2.
  • In the polar regions, mean cloud fraction is too large by up to 0.1, at altitudes beneath 10 km. In the same height range, the frequency of occurrence is too large by up to 0.05 and the amount when present is too large by up to 0.1.
The most likely explanation for this behavior is a problem with the ice scheme within the model. It has been noted before that the main anvil outflow in the tropics is not immediately beneath the cold-point tropopause at around 18 km, but several kilometers lower at the base of the “tropical tropopause layer,” a transition region between troposphere and stratosphere (Gettelman et al. 2004). This explains our finding a peak in mean cloud fraction at around 12 km in the tropics, which is in agreement with ground-based observations (e.g., Mace et al. 2006). By contrast, upper-tropospheric cirrus in midlatitude and polar regions often does extend up to the tropopause, which typically lies between around 8 and 12 km.

Most of the regions where the mean cloud fraction is too high are regions in which we would expect to find ice clouds. Mace et al. (1998) in the southern United States, Beesley et al. (2000) in the polar regions, and Hogan et al. (2001) in the midlatitudes noticed on average that the ECMWF model had too much ice cloud. There is often too much ice present within the higher altitudes. This is most clearly seen by the higher amount when present in the midlatitudes above 7 km.

Although the frequency of occurrence and amount when present are accurate to within 0.05 for most of the lower troposphere beneath 5 km, in the tropics the amount when present is often slightly underestimated by the ECMWF model. This could be due to one of two reasons: either the model underestimates the amount of low cloud in the tropics or that the large amount of ice cloud is attenuating the lidar forward model backscatter sufficiently that some of the lowest clouds are not detectable according to the lidar forward model. Forward modeling of attenuation has the advantage that it is fair to the model, yet conversely, it can sometimes be more difficult to work out what is wrong when differences occur. Hence, although the error estimate for the frequency of occurrence is rather low, the results must be interpreted with caution for these regions of the globe. For high-altitude ice clouds where there is little or no attenuation in the regions between the cloud and the satellite, we can be confident that the ECMWF model has a genuine overestimate. However, in regions where the attenuation is much greater, we cannot be certain.

c. Land and sea comparisons

The performance of the model over the land and the sea surface can be examined by dividing model grid boxes into those that are mostly over the land surface and those that are mostly above the sea surface, using a digital elevation map. Results from performing the analysis over the land surface are shown in Fig. 7 and over the sea in Fig. 8, revealing the following:

  • The model has clouds in roughly the right places above the land and sea surface. Exceptions are the cloud top of the tropical ice cloud, which is approximately 1 km too high. The cloud top over Antarctica is up to 2 km lower in the model than in the observations over the land, and as much as 1 km lower over the sea. The model is more able to accurately represent the locations of boundary layer cloud over the sea.
  • The mean cloud fraction in the model is too high by as much as 0.1 throughout the cloud shown in Fig. 7d. Clouds are only present in very small quantities in the model beneath 8 km and in the tropics. The mean cloud fraction in the tropical boundary layer is rather poorly represented over the land. Over the sea, the mean cloud fraction in the highest clouds in the model is too large by as much as 0.2. This is also true for the higher midlatitudes and the Poles.
  • The frequency of occurrence in the highest cloud at each latitude is too high by up to 0.15 over the land surface. The area of the largest frequency of occurrence, in the tropical ice cloud above 10 km, is slightly larger than in the observations over the land. The frequency of occurrence over the land in the clouds poleward of 50° is too large by roughly 0.15. Over the sea, in the high clouds the frequency of occurrence is too large by up to 0.3, reducing over the Poles to 0.2. The frequency of boundary layer cloud is roughly correct in the model.
  • The amount when present above the land surface is only slightly too large (by up to 0.1) in the majority of the modeled atmosphere, but is too large by up to 0.2 in the polar ice clouds and the Southern Hemisphere ice clouds. Over the sea, the modeled cloud amount when present is generally too large by up to 0.2 across all latitudes and altitudes. However, the amount when present is greatest in the ice clouds above the poles and the ice clouds in the tropics. In the tropical midtroposphere, between 2 and 8 km, the amount when present is too low, by up to 0.2.
Comparing Figs. 7 and 8, we note that over the land, the highest tropical clouds (above 10 km) have a cloud fraction between 0.1 and 0.2; whereas over the sea, mean cloud fraction here is never more than 0.1. It appears that is due to the clouds over land being much more frequently observed than over sea, by a factor of 2. The ITCZ over land appears to be constrained to a small area between 10 and 15 km and just north of the equator, while over the sea the high cloud spreads out between 0° and 20°N, with the thickest cloud between 10 and 12 km. The model gets these different relations in cloud over the land and the sea in the right places, but with too high cloud fraction and frequency of occurrence. It appears that in general, the high-altitude ice clouds over the sea are much more poorly represented than over the land.

Most of the low clouds that were observed at all latitudes in Fig. 6 can be seen in Fig. 8 to occur over the sea surface. This is most likely marine stratocumulus, found in the eastern extremes of ocean basins.

d. Comparisons of attenuated backscatter

The analysis can be extended to see how well the ECMWF model predicts the attenuated backscatter that is observed by ICESat. This can be done by comparing the mean attenuated backscatter over time for different locations around the globe. In addition, a new variable, “backscatter when present” (represented by the symbol βWP) can be derived. This is analogous to the amount when present variable used earlier; the backscatter when present variable consists of the mean attenuated backscatter of all the detected cloud pixels (those above the minimum sensitivity left after any noise removal), while the mean backscatter includes all pixels, including those that have no cloud. Hence, for this reason, it should be noted that mean backscatter can drop beneath the ICESat sensitivity threshold, but βWP will remain above the sensitivity threshold.

Analysis of the mean attenuated backscatter and βWP has been carried out for same the 15-day study period as the cloud fraction climatology analysis. The results have been binned into six latitude bands as before and are shown in Fig. 9. Errors on the lidar forward model, as determined from section 4e have been included on the plots. In general, the ECMWF-simulated mean backscatter is well within the experimental error, but there are a couple of exceptions. In the tropics above 10 km, the mean attenuated backscatter is too high by a factor of 3, and the βWP is too high by a factor of 2, although the latter is just within the bounds of the experimental error. These results correspond to the high cloud fraction and frequency of occurrence noted in the tropics in Fig. 6.

The βWP beneath 5 km in the tropics is also too high, which could mean that the clouds in the boundary layer are too optically thick. This would not necessarily show up in the mean backscatter plots as the backscatter when present includes only points where there is cloud and does not average in any clear-sky regions. As we have seen in Fig. 6, the global cloud-fraction amount when present in the tropical lower troposphere is actually underestimated, so this result could be due to an error in the effective radius in the model for these clouds. The sensitivity to effective radius will be examined later in section 4f. In other regions, the mean attenuated backscatter and backscatter when present are in reasonably good agreement with the observations. The assumed value of the lidar ratio will directly scale the ECMWF backscatter values; however, the effect on cloud fraction of a change in lidar ratio is much smaller as it only affects clouds relative to the sensitivity threshold chosen.

Poleward of −60°, it appears that the lidar forward model model is underestimating the backscatter from ICESat above 10 km by a factor of 3. This corresponds to the lower cloud top seen in Fig. 5.

e. Skill scores

The comparisons so far have evaluated the ability of the model to reproduce the climatological distribution of clouds at a give latitude and height. Skill scores can be introduced to examine the performance of the forecast in terms of whether the clouds occur at the right time and space. This is done by comparing each model grid box of cloud fraction (after running the lidar forward model) with that derived from ICESat. Using a cloud fraction threshold of 0.05, the forecast at each grid box can then be determined as a “hit,” a “false alarm,” a “miss,” or a “correct rejection” as denoted a–d in Table 2. Points where there is total extinction are neglected from the score.

Many different skill scores have been developed for cloud fraction evaluation within models. For example, Mace et al. (1998) and Miller et al. (1999) used hit rate, threat score, false alarm rate, and probability of detection. However, a good skill score should have a low dependence on the frequency of occurrence, and a random (no skill) forecast should always get a low score, close to zero. Hence, this study will concentrate on two of the more useful skill scores, the equitable threat score (ETS) used by Illingworth et al. (2007) and the odds ratio, (θ; Stephenson 2000), defined as follows:
i1520-0493-136-10-3742-e3
i1520-0493-136-10-3742-e4
Both scores are dimensionless. The equitable threat score gives values of zero for a random forecast up to a maximum of 1 (a perfect forecast). It is designed so that it will equal zero if the total number of correct forecasts (a + d) equals the total number of incorrect forecasts (b + c). Forecasts that are worse than a random guess will score a weakly negative value. The minimum for the equitable threat score is −⅓. The odds ratio provides similar information, but it is common to take the natural log of the odds ratio (LOR) for each point, as the odds ratio can vary over several orders of magnitude. For a random forecast, LOR is zero, but for a good forecast it will be unbounded. The odds ratio tends to break down in regions where the event being tested is rare, so the equitable threat score should be more reliable at quantifying model performance in areas of low mean cloud fraction.

In our analysis, points where total extinction occurs have been set to undefined values and hence, are not included in the results. Figure 10 shows both skill scores for the different latitude regions. The following can be seen:

  • Both sets of scores show an increase in skill as we progress from the surface to around 5–10 km. This is most likely due to the model being able to successfully produce higher cloud, which is generally associated with large-scale features, but having difficulty in resolving the small-scale boundary layer cloud.
  • The results from both skill scores show that the ECMWF model is most skillful within the Southern Hemisphere midlatitude storm tracks and is least skillful within the tropical lower atmosphere. This is presumably because the precise location of tropical convection is difficult to get correct in the model.
  • The equitable threat score tends to fall above 10 km. Skill scores are generally low within the tropics above 12.5 km and for polar clouds tend to drop off sharply above 10 km. However, because of the smaller number of points included within the study from these regions, these results are less robust.

f. Sensitivity to effective radius

In section 4e it was noted that effective radius was predefined by the ECMWF model and hence was not be included within the error bars of the lidar forward model. To test the lidar forward model response to changes in the effective radius, two experiments were performed: one with both liquid and ice effective radius doubled and the other with effective radius halved. Figure 11 shows the resulting changes to mean cloud fraction, frequency of occurrence, amount when present, mean backscatter, and backscatter when present averaged over the globe. Examining (2), we see that a doubling in effective radius is also equivalent to a halving of ice and liquid water contents and hence, these results also indicate the sensitivity to changing of cloud water contents. Changing the effective radius tends to change the ice phase in a different way to the liquid phase in the lidar forward model. In (1), we can see that α appears twice, both within the exponential reducing the apparent backscatter and outside the exponential increasing the apparent backscatter. At the top of ice clouds, the attenuation term is small when compared to the attenuation by liquid clouds and the terms outside the exponential. In the case of liquid clouds, α is large with respect to ice and the exponential term will tend to dominate. Referring to Eqs. (1) and (2), it can be seen that a decrease in effective radius will lead to more attenuation and a smaller backscatter, meaning less cloud is detected. Hence, the sensitivity lines in each plot cross at some altitude within the profile. This does not have to be the same altitude in each profile. For the cloud fraction statistics, the crossover generally occurs in the midtroposphere. For the backscatter statistics, the crossover point occurs in the lower troposphere, as this is where the most optically thick clouds lie.

For ice clouds, the attenuation part of (1) will tend to be small. This is due to small values of α within these clouds and due to the fact that the lidar signal has not met many clouds in order to cause extinction. Hence, the main contribution to the backscatter will be from the visible extinction coefficient outside of the exponential and then doubling and halving the effective radius will tend to double and halve the attenuated backscatter and backscatter when present as shown in Figs. 11d,e.

When considering the mean cloud fraction (Fig. 11a) and the amount when present (Fig. 11c), the higher cloud amounts vary by up to 0.04 in response to the change in effective radius. This is due to more (or less) clouds falling within the ICES at sensitivity threshold. Frequency of occurrence does not change by more than 0.02. This is because for there to be a large change in frequency of occurrence, clouds have to appear in a large number of model grid boxes that were free of clouds in the control run, yet cloudy in the raw ECMWF model cloud fraction. In most cases, changing the effective radius simply changes the cloud amount in a grid box where clouds were already present in the control run.

At lower altitudes in each profile, the situation is not as simple. Changing the effective radius for liquid clouds allows more or less attenuation to take place, but since the attenuation term is dominant for liquid clouds, a decrease in effective radius can increase the extinction coefficient and act to reduce the cloud fraction in a grid box. The sensitivity of the mean backscatter and backscatter when present at altitudes beneath 5 km is very small, and the change that is present is probably due to increased attenuation.

5. Conclusions and future work

This study has evaluated a forward-modeling technique to compare lidar observations with an operational weather model. Forward modeling is useful for radar and lidar observations, as it allows a fair comparison with the model, accounting for attenuation of the signal due to clouds. However, when differences arise between the observed and simulated cloud fraction statistics, it can be difficult to pinpoint the cause of the difference.

A comparison has been made between the ECMWF model and observations from the ICESat lidar over a 15-day period. It has been shown that the model has a tendency to produce too much ice cloud within the model, which will affect the formation of precipitation and the way the model interacts with incoming and outgoing radiation. Tests have also shown that the forward model has a 10% error due to the largest unknowns within the lidar forward model: the lidar ratio (varied by a factor of 2) and the multiple-scattering factor (varied by 20%). In addition, the doubling or halving effective radius results in up to a 0.05 change in cloud fraction, although this is not strictly an error in the forward model.

Comparing the results with those of Palm et al. (2005), we find agreement in the verdict on the performance of the ECMWF model at high altitudes, but a different assessment at low altitudes where attenuation is important. Palm et al. (2005) noted a higher frequency of ice cloud within the model than observed, with the largest overestimate above 12 km, which agrees with the tropical ice cloud overestimate shown in Fig. 6. Other experiments also show that the ECMWF model produces too much ice cloud. The results obtained from this study for the 30°–60° latitude region agree very well with Hogan et al. (2001), although the large model overestimate in the amount when present in clouds above 8 km that was observed by Hogan et al. (2001) is not as large in this study.

Comparison of the ECMWF model with lidar observations via a forward model is an essential precursor to systems involving the data assimilation of lidar data into a forecast model. In this case, the forecast model would be used to predict backscatter from the lidar, followed by changes to the water content to reduce the difference between the backscatter signals in a least squares sense. This method could also be used for testing new model schemes.

Although this study has shown the value of active remote sensing from space, it has concentrated on only a few days of data. Longer periods of lidar data are now available from the Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) lidar, which is part of the A-Train of satellites (Vaughan et al. 2004). So far, these data do not suffer from the laser degradation problems that have affected long-term measurements with the ICESat satellite. However, because of the strong attenuation of liquid water clouds from the lidar, measurements by radar from space, such as CloudSat will be more suited to model evaluation for thicker clouds. Indeed, a CloudSat simulator has been developed by Haynes et al. (2007), which enables the same forward-modeling approach taken in this paper to be applied to radar. Nonetheless, lidars are very important to sense optically thin ice clouds. By combining radar and lidar (Hogan et al. 2006; Delanoë and Hogan 2008), the vast majority of clouds within the atmosphere can be detected and characterized, allowing for a fairer evaluation of models of the earth’s climate system.

Acknowledgments

We thank Ewan O’Connor for computer support and Jim Spinhirne and Alan Blyth for useful discussions. The first author was supported by an NERC studentship. This work was also supported by NERC Grant NE/D002370/1.

REFERENCES

  • Arking, A., 1991: The radiative effects of clouds and their impact on climate. Bull. Amer. Meteor. Soc., 72 , 795813.

  • Beesley, J. A., , C. S. Bretherton, , C. Jakob, , E. L. Andreas, , J. M. Intrieri, , and T. A. Uttal, 2000: A comparison of cloud and boundary layer variables in the ECMWF forecast model with observations at Surface Heat Budget of the Arctic Ocean (SHEBA) ice camp. J. Geophys. Res., 105 , 1233712349.

    • Search Google Scholar
    • Export Citation
  • Chen, W-N., , C-W. Chiang, , and J-B. Nee, 2002: Lidar ratio and depolarization ratio for cirrus clouds. Appl. Opt., 41 , 64706476.

  • Chepfer, H., , M. Chiriaco, , R. Vautard, , and J. Spinhirne, 2007: Evaluation of MM5 optically thin clouds over Europe in fall using ICESat lidar spaceborne observations. Mon. Wea. Rev., 135 , 27372753.

    • Search Google Scholar
    • Export Citation
  • Chiriaco, M., , R. Vautard, , H. Chepfer, , M. Haeffelin, , J. Dudhia, , Y. Wanherdrick, , Y. Morille, , and A. Protat, 2006: The ability of MM5 to simulate ice clouds: Systematic comparison between simulated and measured fluxes and lidar/radar profiles at the SIRTA atmospheric observatory. Mon. Wea. Rev., 134 , 897918.

    • Search Google Scholar
    • Export Citation
  • Delanoë, J., , and R. J. Hogan, 2008: A variational scheme for retrieving ice cloud properties from combined radar, lidar, and infrared radiometer. J. Geophys. Res., 113 .D07204, doi:10.1029/2007JD009000.

    • Search Google Scholar
    • Export Citation
  • Foot, J. S., 1988: Some observations of the optical properties of clouds. II: Cirrus. Quart. J. Roy. Meteor. Soc., 114 , 145164.

  • Gettelman, A., , P. M. de F. Forster, , M. Fujiwara, , Q. Fu, , H. Vömel, , L. K. Gohar, , C. Johanson, , and M. Ammerman, 2004: Radiation balance of the tropical tropopause layer. J. Geophys. Res., 109 .D07103, doi:10.1029/2003JD004190.

    • Search Google Scholar
    • Export Citation
  • Haynes, J. M., , R. T. Marchand, , Z. Luo, , A. Bodas-Salcedo, , and G. L. Stephens, 2007: A multi-purpose radar simulation package: Quickbeam. Bull. Amer. Meteor. Soc., 88 , 17231727.

    • Search Google Scholar
    • Export Citation
  • Hogan, R. J., 2006: Fast, approximate calculation of multiply scattered lidar returns. Appl. Opt., 45 , 59845992.

  • Hogan, R. J., , and A. Battaglia, 2008: Fast lidar and radar multiple-scattering models. Part II: Wide-angle scattering using the time-dependent two-stream approximation. J. Atmos. Sci., in press.

    • Search Google Scholar
    • Export Citation
  • Hogan, R. J., , C. Jakob, , and A. J. Illingworth, 2001: Comparison of ECMWF winter-season cloud fraction with radar-derived values. J. Appl. Meteor., 40 , 513525.

    • Search Google Scholar
    • Export Citation
  • Hogan, R. J., , D. P. Donovan, , C. Tinel, , M. E. Brooks, , D. Bouniol, , A. J. Illingworth, , and J. P. V. Poiares Baptista, 2006: Independent evaluation of the ability of spacebourne radar and lidar to retrieve the microphysical and radiative properties of ice clouds. J. Atmos. Oceanic Technol., 23 , 211227.

    • Search Google Scholar
    • Export Citation
  • Illingworth, A. J., and Coauthors, 2007: Cloudnet—Continuous evaluation of cloud profiles in seven operational models using ground-based observations. Bull. Amer. Meteor. Soc., 88 , 883898.

    • Search Google Scholar
    • Export Citation
  • Jakob, C., 1994: The impact of the new cloud scheme on ECMWF’s intergrated forecasting system (IFS). Proc. ECMWF/GEWEX Workshop on Modelling, Validation and Assimilation of Clouds, Reading, United Kingdom, ECMWF, 277–294.

    • Search Google Scholar
    • Export Citation
  • Jakob, C., 1999: Cloud cover in the ECMWF reanalysis. J. Climate, 12 , 947959.

  • Mace, G. G., , C. Jakob, , and K. P. Moran, 1998: Validation of hydrometeor occurrence predicted by the ECMWF model using millimeter wave radar data. Geophys. Res. Lett., 25 , 16451648.

    • Search Google Scholar
    • Export Citation
  • Mace, G. C., , M. Deng, , B. Soden, , and E. Zipser, 2006: Association of tropical cirrus in the 10–15-km layer with deep convective sources: An observational study combining millimeter radar data and satellite-derived trajectories. J. Atmos. Sci., 63 , 480503.

    • Search Google Scholar
    • Export Citation
  • Martin, G. M., , D. W. Johnson, , and A. Spice, 1994: The measurement and parameterization of effective radius of droplets in warm stratocumulus clouds. J. Atmos. Sci., 51 , 18231842.

    • Search Google Scholar
    • Export Citation
  • McCormick, M. P., and Coauthors, 1993: Scientific investigations planned for the Lidar In-Space Technology Experiment (LITE). Bull. Amer. Meteor. Soc., 74 , 205214.

    • Search Google Scholar
    • Export Citation
  • Miller, S. D., , G. L. Stephens, , and A. C. M. Beljaars, 1999: A validation survey of the ECMWF prognostic cloud scheme using LITE. Geophys. Res. Lett., 26 , 14171420.

    • Search Google Scholar
    • Export Citation
  • Morcrette, J-J., , and C. Jakob, 2000: The response of the ECMWF model to changes in the cloud overlap assumption. Mon. Wea. Rev., 128 , 17071732.

    • Search Google Scholar
    • Export Citation
  • O’Connor, E. J., , R. J. Hogan, , and A. J. Illingworth, 2005: Retrieving stratocumulus drizzle parameters using Doppler radar and lidar. J. Appl. Meteor., 44 , 1427.

    • Search Google Scholar
    • Export Citation
  • Ou, S-C., , and K-N. Liou, 1995: Ice microphysics and climatic temperature feedback. Atmos. Res., 35 , 127138.

  • Palm, S. P., , A. Benedetti, , and J. D. Spinhirne, 2005: Validation of ECMWF global forecast model parameters using GLAS atmospheric channel measurements. Geophys. Res. Lett., 32 .L22S09, doi:10.1029/2005GL023535.

    • Search Google Scholar
    • Export Citation
  • Pinnick, R. G., , S. G. Jennings, , P. Chýlek, , C. Ham, , and W. T. J. Grandy, 1983: Backscatter and extinction in water clouds. J. Geophys. Res., 88 , 67876796.

    • Search Google Scholar
    • Export Citation
  • Platt, C. M. R., 1973: Lidar and radiometric observations of cirrus clouds. J. Atmos. Sci., 30 , 11911204.

  • Platt, C. M. R., , D. M. Winker, , M. A. Vaughan, , and S. D. Miller, 1999: Backscatter-to-extinction ratios in the top layers of tropical mesoscale convective systems and in isolated cirrus from LITE observations. J. Appl. Meteor., 38 , 13301345.

    • Search Google Scholar
    • Export Citation
  • Quante, M., 2004: The role of clouds in the climate system. J. Phys. IV France, 121 , 6186.

  • Solomon, S., , D. Qin, , M. Manning, , M. Marquis, , K. Averyt, , M. M. B. Tignor, , H. L. Miller Jr., , and Z. Chen, 2007: Climate Change 2007: The Physical Sciences Basis. Cambridge University Press, 996 pp.

    • Search Google Scholar
    • Export Citation
  • Stephens, G. L., , S. C. Tsay, , P. W. Stackhouse Jr., , and P. Flatau, 1990: The relevance of the microphysical and radiative properaties of cirrus clouds to climate and climate feedback. J. Atmos. Sci., 47 , 17421753.

    • Search Google Scholar
    • Export Citation
  • Stephenson, D. B., 2000: Use of the “odds ratio” for diagnosing forecast skill. Wea. Forecasting, 15 , 221232.

  • Tiedtke, M., 1993: Representation of clouds in large-scale models. Mon. Wea. Rev., 121 , 30403061.

  • Vaughan, M., , S. Young, , D. Winker, , K. Powwell, , A. Omar, , Z. Liu, , Y. Hu, , and C. Hostetler, 2004: Fully automated analysis of space-based lidar data: An overview of the CALIPSO retrieval algorithms and data products. Laser Radar Techniques for Atmospheric Sensing, U. N. Singh, Ed., International Society for Optical Engineering (SPIE Proceedings, Vol. 5575), 16–30.

  • Wetzel, M. A., , and G. T. Bates, 1995: Comparison of simulated cloud cover with satellite observations over the western United States. J. Climate, 8 , 296314.

    • Search Google Scholar
    • Export Citation
  • Winker, D. M., , and L. R. Poole, 1995: Monte-Carlo calculations of cloud returns for ground-based and space-based lidars. Appl. Phys. B, 60 , 341344.

    • Search Google Scholar
    • Export Citation
  • Zwally, H. J., and Coauthors, 2002: ICESat’s laser measurements of polar ice, atmosphere, ocean and land. J. Geodyn., 34 , 405445.

Fig. 1.
Fig. 1.

Calculation of the backscatter that would be measured by ICESat in (a) a 4-km-thick ice cloud with an optical depth of 4, and (b) a 1-km-thick liquid cloud with an optical depth of 10. The black lines show the results of the Hogan and Battaglia (2008) multiple-scattering model using the lidar characteristics given in Table 1 with two values of effective radius. The difference between these two lines is predominantly due to the difference in scattering asymmetry factor, but is also affected by the difference in the width of the forward-scattered lobe in the phase function. The gray solid line shows the approximation due to Platt (1973) with η = 0.5. Also shown is the backscatter due to single-scattering alone (equivalent to Platt’s approximation with η = 1).

Citation: Monthly Weather Review 136, 10; 10.1175/2008MWR2309.1

Fig. 2.
Fig. 2.

(a) High-resolution simulation of extinction coefficient, α (m−1), using ECMWF variables and Eq. (2). (b) Corresponding simulation of backscatter (sr−1 m−1), using the lidar forward model. The data are taken at approximately 0118–0121 UTC 30 Sep 2003 and from longitudes −52° to −63° on the right-hand side. In both (a) and (b), the vertical black lines show the model horizontal grid resolution. The influence of the maximum-random overlap can be seen in each plot. The rapid change in α and β close to 4 km and between latitudes −78° and −74° is caused by a change in phase in the model from ice to liquid cloud.

Citation: Monthly Weather Review 136, 10; 10.1175/2008MWR2309.1

Fig. 3.
Fig. 3.

Demonstration of the processing of ECMWF model data at 0117–0129 UTC 30 Sep 2003, with longitudes from −27° to −74°. This corresponds to a longer time series of the same data presented in Fig. 2, the location of which is shown with black dashed lines. (a) Simulated lidar forward model backscatter (sr−1 m−1), corrected for instrument sensitivity, on the model grid. (b) ECMWF model cloud fraction in its raw state. (c) The ECMWF cloud fraction after the lidar forward model has been run. The corresponding observations are shown in Fig. 4. Note the horizontal grid resolution in Fig. 2 is 10 times as high as that in this figure to show subgrid-scale cloud variability. The vertical grid resolution is also much less in this figure. Therefore, not all cloud features match perfectly.

Citation: Monthly Weather Review 136, 10; 10.1175/2008MWR2309.1

Fig. 4.
Fig. 4.

Processing of ICESat lidar data for the same case as in Fig. 3. (a) Attenuated backscatter (m−1 sr−1) (b) after processing to remove noise and aerosol and (c) when averaged on to the model grid, and (d) cloud fraction calculated on the model grid.

Citation: Monthly Weather Review 136, 10; 10.1175/2008MWR2309.1

Fig. 5.
Fig. 5.

(a) Lat and height variation of mean ECMWF model cloud fraction for 15 days from 0000 UTC 30 Sep 2003 to 2359 UTC 14 Oct 2003 under the ICESat track. (b) Cloud fraction from the ECMWF model from the same time period as (a), but after the lidar forward model had been run and the cloud fraction resampled. (c) ICESat cloud fraction data for the same period as in (a) and (b). The assumed lidar ratio for the forward model is 20 sr.

Citation: Monthly Weather Review 136, 10; 10.1175/2008MWR2309.1

Fig. 6.
Fig. 6.

(top) Mean cloud fraction, (middle) mean frequency of cloud occurrence, and (bottom) mean amount when present for the tropics, midlatitudes, and polar regions in each hemisphere. The thick solid line indicates the ICESat lidar measurements, the thin solid line shows the results of the lidar forward model applied to the ECMWF model, and the dashed line shows the raw ECMWF model cloud fraction. The gray area shows the errors likely on the lidar forward model, determined by the method in section 2e. The lat bands to which each column are referring is given at the top of each column.

Citation: Monthly Weather Review 136, 10; 10.1175/2008MWR2309.1

Fig. 7.
Fig. 7.

Comparison of ICESat lidar and the lidar forward model applied to the ECMWF model, for the 15-day study period, where the only points selected are those above the land surface. (a) ICESat lidar mean cloud fraction, (b) ICESat lidar frequency of occurrence, (c) ICESat lidar amount when present, (d) lidar forward model mean cloud fraction, (e) forward model frequency of occurrence, and (f) forward model amount when present.

Citation: Monthly Weather Review 136, 10; 10.1175/2008MWR2309.1

Fig. 8.
Fig. 8.

As in Fig. 7, but for grid points over the sea only.

Citation: Monthly Weather Review 136, 10; 10.1175/2008MWR2309.1

Fig. 9.
Fig. 9.

(top) Mean backscatter and (bottom) backscatter when present taken from ICESat and the ECMWF model over the 15-day study period. The lat region of each profile is denoted at the top in the top row.

Citation: Monthly Weather Review 136, 10; 10.1175/2008MWR2309.1

Fig. 10.
Fig. 10.

(top) The latitude (grouped into bands of 30° width) and height variation of ETS and (bottom) the log of odds ratio for the ECMWF model, after the lidar forward model has been run.

Citation: Monthly Weather Review 136, 10; 10.1175/2008MWR2309.1

Fig. 11.
Fig. 11.

Global mean variation of cloud statistics and lidar simulations with changing effective radius in the lidar forward model: (a) mean cloud fraction, (b) frequency of occurrence, (c) amount when present, (d) mean attenuated backscatter, and (e) backscatter when present.

Citation: Monthly Weather Review 136, 10; 10.1175/2008MWR2309.1

Table 1.

ICESat characteristics relevant to this study, from Zwally et al. (2002).

Table 1.
Table 2.

Definition of skill score parameters, the letters denote the symbols used to represent these variables in the skill score equations. An extra variable, n, is the sum of a, b, c, and d.

Table 2.

1

A term used to describe a cloud with very little or no ice water content in the model, yet significant cloud fraction. Because of its low ice water content, such a cloud would not be detected by the lidar.

Save