Retrospective-analysis (or reanalysis) systems merge observations and models to provide global four-dimensional earth system data encompassing many physical and dynamical processes. Precipitation is one critical diagnostic that is not only sensitive to the observing system and model physics, but also reflects the general circulation. Climate records of observed precipitation through a merged satellite and gauge dataset provide a reference for comparison, though not without their own uncertainty. In this study, five reanalyses precipitation fields are compared with two observed data products to assess the strengths and weaknesses of the reanalyses. Taylor diagrams show the skill of the reanalyses relative to the reference dataset. While there is a general sense that the reanalyses precipitation data are improving in recent systems, it is not always the case. In some ocean regions, NCEP–NCAR reanalysis spatial patterns are closer to observed precipitation than NCEP–Department of Energy. The 40-yr ECMWF reanalysis (ERA-40) produces reasonable comparisons over Northern Hemisphere continents, but less so in the tropical oceans. On the other hand, the most recent reanalysis, the Japanese 25-yr reanalysis (JRA-25), shows good comparisons in both the Northern Hemisphere continents and the tropical oceans but contains distinct variation according to the available observing systems. The statistics and methods used are also tested on short experiments from a data assimilation system proposed to perform a satellite-era reanalysis.
Retrospective analyses (or reanalyses) have become a valuable source of data for studying weather systems and climate variability. A reanalysis system consists of a background forecast model and data assimilation routine. Input observations are irregular in space and time. The data assimilation merges the available observations with the background model forecast to generate uniform gridded data. One of the key utilities in a reanalysis is that the output generated from the model physics provides data not easily observed, but is consistent with the analyzed observed data. So, while the data are guided by the observations, model physics and uncertainties still lead to uncertainty in the resultant data products. Betts et al. (2006) summarize strengths, weaknesses, and the utility of reanalyses, especially regarding hydroclimate studies. Precipitation is one of the critical components of the water and energy cycles, but is also largely related to modeled physical parameterizations. However, global observed precipitation datasets have substantial uncertainty (Adler et al. 2001), so that in evaluating reanalysis precipitation, the uncertainty of the observations should be considered as well.
Observations assimilated into reanalysis systems and the model parameterizations each affect the subsequent forecast precipitation from the system. Additionally, the complex interactions between the model and observations also affect the reanalysis precipitation. Kalnay et al. (1996) classified precipitation as being very close to model simulated data and subject to large uncertainty. Newman et al. (2000) showed that there is internal consistency of precipitation, outgoing longwave radiation, and upper-level divergence within three different reanalyses, but the consistency between the reanalyses was very low. Chen et al. (2008a, b) show that the Hadley circulation in the 40-yr European Centre for Medium-Range Weather Forecasts (ECMWF) reanalysis (ERA-40) has changed in time significantly, but this may be more related to a spurious trend in precipitation (and coincident latent heating). This indicates that the precipitation, being an integral component of the energy and water cycles, as well as the dynamical circulation, should be a critical metric in the quality of a reanalysis for climate studies. Various regional research projects have diverse needs from reanalyses precipitation fields; regional hydrology forcing and regional net freshwater exchange are two examples.
Being a primary source of freshwater for the Arctic Sea, terrestrial drainage connects the cryosphere with the global climate system. Several studies have tried to apply reanalysis precipitation as forcing for river discharge models. Serreze and Hurst (2000) found reasonable spatial patterns at large scales and high northern latitudes in reanalyses, with some notable biases. The biases also had seasonality (better in winter, worse in summer). The precipitation bias was also related to high incoming shortwave radiation biases, which provided energy for evaporation then precipitation. Pavelsky and Smith (2006) used two reanalyses and two observed data products, finding that a few positive points in the reanalyses were offset by substantial errors in variability and trends of the data. Observations at high latitudes also have problems; for example, snow undercatch is a limitation. Serreze et al. (2003) conclude that, while needing improvements, reanalyses are useful to study the high-latitude water cycle. In the Antarctic region, Cullather et al. (1998) find that reanalyses generally agree on the main features of precipitation, but focusing on any region may lead to discrepancies. Bromwich et al. (2000) showed that the teleconnections between ENSO and Antarctic precipitation are influenced by how effectively observations input to the reanalysis are used.
Basin-scale studies allow comprehensive budget studies and the potential for independent observations to validate reanalysis systems. Hagemann and Gates (2001) used large basin-scale discharge to intercompare reanalyses and identify weaknesses in the reanalyses physics. Fekete et al. (2004) also computed runoff from observed and reanalysis precipitation, and found the largest errors and sensitivity in arid and semiarid regions. Basin-scale studies also allow for the evaluation of the coupling of the water and energy cycles in reanalyses (as in Roads and Betts 2000; Fernandes et al. 2008), and also the assessment of the impact of observations through the data assimilation and the spinup in the subsequent forecast (Viterbo and Betts 1999). The difference between a background forecast and the verifying analysis is called the analysis increment and may be considered the error of the background model. Schubert and Chang (1996) used multiple linear regression and the time series of analysis increments of atmospheric water and the atmospheric water budget to attribute the analysis increment contributions back to corrections of precipitation and evaporation. This method was later applied to monthly mean water budgets with favorable comparisons to observations (Bosilovich and Schubert 2001).
Janowiak et al. (1998) tested the National Centers for Environmental Prediction–National Center for Atmospheric Research (NCEP–NCAR) reanalysis precipitation with several statistical approaches: first, mean differences from an observed dataset; in this case, the Global Precipitation Climatology Project (GPCP) merged precipitation data (Adler et al. 2003) were examined. While large differences likely indicate that the reanalysis system has a bias, the merged dataset’s own uncertainties make the results less clear.
In addition to mean differences, Janowiak et al. (1998) used temporal correlations, empirical orthogonal function (EOF) analysis, and anomaly correlations. While these analysis techniques provide additional information on the reanalysis precipitation, they rely on the existence of a sufficiently long time series. When developing a new reanalysis system, a long time series or even multiple years are generally not available. Also, these time series evaluations tend to assume one or another observed precipitation dataset for comparison. However, there are differences in observed datasets relating to developing retrieval algorithms, input data, treatment of gage uncertainties, and quality flags. Gruber et al. (2000) and Yin et al. (2004) compared GPCP and Climate Prediction Center (CPC) Merged Analysis of Precipitation (CMAP) datasets (for 1979–2001) and found relatively high spatial correlation over land but significantly low correlation over ocean.
The precipitation in reanalyses is closely related not only to physical aspects of the system, but also to the assimilation of data. While the present global reanalyses do not assimilate precipitation [as in the North American Regional Reanalysis (NARR); Mesinger et al. 2006], the data assimilation strongly affects reanalysis precipitation output. Most of the studies discussed above compare one or two reanalysis systems with observations. With the recent release of the latest reanalysis from the Japanese Meteorological Agency (JMA) and plans at ECMWF, NCEP, and the National Aeronautics and Space Administration (NASA) for the third generation of reanalyses, researchers will have many considerations over which product is most applicable to their research. This paper aims to better quantify the uncertainties in precipitation from reanalysis and data assimilation systems at broad scales, and to provide a benchmark for system development. Both GPCP and CMAP will be used, approximating the uncertainty in the observed data record, and we consider the application for short developmental data assimilation experiments.
2. Data and methodology
Because GPCP (Adler et al. 2003) comprises observations with global coverage, it will be used as the reference dataset. However, GPCP precipitation does have uncertainty. The CMAP precipitation (Xie and Arkin 1996) will be compared with GPCP in an effort to represent uncertainty [as in Phillips and Gleckler (2006)]. The CMAP precipitation provides two products; one includes NCEP reanalysis information to fill missing data in the other. The CMAP observed time series will be used for comparison. CMAP contains missing observations poleward of 60°, so that no comparisons to CMAP are computed south of 60°S and only land points north of 60°N are included.
We evaluate five global atmospheric reanalyses for the period of 1979 through 2005 (if available). The Japanese 25-yr reanalysis (JRA-25) is the most recent, released for use in March 2006 (Onogi et al. 2005, 2007). In this study, we have prepared and used January 1979–December 2004 data from the JRA-25. The 40-yr ECMWF reanalysis (Uppala et al. 2005) stopped in August 2002. NCEP has released two reanalyses labeled here as NR1 (NCEP–NCAR; Kalnay et al. 1996) and NR2 [NCEP–Department of Energy (DOE); Kanamitsu et al. 2002]. We also include a reprocessing of this period using the NASA Global Earth Observing System, version 4, (GEOS4, also labeled as G4C in the figures) data assimilation system (Bloom et al. 2005). GEOS4 was the operational analysis for NASA from 2003 to 2006.
Monthly means from each of the reanalyses are used to evaluate the climatology and time series of precipitation. In the climate system, the global pattern of precipitation is as important as the mean bias of precipitation. In other words, are the reanalyses producing precipitation, or the lack thereof, in the right places? All monthly means are regridded to 2.5° × 2.5° resolution (box averaged). All spatial averaging for bias and correlation calculations uses area weighting.
3. Global and regional correlation and bias
Here we present the comparisons of the time series of annual average spatial correlations and mean differences, and the mean annual cycles of precipitation for the globe and several continental and oceanic regions (Fig. 1). In this section, we will review the regional biases in each of the reanalyses’ climatology. We will also analyze the skill at reproducing monthly and annual precipitation spatial distribution with Taylor diagrams (Taylor 2001). This will lead to a discussion on each region’s bias and spatial patterns.
a. Bias and Taylor diagrams
Figure 2 shows the climatological maps of precipitation differences for all of the reanalysis datasets under consideration as well as CMAP observations relative to the GPCP observations. The first point to make is that there are well-known systematic differences between GPCP and CMAP, notably in tropical precipitation where CMAP has the strong influence of the atoll station observations, while GPCP does not include the atoll stations (Adler et al. 2003; Yin et al. 2004; Schlosser and Houser 2007). In addition and less apparent in the annual mean, GPCP implements a precipitation under catch correction to station data, which strongly influences continental winter observations when snow occurs. All of the reanalyses tend to have tropical overestimates of precipitation. JRA-25, GEOS4, and to a certain extent ERA-40 underestimate Amazonian precipitation. Likewise, precipitation along the east coast of the midlatitude continents is generally underestimated not only by the reanalyses, but also CMAP compared to GPCP. Table 1 provides the area average GPCP precipitation climatology for the globe, several latitude bands and several continental and oceanic regions (as in Fig. 1), and differences from GPCP for the reanalyses and CMAP for January and July. This also shows the largest biases, not surprisingly in the tropics, where precipitation is already large, but also some large biases at high latitudes in the North Pacific Ocean and North America during summer. We will pursue some regional and seasonal biases in the subsequent sections.
Taylor diagrams allow evaluation of model data performance regarding the matching of spatial patterns using spatial correlation and standard deviation (Taylor 2001). Spatial correlation shows the degree to which the patterns match a reference dataset, while the standard deviation compares the amplitude of the variations. If the standard deviations are normalized to the reference dataset, then proximity to the 1, 1 coordinate location is related to the skill of the model to reproduce the spatial pattern. For example, Phillips and Gleckler (2006) used Taylor diagrams to show that an ensemble of different twentieth-century atmospheric general circulation model–simulated precipitation is a better representation of the precipitation field than any one of the contributing models. Here, we compute the spatial correlations and standard deviations from monthly mean precipitation for all the reanalyses and CMAP observations relative to GPCP. The calculations are made for each month from January 1979 to December 2005 (at the time the calculations were made, ERA-40 stopped at August 2002 and JRA-25 at December 2004), and global refers to 60°S–90°N. The calculations were performed for 20 global and regional domains. In addition to the continental and oceanic regions in Fig. 1, several latitudinal bands as well as global land and global ocean areas were evaluated.
Inclusion of CMAP along with GPCP in the Taylor diagrams allows for a certain measure of the uncertainty of the observations in the consideration of the skill of the reanalysis data. However, in a similar consideration, Taylor (2001) pointed out that different datasets may not be composed of independent observations. Indeed, GPCP and CMAP include mostly the same sources of data, though in their final form each has arrived at different data through different processing decisions. The comparison of GPCP and CMAP does not provide all of the observational uncertainty that would be found from independent observations, but rather one measure of the minimum uncertainty we should expect from a reanalysis dataset. Similarly, monthly precipitation patterns can have substantial similarity from year to year, owing to the general circulation, tropical convergence zones, land–sea contrast, or topography. So, precipitation from the same month of different years will likely yield a positive correlation. While the matched monthly correlations of GPCP and CMAP provide an expected maximum correlation, we can also define a minimum of skillful spatial correlation by correlating monthly means of different years. We call this minimum the “unmatched correlation” and interpret it as the average correlation that can be obtained simply by choosing monthly precipitation from different years, or as a climatological persistence correlation (see appendix). If a reanalysis has lower correlation than this unmatched correlation, then on average, a better pattern can be achieved from choosing, on average, any month of the climate record, and the reanalysis has little useful skill. The appendix describes the calculation of the unmatched correlations, and compares them with the matched correlations for the regions considered here.
Figure 3 shows the Taylor diagrams for annual means of the monthly correlations of precipitation in regions. The normalized standard deviations increase with radial distance from the origin. All standard deviations are normalized to GPCP so that a value of 1.0 matches GPCP. Spatial correlation are plotted as the radial lines, so that the 1, 1 point is identical to GPCP, and the linear distance of a point to 1, 1 indicates the area-weighted RMS error from GPCP after removing the means. As discussed above, because there is uncertainty in the observed dataset, plotting an observation dataset to compare the reference data will provide a secondary reference point. In this case, we use CMAP precipitation (purple points). In general, CMAP tends to be tightly grouped with close proximity to the GPCP reference point. There are cases where the spread is large and CMAP and GPCP are far apart, such as high latitudes. We will discuss details of these in regional analyses below. The purple radial line shows the average of all correlations of GPCP and CMAP as a reference. Similarly, the red radial line shows the average unmatched correlation between GPCP and CMAP (see appendix for the calculation of these values). This shows an average correlation that may be generated by different months of GPCP and CMAP, and therefore an estimate of low skill that the reanalyses should exceed.
The annual averages of correlation and standard deviation provide general assessments of the reanalyses, as the groupings of points are close. Seasonality of the errors in a system helps explain where problems may appear, but the scatter of the points increases for the reanalyses, and less so for the CMAP–GPCP comparison (Figs. 4, 5). The use of two observed datasets allows some assessment of the uncertainty of the observations (i.e., where there is large uncertainty, further developments to the reanalysis system must be validated by different means). In the following sections, we evaluate some notable regional and global variations in the reanalyses, referring back to Figs. 3 –5 but also looking at the time series of the correlation and bias.
b. Global and tropical regions
The global regions, partitioned into land and ocean averages, provide a large-scale evaluation of the precipitation. In the Taylor diagrams (Figs. 3 –5), each reanalysis annual mean is more clustered than for a given month, and starts to show the character of the reanalysis comparison to GPCP. For the annual means of global domain, JRA-25 tends to be closer to CMAP and GPCP than the other reanalyses (Fig. 3). However in the tropics, the JRA-25 points are spread out more than some other global regions. In the 30°–60°N latitude band, JRA-25 and ERA-40 are a step closer to GPCP and CMAP than the other reanalyses. For January, NR1 is farther away from the observations than the other reanalyses at high latitudes, but closer in to the 30°S–30°N band. ERA-40 has reasonable correlations in the tropical oceans in January, but the amplitude of variation is much larger than the observations and other reanalyses. In a general sense, the JRA-25 January precipitation is closest to CMAP and GPCP compared to the other reanalyses. For July precipitation (Fig. 5), JRA-25 global precipitation is the closest to GPCP and CMAP. In the tropical latitudes, NCEP reanalyses have spatial correlations less than the unmatched correlation. Indeed, most of the reanalyses monthly means for July (Fig. 5) are lower than the unmatched correlations in the 15°S–15°N latitude band, but this looks to be more related to errors from the equator to 30°N, than south of the equator. ERA-40 and JRA-25 July precipitation in 30°–60°N is closest to the CMAP and GPCP datasets.
The time series of global precipitation correlations generally show that the JRA-25 has increasing correlations with time, with a notable increasing shift in mid-1987 near the initial availability of the Special Sensor Microwave Imager (SSM/I) (Figs. 6a,c and d). There is increased correlation over the land and ocean. The JRA-25 global bias tends to be lower than most of the other reanalyses in the recent years as well. This is in large part due to improved tropical precipitation (Figs. 6d, 7d; Onogi et al. 2005, 2007). Interestingly, the GEOS4 has a different response to SSM/I availability, with a slight drop (rise) in correlation over ocean (land). ERA-40 generally shows good correlation values, compared to the other reanalyses, but the tropical precipitation bias greatly affects the time series of precipitation. An increasing precipitation trend is not apparent or significant in the global observation data. However, careful analysis of the data and removal of the El Niño–Southern Oscillation and volcanic aerosol signals in the precipitation observations yields a small but statistically significant increase of oceanic precipitation and decrease of continental precipitation (Gu et al. 2007). Likewise, model simulations of the twentieth-century climate have shown similar systematic global-scale changes in precipitation as well (Kumar et al. 2004; Bosilovich et al. 2005). The reanalyses, except for NR1, show a global increasing trend, greater than the GPCP data, in the tropics.
Roads (2003), making correlation and variance calculations for the tropics over a short 2-yr period, showed a similarity of correlations between NRI and NR2 to tropical precipitation from the Tropical Rainfall Measuring Mission (TRMM). While a similar conclusion can be found in Fig. 6d, the separation in the two reanalyses is clear when the amplitude of variation is considered (as in Figs. 3 –5). While Roads (2003) evaluated only 2 yr of the NR1 and NR2, the differences noted in that study appear to be consistent throughout the period, as shown here.
CMAP and GPCP tend to show better correlation to each other over land than ocean because of the use of gauge observations (Fig. 6). While the JRA-25 and ERA-40 correlations over ocean seem to show improvement for the more recent reanalysis there, there has been little improvement over land (when taken in the global sense). However, JRA-25 does have marginally higher land correlations in the SSM/I period than the other reanalyses. Of course, regional seasonal biases and spatial patterns (discussed next) will vary for the reanalyses.
c. Continental regions
Annual correlations for ERA-40 and JRA-25 in continental North America and Europe are generally higher than the other reanalyses (Fig. 3), though the JRA-25 standard deviations are higher than GPCP while ERA-40 is less. All of the reanalysis Eurasia points in the annual Taylor diagrams are clustered, with the exception of NR1, which is farther away, related to the mean anomalies apparent in Fig. 2. In the annual mean, no reanalysis spatial correlations exceed the mean unmatched correlation between GPCP and CMAP for Africa and South America (Fig. 3). This indicates a deficiency in the reanalyses that may affect the use of the precipitation data over South America. For example, while Betts et al. (2005) found small correctable biases between Amazon averaged precipitation observations and the ERA-40 data, the improper distribution of precipitation across the basin may affect runoff and river discharge diagnostics (Fernandes et al. 2008).
For January, the spatial correlations of the reanalyses are generally closer to CMAP than in the annual mean (Fig. 4), and correlations are lower in July. However, January South America correlations are still very low in all reanalyses. This would seem to indicate that the reanalyses have systematic deficiencies in determining precipitation across South America. However, gauge observation coverage of GPCP and CMAP in South America and Africa is generally much less than other continental regions. In general, the Taylor diagrams for land areas show that all the reanalyses are clustered together, and it is not clear which may be closer to observations. However, in July, ERA-40 and JRA-25 precipitation correlations are generally more closely related to GPCP and CMAP in Europe and Eurasia (Fig. 5). In North America, ERA-40 standard deviation and correlation to GPCP are clustered together and much closer to the observations than any of the other reanalyses. Roads and Betts (2000) noted that ERA-40 precipitation was much closer to observations than NR1. This may be a result of the ERA-40 use of surface stations to nudge soil moisture, but is surprising given that at higher frequencies, the nudging causes problems in the water balance and precipitation has a substantial spindown effect in the Mississippi River basin (Betts et al. 1999). The ability of reanalysis to represent convective precipitation is likely apparent in these results.
The spatial correlations over North America and Europe for JRA-25 and ERA-40 show superior skill in precipitation compared to the others, with most of this higher skill realized over Europe (Figs. 8a,b). The mean biases are different, however. In North America, most of the reanalyses overestimate both observed datasets, except ERA-40 (Fig. 9). Over Europe, the reanalyses precipitation data are consistently lower than GPCP observations by about 0.4 mm day−1 (Fig. 9b). In South America, all of the reanalyses have low spatial correlations, but they seem to be increasing slightly with time. For Africa, there is a slight decreasing trend of the matched correlation of the observational datasets (Fig. 8d). JRA-25 has a sharp decrease of the correlation in 1998. The GEOS4 precipitation shows a sharp increase in correlation over Africa in 1987, when SSM/I becomes available. GEOS4 assimilates retrieved SSMI total column water over the ocean, so that any positive effect is only indirectly related to the changing observing system. ERA-40 has generally the most consistent and highest time series of spatial correlation. All of the reanalyses have significant biases in South America, though not necessarily of the same sign (Fig. 9c).
In North America, JRA-25 has a distinct annual cycle of spatial correlation, where it is high in winter and spring but drops in summer (Fig. 8e). The annual cycles of other reanalyses are similar to JRA-25, with the exception that the correlations are generally lower than JRA-25. On the other hand, ERA-40 has smaller amplitude of the annual cycle, and the summer correlations are higher than JRA-25. Mean biases are generally larger in the summer season in many of the regions. However, ERA-40 has a smaller amplitude of the mean bias than the other reanalyses (Fig. 9e). The South American biases are generally of large magnitude. There are also regional differences between the observational datasets. The North American difference of precipitation is continuous, around 0.2 mm day−1. In Europe, the differences are large in winter (0.8 mm day−1) and smaller in summer (0.2 mm day−1) (Fig. 9f). Also noticeable is the winter difference between GPCP and CMAP, owing to GPCP’s precipitation undercatch adjustment (Adler et al. 2003), but that the reanalyses mean winter precipitation is generally more comparable to CMAP in Europe than GPCP.
d. Oceanic regions
In the previous comparisons, the global statistics are largely related to the tropics, so that separating various regions provides more detail in the evaluation of the reanalyses. In the annual mean, the JRA-25 precipitation in the west tropical Pacific ocean region is better correlated to the observations, though there is still much room for improvement (Fig. 3). In the east tropical Pacific, JRA-25 is quite close to the cluster of CMAP points with slightly higher correlations than ERA-40. In the North Atlantic Ocean area, ERA-40 is closer to the observations. The January North Atlantic precipitation statistics show JRA-25 and ERA-40 essentially indistinguishable from the CMAP data points (Fig. 4). However, in July (Fig. 5) the reanalyses data points for the North Atlantic are quite scattered, with the exception of ERA-40.
The oceanic precipitation in JRA-25 shows very good correlations in the tropical Pacific Ocean and Indian monsoon regions, with the noticeable increase after SSM/I is included (Figs. 10c,d,e). Likewise, ERA-40 seems to have good correlation for many of the oceanic basins evaluated. All the reanalyses have too much precipitation in these tropical regions (the time series is not shown, but Fig. 2 is a reasonable indicator). ERA-40 has better correlations over the North Atlantic Ocean, with small or low biases. The GEOS4 analysis performs fairly well in the tropical oceans, but over the northern oceans, there appears to be distinct deficiencies. The Northern Pacific and Atlantic Ocean regions extend to 70°N. At higher latitudes, precipitation estimation from satellite data is less reliable (Adler et al. 2001).
4. Application to GEOS5 development
The Global Modeling and Assimilation Office (GMAO) has been developing its next global data assimilation system to support NASA projects with operational data products and eventually a retrospective analysis of the satellite era (1979–present). This system is called the Goddard Earth Observing System, version 5 (GEOS5; Rienecker et al. 2007). While a full reanalysis will take time to process, the system has been evaluated against observations and existing reanalyses for a few case studies. Here, precipitation from the GEOS5 data assimilation results for the months of January and July 2004 is compared with the existing reanalyses relative to GPCP and CMAP to assess the character of the monthly precipitation in a developmental system. While the results will vary in different years, this is a preliminary evaluation of the system prior to running a reanalysis, and these statistics will be evaluated during the processing of the reanalysis.
Rienecker et al. (2007) thoroughly describe the GEOS5 numerical model and data assimilation system. In addition to the conventional observations (radiosonde, station, aircraft, ship), SSM/I radiances and retrieved winds, Television and Infrared Observation Satellite (TIROS) Operational Vertical Sounder (TOVS) radiances, Atmospheric Infrared Sounder (AIRS) radiances, and scatterometer wind retrievals are also assimilated. The experiments were initialized from spunup conditions from a climate model. The first 15 days of analysis (16–31 December 2003) are omitted for further spinup, and the periods 1–31 January and 1–31 July 2004 are time averaged to generate the monthly means. The experiments were performed at the spatial resolution of ½° latitude by ⅔° longitude. The analysis was performed by the NCEP Gridpoint Statistical Interpolation (GSI; Wu et al. 2002). The model is then updated incrementally with the analysis through an incremental analysis update (IAU; Bloom et al. 1996). The shock of the analysis at the forecast initialization is greatly reduced, so that the spindown of precipitation is a small factor in this system. In ERA-40, much of the water vapor assimilated into the model was converted into precipitation within the first few forecast hours, contributing to some of the more serious problems with ERA-40 precipitation (Andersson et al. 2005).
Figure 11 shows the January and July 2004 monthly mean precipitation for the two GEOS5 experiments, their differences from GPCP, and the differences of JRA-25, the most recent reanalysis dataset. While GEOS5 is generally biased toward high precipitation in the tropics, the bias is less than JRA-25 for these experiments. The standard deviations of the difference fields for GEOS5 (Figs. 11c,d) are somewhat lower than those for JRA-25 (Figs. 11e,f).
While differences between the reanalyses and GPCP are apparent in the map figure, the Taylor diagrams provide further quantitative assessment of the spatial variability of the precipitation compared to GPCP. Figures 12 and 13 show the Taylor diagrams for January and July 2004 monthly mean precipitation for JRA-25, NR1, NR2, and CMAP, as well as the GEOS5 experiments (ERA-40 data are not available for 2004). When compared with the previous reanalyses, GEOS5 global precipitation is a step closer to both GPCP and CMAP. At the global scale, this is driven by the tropical precipitation. For the Northern Hemisphere land areas, the different reanalyses precipitation all has reasonable comparisons to the observational datasets in January, similar to Fig. 4. In North America, the amplitude of variance in the GEOS5 January experiment is higher than the other analysis (Fig. 12), owing in part to the large amount of coastal precipitation in the Pacific Northwest region. This precipitation is related to the orography, but it is not likely that GPCP represents this well either in the input station observations or spatial resolution. TRMM ¼° precipitation shows more structure along the coast, but the GEOS5 precipitation is still higher than that (figure not shown).
Most of the reanalyses precipitation degrades in the summer of northern continents. While JRA-25 and GEOS5 are closer to the observations than the NCEP reanalyses, there is still room to make improvements (Fig. 13). This is especially true for continental Africa and South America, where the spatial correlations of the reanalyses are not well represented in either the existing reanalyses or GEOS5. In a data assimilation system, tropical land precipitation is difficult to reproduce, as there are few conventional observations, retrievals over land can be complicated by cloudy conditions, and the land–atmosphere interactions are difficult to parameterize. This is an important challenge in the development of future reanalysis systems. Over continental India and the Indian Ocean, GEOS5 shows some improvement in the other reanalyses. In the northern Pacific and Atlantic Oceans, GEOS5 relates as well to GPCP as CMAP does in both correlation and standard deviation in January (Fig. 12). While the existing reanalyses produce reasonable correlations there, the amplitude of variations are typically too large compared to the observations. In July, GEOS5 underestimated the northward extent of the intertropical convergence zone and also the precipitation in the northeastern Pacific Ocean (Fig. 11), leading to an underestimate of the standard deviation there (Fig. 13).
In a general sense, Figs. 11 –13 suggest that the GEOS5 system produces monthly precipitation of either comparable or better quality compared to existing reanalyses. While a few other months have been tested and support the 2004 results presented here, there is interannual variability in the regional biases, correlations, and standard deviations. These diagnostics will be monitored as GEOS5 is used to produce a retrospective analysis for the satellite era.
5. Summary and conclusions
In this study, we investigate a metric to evaluate global and regional precipitation in reanalysis data products. Spatial correlations provide an estimate of the agreement in spatial patterns, while standard deviations represent the amplitude of variation. By comparing two observed datasets at matched times, we can identify an estimate of uncertainty in those data. When correlating the observed data at unmatched times, we can estimate a minimum value of correlation that the reanalyses need to attain when compared with an observation dataset. Because precipitation has certain patterns that recur annually, the unmatched correlation mean represents the average minimum correlation that exists in the real climate system. We have used this to interpret where and when existing reanalyses excel or fail.
While other methods, such as anomaly correlation and EOF analysis, can provide similar results, they require existing long periods of data. The current analysis requires only monthly means, and thus could be used in developing a new system. Gruber et al. (2000) used a filter to remove fine spatial structures from CMAP and GPCP, and found that the ocean anomaly correlations were substantially increased. As reanalyses and observed precipitation datasets move to finer spatial scales, this approach must be considered carefully. Some of the fine structure may be important in the evaluation of the finescale reanalyses (e.g., topography). Higher-resolution precipitation datasets (e.g., Huffman et al. 2007; Ruane and Roads 2007) will help to evaluate reanalyses as finer resolutions are implemented. For example, Ruane and Roads (2007) show that differences in the frequency of precipitation in two reanalyses are related to the parameterization of convection processes. Widmann and Bretherton (2000) find that the NR1 can reproduce the large-scale features of Pacific Northwest precipitation, but resolution limits the finer scales. Given that precipitation is a nonlinear field, there are limits to the discretization of length and time scales used in this method of evaluation. Here, we have simply reduced the resolution of the reanalyses to match the GPCP monthly data. Curtis et al. (2001) compared NR1 tropical anomalies to GPCP and found differences in the phase and strength of the reanalysis representation of ENSO, so that there are temporal modes that are also not captured well with the evaluation presented here.
While there is a general sense that the reanalyses precipitation data are improving with recent systems, it is not always the case. In some ocean regions, NR1 spatial patterns are closer to GPCP than NR2. ERA-40 produces reasonable comparisons over the Northern Hemisphere continents, but less so in the tropical oceans. On the other hand, the most recent reanalysis, JRA-25, shows good comparisons in both the Northern Hemisphere continents and the tropical oceans, but contains distinct variation according to the available observing systems where the SSM/I observations strongly affect precipitation and other hydrological data (Onogi et al. 2007). Hou et al. (2001) show improvement in precipitation from an analysis system by assimilating an estimate of the observed precipitation. However, improvements to a reanalysis dataset may be limited to the availability (both space and time) and quality of the assimilated data.
We have applied these diagnostics to a new data assimilation system, GEOS5, in preparation for a new satellite-era reanalysis. This results in short (one month) validation experiments showing that GEOS5 monthly precipitation is either near the existing reanalyses or closer to the observations. Given the nature of these statistics, they can be monitored during production of the reanalysis as compared with the existing reanalyses or operational systems that may not have a climatological history.
GPCP data were provided by the Laboratory for Atmospheres, NASA Goddard Space Flight Center (http://precip.gsfc.nasa.gov/); CMAP data provided by the NOAA/NWS/CPC (http://www.cpc.ncep.noaa.gov/products/global_precip/html/wpage.cmap.html); NCEP–NCAR reanalysis data and NCEP–DOE reanalysis 2 data provided by the NOAA/OAR/ESRL PSD, Boulder, CO (http://www.cdc.noaa.gov/); ERA-40 data provided by the European Centre for Medium-Range Weather Forecasts (http://www.ecmwf.int/); and JRA-25 data provided by JMA (http://www.jreap.org/). Discussions with Siegfried Schubert were valuable in developing the evaluation of reanalyses. Derek van Pelt provided the scripts and time in developing the Taylor diagrams for these data. George Huffman, John Janowiak, and Phil Arkin provided many useful discussions on the precipitation datasets. The GEOS5 system development is the result of the tremendous efforts of many in the Global Modeling and Assimilation Office.
In this study, GPCP and CMAP datasets are used to estimate the quality of precipitation output from reanalyses considering some uncertainty of the validation data. The matched correlations between GPCP and CMAP, in which monthly means are correlated for the same months throughout the 1979–2003 time series, should be high but not equal to one owing to the different merging techniques and variations in sources of data.
On the other hand, precipitation at monthly time scales does have a regular distribution. For example, the location of the ITCZ or arid regions of North America and Africa occur regularly or persistently in the same location related to the annual cycle of the general circulation; thus very small or negative correlations between reanalyses and observation would be difficult, if not impossible, given that the reanalyses can likewise reproduce some of these large-scale patterns. For long time series, anomaly correlations can address this issue, but for short experiments in the development stage of a data assimilation system, long time series are not available and sometimes not practical. Likewise, some observation datasets may not exist for a long enough period of time. Below we define a minimum of spatial correlation to determine the quality of reanalysis values that can be applied to long and short time series of reanalysis data.
We have correlated unmatched monthly means of the GPCP and CMAP datasets. For example, January 1979 of GPCP is correlated to every January during 1980–2003 of CMAP. The accumulation of the unmatched correlations follows:
For averaging, the N number of correlations is determined by summing δi,j, and the seasonal cycle of the unmatched correlation (summing only January, February, . . .) can also be determined. Here, Pg and Pc represent GPCP and CMAP precipitation (monthly maps), respectively. These values represent the mean of spatial correlations of different years. The values are generally positive, owing to the fact that precipitation patterns have some regularly occurring features. It is the average correlation one would expect from comparing random GPCP and CMAP months and represents the degree of natural correlation that exists in the precipitation field. If the reanalyses cannot have a spatial correlation greater than this mean value, the skill of the reanalysis to produce reality may be in question. A high value would indicate that the pattern of precipitation occurs regularly; for example, there are strong latitudinal and geographic constraints in the defined Africa region. A low value of unmatched correlation indicates that the precipitation patterns are quite different from year to year (as in the Europe region, which has more longitudinal distance) and subject to transient synoptic weather systems.
Figure A1 shows the mean of all the unmatched spatial correlations for the regions discussed earlier and also some latitudinal bands. For the global spatial correlation, 0.69 is the mean, so that in most years, the reanalyses annual means are above this value (Fig. 3). While this is also true for the ocean, the land unmatched correlation of 0.80 is in the middle of the reanalyses annual means. So in a general sense, the reanalyses need to improve the spatial distribution of precipitation over land. It is difficult to compare the different regions, as the geography of the region plays a role in the result, but each can be compared with other data for the same region. None of the reanalyses precipitation exceeds the unmatched correlation in South America and Africa (though there are a few years for which ERA-40 is close in Africa). While GPCP and CMAP are closely related for Africa and South America (as in Fig. 3), the number of gauge observations are limited relative to the Northern Hemisphere continental regions, so that both GPCP and CMAP may have more uncertainty than is apparent when comparing the two datasets (G. Huffman 2007, personal communication). This broad conclusion is presented in the context of the current study. However, Rao et al. (2002), in a regional analysis, compare NR1 precipitation to many Brazilian gauge stations and find regions of reasonable comparison, as well as significant deficiencies.
Europe has a low unmatched correlation (0.42) compared to other regions. In the mean annual cycle, the lowest values are February–May (not shown). All of the reanalyses have annual values that exceed this correlation, likely related to the large-scale storm track in the region. In North America, the lowest unmatched correlations (as well as matched correlations) occur during July–September, during the end of the warm season, when land–atmosphere interactions and the soil water availability contribute to the precipitation variability (Bosilovich and Schubert 2001; Bosilovich and Chern 2006).
For the Indian monsoon region, the JRA-25 is less than the unmatched correlation mean before SSM/I becomes available. However, it is well above that when SSM/I is being assimilated. Most of the reanalysis precipitation is above the unmatched correlations in the Indian monsoon region. Last, the matched and unmatched correlations for Southern Hemisphere midlatitudes are lower than other latitude bands (in July, matched correlation is just lower than unmatched), which may be related to both a high degree of variability as well as observational uncertainty.
Corresponding author address: Michael G. Bosilovich, Global Modeling and Assimilation Office, Code 610.1, NASA Goddard Space Flight Center, Greenbelt, MD 20771. Email: email@example.com