Researchers began paving the way for efficient use of dedicated supercomputing facilities to enable higher resolution climate modeling with potentially large improvements in fidelity.
Based on a great many peer-reviewed articles as well as multiple assessments by the Intergovernmental Panel on Climate Change (IPCC) Fourth Assessment Report (AR4; Alley et al. 2007), it is now considered highly likely that human activity is contributing substantially to global climate change. Efforts to mitigate the impacts of and successfully adapt to a changing climate will require the investment of trillions of dollars worldwide over the next several decades. In order for these investments to be made efficiently and effectively, accurate predictions of changes in both the mean climate and the frequency of extreme events will be required at the regional level. However, state-of-the-art climate models cannot accurately predict regional climate variations, due largely to their relatively coarse horizontal resolution, typically O(100 km),1 and the associated weaknesses in parameterizing subgrid-scale features.
A key step in meeting the societal need for accurate prediction of regional climate variability and change is to take models of the climate system to a new level of capability in which salient features and processes of weather and climate are explicitly resolved. Previous work has shown that organized mesoscale motions must be realistically represented in climate models to achieve fidelity in simulations (Orlanski 2008) and that misrepresentation of the salient mesoscale atmospheric and oceanic phenomena in models can be directly linked to their exhibited biases. For example, Bauer and Del Genio (2006) showed that a climate model's underprediction of cloudiness and humidity in the subpolar region results from the lack of moisture transport by extratropical cyclones. Such inadequacies are directly attributable to insufficient model resolution (e.g., Jung et al. 2006; Jung and Rhines 2007).
In recognition of the pressing need for the capability to perform multidecadal climate simulations with resolution beyond the capability of contemporary climate models, the World Climate Research Program (WCRP) sponsored a World Modeling Summit (WMS) for Climate Prediction in 2008, with the goal of developing a strategy to revolutionize the prediction of climate (Shukla et al. 2009). Among the conclusions reached by the WMS was that international dedicated high-end computing (HEC) facilities should be established for the development and application of high-resolution climate models. This was further articulated by Shukla et al. (2010), who suggested that “to accelerate progress in understanding and predicting regional climate change, national climate research facilities must be enhanced and dedicated multinational facilities should be established.” Accurately simulating the past and future climate at regional scales is a grand challenge in the weather, climate, computational, and social sciences that is essential to address the pressing problem of global climate change. International cooperation will be an essential component of any such project, as demands for computational and scientific resources and expertise are likely to exceed the ability or will of any single country to provide them.
PROJECT ATHENA.
In response to the recommendations from the WMS, the U.S. National Science Foundation (NSF) fostered an international collaboration for advancing climate prediction and found a serendipitous way to provide the required large computational resource. In mid-2009, the Athena supercomputer, a Cray XT-4 with 4,512 quad-core nodes (a total of 18,048 computational cores) operated by the University of Tennessee's National Institute for Computational Science (NICS) and hosted by Oak Ridge National Laboratory (ORNL), was scheduled to be decommissioned following its replacement by a much larger machine (Kraken; www.nics.tennessee.edu/computing-resources/kraken). Because Athena was a highly capable machine [ranked 30th on the top 500 (www.top500.org/) list at that time], the NSF agreed to meet the operating and maintenance costs for Athena for an additional year and to provide six months of dedicated use for climate simulation and prediction (1 October 2009–31 March 2010). Project Athena was born.
At the outset of the project, a robust international collaboration was established among five groups spread across three continents. The lead institution [Center for Ocean–Land–Atmosphere Studies (COLA)] approached the European Centre for Medium-Range Weather Forecasts (ECMWF), the University of Tokyo, and the Japanese Agency for Marine-Earth Science and Technology (JAMSTEC). ECMWF develops and operationally runs a highly successful numerical weather prediction (NWP) model [the Integrated Forecast System (IFS); www.ecmwf.int/research/; Jung et al. 2012, and references therein], while JAMSTEC develops and maintains a global high-resolution model with explicit-only convection [Nonhydrostatic Icosahedral Atmospheric Model (NICAM); Satoh et al. 2008]. Close collaboration was also established with personnel at NICS, who provided the supercomputing resources as well as hardware, software, applications, and data support, including participating in weekly teleconferences with the project team during the computationally intensive phase of the project. As suggested by Navarra et al. (2010), the involvement of the computing center staff and even vendor technical experts was critical to the success of the project. The tight schedule and daily challenges of 24/7 production computing engendered a certain esprit de corps among the diverse groups participating in the project, and team members expressed great satisfaction with the opportunity to participate in the project.
Project Athena represents a pilot program, intended to demonstrate the impact of dedicated HEC support on the progress in the area of climate modeling. Guided by the recommendations of the WMS, the project set out to address three main assertions regarding the impact of resolution and process-resolving models on climate simulations. It should be noted that while these assertions were the guiding premises of the experiments that were conducted, an exhaustive test was beyond the scope of even the dedicated resources available to Project Athena.
Assertion 1:
Increasing climate model resolution to more accurately resolve mesoscale phenomena in the atmosphere can improve the fidelity of the models in simulating the mean climate and the distribution of variances and covariances and in predicting the seasonal climate.
Addressing the seamless nature of the problem of predicting weather and climate (Palmer et al. 2008; Shapiro et al. 2010), a state-of-the-art high-resolution NWP model (IFS) was run in climate mode on multiyear time scales to assess the impact of high resolution on systematic error. These integrations will also inform numerical weather prediction centers in guiding their future operational strategies for increasing the resolution of models in intraseasonal and seasonal forecast mode. While properly resolving mesoscale phenomena in the ocean (eddies) is also likely to improve model fidelity, this pilot project was limited to land–atmosphere models only, due to the increased difficulty of porting and achieving sufficient scaling in a coupled atmosphere–ocean system.
Assertion 2:
Simulating the effect of increased greenhouse gases on regional aspects of climate, such as precipitation and storminess, may, for some regions, depend critically on the resolution of the underlying climate model.
Time-slice climate change simulations, with sea surface temperature (SST) anomalies taken from simulations of the latter part of the twenty-first century, were made using IFS both at NWP resolutions (~16 km) as well as those used in typical climate change integrations (~125 km). Analysis of these integrations allows for greater understanding of the impact of resolution on the simulated regional response to greenhouse gases in the statistics of weather, extreme events, and the hydrologic cycle.
Assertion 3:
Explicit representation of important processes in the atmosphere such as cloud motions and transports, without parameterization, can improve the fidelity of the models, especially in describing and predicting the regional structure of weather and climate.
It should be noted that the available computing resources in Project Athena facilitated a 10-fold increase in the horizontal resolution compared to contemporary climate simulations. However, fully exploring the fidelity of climate models by explicitly resolving important atmospheric or oceanic processes is still beyond the computing facilities and model software available today. Comparisons between IFS, a model with parameterized convection, and NICAM, an atmospheric model (run at 7-km resolution in the simulations described here) that explicitly simulates cloud processes and convection, were made to evaluate the impact of resolving cloud processes on the simulation of seasonal climate.
IMPLEMENTATION AND EXPERIMENTAL DESIGN.
During summer 2009, the IFS and NICAM models were transferred to Athena and detailed workf low and data-management plans were devised and implemented. Cray, Inc. provided code conversion and optimization assistance for the NICAM model, which had never before been run on a U.S. computer. New job control scripts, based on operational software used at ECMWF for IFS and developed anew for NICAM, were written to manage the f low of jobs and data. Initial and boundary conditions were obtained from the 40-yr ECMWF Re-Analysis (ERA-40; for IFS) and the National Centers for Environmental Prediction (NCEP) Global Data Assimilation System (GDAS; used in NICAM experiments). The SST used in the IFS experiments was taken from best available estimates (for details, see Jung et al. 2012). The SST used in the NICAM experiments was the NOAA operational daily 1/4° grid analysis (Reynolds et al. 2007), which is finer resolution than that used with IFS (~1° grid).
Careful organization of job size and priority, along with meticulously designed work and data flows, allowed the relatively small team of project scientists to maintain utilization of the supercomputer above 95% of full capacity for nearly the entire 6-month period. Overall, the project used over 72 million core hours on Athena out of a theoretically possible 78.8 million core hours and completed over 220 simulated years of model integration (see Table 1 for a complete list of experiments).
Project Athena experiments.
Because of the contrasting computational characteristics of the two models, a much larger number of the simulations were done with IFS. In the case of IFS, the hydrostatic dynamics and the semi-implicit, semi-Lagrangian algorithm employed for stepping the equations forward in time permitted very long time steps. NICAM's nonhydrostatic dynamics, including a prognostic equation for the vertical component of velocity, along with the explicit treatment of convection, required a very short time step to avoid numerical instability: 30 seconds at 7-km resolution, in contrast to a time step of 450 seconds at 10-km resolution for IFS. Thus, the same unit of simulated time was over 50 times more costly in computational terms with the NICAM model than with the IFS. The drive to reduce or eliminate the use of simplifying assumptions and parameterizations in order to achieve desired accuracy in representing regional climate can have a costly impact on model performance. Notably, NWP centers worldwide are actively developing new modeling strategies beyond the “large-scale hydrostatic” realm that has served them so well in the past 30 years.
As noted above, the experiments included several types of integrations. The fidelity of both the IFS and NICAM models in representing features of the global climate were evaluated and directly compared in a series of integrations covering selected boreal summers. Features of interest include the pluvial and drought conditions over extratropical continents and the formation, propagation, and demise of tropical cyclones. In addition, the IFS model was integrated over several 13-month periods, each initialized on 1 November for all years from 1960 to 2007. Because each of these integrations includes two Novembers—one within one month of the atmospheric and land surface initial conditions and one removed from the initial state by one year—it is possible to use them to assess the drift of the model climate away from the observed climatology and the reproducibility of the solution for a given specified set of boundary conditions. While much of the climate drift of an atmosphere-only model should occur within several weeks of the initial condition, there is, however, also a drift in the land surface conditions that can extend the period over which the model experiences drift. Several ensembles of integrations using perturbed initial states and the same boundary conditions were also made with the IFS model in order to get a measure of the reproducibility of the model climate in both winter and summer seasons.
The entire 47-yr period was also simulated in a single continuous run [a climate of the twentieth century (C20C) run; also sometimes referred to as an Atmospheric Model Intercomparison Project (AMIP) run; http://www-pcmdi.llnl.gov/projects/amip/NEWS/overview.php] starting in January 1961. Finally, a time-slice run was made to assess the impact on the global atmosphere of the anticipated change in SST associated with global climate change. The difference in the annual cycle of SST at each grid point between the last 30 years of the twenty-first century and the last 30 years of the twentieth century, taken from the IPCC AR4 integration of Community Climate System Model, version 3.0 (CCSM3.0), was added to the observed record of SST for the 1961–2007 period. The resulting SST represents an estimate of the future SST, with the assumptions that 1) models used to project the change in climate to the end of the twenty-first century quite reliably depict the change in the mean annual cycle and 2) the future SST will have the same interannual variability as it has in the current climate.
All the runs with the IFS model were done at multiple resolutions, including a grid spacing representative of current climate model resolution (T1592 or 125 km), one representative of the NWP models that were in operational use in the last few years (T511 or 39 km), one representative of the NWP models being deployed for operations in the near term (T1279 or 16 km; this resolution was made operational at ECMWF in January 2010), and an experimental resolution presumed to be at the upper limit of what can reasonably be used in models with hydrostatic dynamics and parameterized convection (T2047 or 10-km grid spacing). It should be noted that the runs made with the IFS are similar to but at much higher spatial resolution than several earlier experiments [e.g., Iorio et al. (2004), who used up to T239 resolution, or Mizuta et al. (2006), who used a 20-km grid]. More details about the IFS simulations are given in Jung et al. (2012).
RESEARCH HIGHLIGHTS.
Analyses of the simulations produced in Project Athena are ongoing, with several papers already appearing that document various aspects of the simulations (Dirmeyer et al. 2011; Jung et al. 2012; Satoh et al. 2012; Manganello et al. 2012; Dirmeyer et al. 2012). In general, the impact of increasing horizontal resolution (assertion 1), while not a panacea, greatly improves the representation of diverse features of the mean climate and its variability ranging from extratropical blocking (Jung et al. 2012) to tropical cyclones (Manganello et al. 2012). On the other hand, some aspects such as the diurnal cycle of precipitation (Dirmeyer et al. 2011) and boreal summer intraseasonal variability of the tropical circulation (Jung et al. 2012; Satoh et al. 2012) are relatively insensitive to changes in spatial resolution.
The primary change in precipitation with increasing resolution is to reduce the biases where they occur in boreal winter: that is, there is no appreciable shift in the distribution of precipitation but the magnitude of the error is reduced by higher resolution (Jung et al. 2012, their Fig. 5). However, an opposite effect is detected in boreal summer when the effect of increasing resolution exacerbates the error of producing too much rainfall in the vicinity of the intense rainfall regions of south Asia, the Maritime Continent, and adjacent seas.
Most of the improvements in the climate simulations with IFS come from the increase in resolution from T159 to T511 with more modest improvements for further resolution increases to T1279 and T2047. There is some evidence that increasing horizontal resolution to T1279 leads to moderate increases in seasonal forecast skill during boreal winter in the tropics and northern extratropics (Jung et al. 2012) and the representation of blocking (see discussion below). Problems in simulating the Madden–Julian oscillation remain unchanged for all resolutions tested, although there are some features of some intraseasonal events that are captured by the highresolution models, particularly the NICAM with its explicit representation of deep convection (Satoh et al. 2012).
With respect to the effect of resolution on the representation of climate change (assertion 2), these experiments have also confirmed the findings of earlier papers that aspects of regional climate change are quite sensitive to spatial resolution (highlighted below). Finally, in regard to the impact of explicit representation of critical atmospheric processes (assertion 3), some features of the hydrological cycle, particularly diurnal phase of precipitation (Dirmeyer et al. 2011) and convectively driven subseasonal circulation in the tropics (Satoh et al. 2012), have been shown to be strongly influenced by whether cloud processes are parameterized.
Here we highlight five selected results that illustrate some of the advantages and challenges of high-resolution models, particularly as they relate to the guiding assertions of Project Athena.
North Atlantic blocking.
As discussed in the introduction and addressed in assertion 1, the representation of mesoscale features such as extratropical cyclones can be critical to the fidelity of climate simulation. Another feature of atmospheric circulation in the extratropics that is influenced by cyclones can influence the formation and track of cyclones and has a bearing on climate simulation is the frequency of winter blocking in the North Pacific and the North Atlantic sectors (Tibaldi and Molteni 1990). The latter is especially important for European climate. The models used in the AR4 do not reproduce the observed frequency of blocking with simulated events generally shorter and rarer than observed events (see Solomon et al. 2007, section 8.4.5). As examined in detail by Jung et al. (2012) and shown in Fig. 1, the representation of blocking in the boreal winter is inadequate for models with resolutions coarser than 39 km, achieves a high degree of realism at a resolution of 39 km, and is very similar for the tested resolutions finer than 39 km. This is particularly apparent for the Atlantic sector (20°W–50°E), where the observed maximum in the blocking index is well reproduced by the 16-km simulation (T1279 curve) but not by the 125-km simulation (T159 curve). It should be noted, however, that although Fig. 1 suggests convergence of the solution at T511, in fact, there are some indications that, for longer-lived Euro–Atlantic blocks (>5 days), the T1279 simulations have a substantial advantage over T511 and T159 (Dawson et al. 2012). The implication that there is a minimum resolution coarser than which blocking is insufficiently well simulated is highly suggestive for the threshold resolutions that should be considered for climate simulation.
Frequency of occurrence (%) of days at which the winter (December–March) Northern Hemisphere extratropical flow is blocked: reanalysis (black with 95% confidence level shaded in gray), 125-km simulation (blue curve labeled T159), and 16-km simulation (red curve labeled T1279) for the AMIP experiment over the period 1960–2007. The dots indicate significant differences of modelsimulated values from reanalysis.
Citation: Bulletin of the American Meteorological Society 94, 2; 10.1175/BAMS-D-11-00043.1
Tropical cyclones.
The simulation of tropical cyclone (TC) intensity has lagged for years behind the simulation of genesis location and tracks, both in prediction models and climate models (for a review, see Hamilton 2008). In addition to such factors as inadequate model physics and insufficient understanding of internal processes (e.g., Wang and Wu 2004), the coarseness of model resolution has strongly contributed to this deficiency. For example, as shown by Zhao et al. (2009), at 50-km resolution, a general circulation model (GCM) can simulate genesis location, tracks, and frequency well, but it still significantly underestimates storm intensity.
As noted by Hamilton (2008), the intensity of simulated tropical cyclones may have a bearing on the climatology of both the tropics and extratropics. The Athena project has presented a unique opportunity to assess the impact of a dramatic resolution increase on the simulation of TC intensity distribution, as discussed in more detail in Manganello et al. (2012) and Satoh et al. (2012). Figure 2 shows a snapshot from one of the NICAM boreal summer simulations, which is characteristic of many of the simulations. There is a strong tropical cyclone in the Bay of Bengal with stunningly realistic features, including a well-formed eye and spiral bands whose cloud distribution and rainfall intensity are quite similar to those found in nature. Animations of the NICAM simulations (http://wxmaps.org/athena/home/mov/NICAM_p09.mov) are difficult to discriminate from animations of satellite observations of tropical systems. A more in-depth evaluation of the vertical structure of TCs, which is the important aspect of these storms that evolves the climate forward, is provided in Manganello et al. (2012).
A snapshot at 0500 UTC 23 May 2009 from the NICAM model simulation of 21 May–31 Aug 2009, with cloudiness (based on the simulated outgoing longwave radiation) in shades of gray and precipitation rate in colors. The cloudiness is shaded in brighter gray for thicker clouds, and the colors range from shades of green (indicating precipitation rates less than 1 mm day−1) to yellow and orange (1–16 mm day−1), to red (16–64 mm day−1), and to magenta (>64 mm day−1).
Citation: Bulletin of the American Meteorological Society 94, 2; 10.1175/BAMS-D-11-00043.1
Model spatial resolution also strongly affects the statistics of North Atlantic hurricanes. Figure 3 shows the frequency of occurrence of maximum 10-m wind speed and minimum sea level pressure (SLP) in the North Atlantic hurricanes simulated by IFS at two of the resolutions: 39 and 10 km. The qualitative difference between the two model intensity distributions is clear: the peaks of the distributions are too high and are skewed toward lower wind speeds (or higher pressures) in the lower-resolution simulation. The tails of the distributions that include secondary peaks are not reproduced at all at 39 km compared to 10 km. The most intense storms, achieved only in the 10-km simulations, correspond to category-4 or category-5 hurricanes in maximum wind speed and minimum central pressure, respectively. Figure 4 shows a similar distribution of minimum attained SLP for the entire globe, restricted to June–August of the years for which both NICAM and IFS 10-km simulations are available. The IFS simulations tend to overpredict storms in the lower intensity bins (968 hPa and higher) and underpredict the higher intensity storms, compared to the observed. In contrast, the NICAM simulations include a smaller proportion of less intense storms than observed, with a higher proportion of very intense storms (below 968 hPa).
Distribution of (top) maximum attained 10-m wind speed and (bottom) minimum SLP in the North Atlantic hurricanes from the International Best Track Archive for Climate Stewardship (IBTrACS) data (black bars), IFS 10-km simulation (red bars), and IFS 39-km simulation (green bars) for the May–November seasons of 1990–2008. The inset plots show the tails of the distributions.
Citation: Bulletin of the American Meteorological Society 94, 2; 10.1175/BAMS-D-11-00043.1
Distribution of minimum attained SLP over the entire globe (all basins) from the IBTrACS data (black bars), IFS 10-km simulation (red bars), and NICAM simulation (blue bars) for June–August of 2001–02 and 2004–09. The inset plots show the tails of the distributions.
Citation: Bulletin of the American Meteorological Society 94, 2; 10.1175/BAMS-D-11-00043.1
The faithful representation of surface characteristics of TCs is of prime importance in order to obtain a realistic assessment of their damage potential. Figure 5 shows the quality of simulation of such features in NICAM and IFS at two different resolutions, using as an example a snapshot of the most intense TC at the peak of its intensity. The NICAM simulated TC (Figs. 5a,d) shows a storm of relatively small size with a clearly delineated and large eye, a relatively large radius of maximum winds (0.35°), and relatively few rainbands. Comparing the two different resolutions of IFS simulations, the 10-km simulated TC (Figs. 5b,e) shows smaller scale and greater detail in the inner core, as well as more intense gradients in the eyewall than the IFS 39-km resolution (Figs. 5c,f), and agrees better with observations of the most intense TCs (e.g., Kimball and Mulekar 2004). The 10-km simulated TC has also a clearly visible eye surrounded by a tight eyewall, though not as well defined as in the NICAM simulated TC, and multiple rainbands. In contrast, the eye and eyewall are blurred in the 39-km simulated TC, and the rainbands are not discernible (Fig. 5d). The eye in the NICAM simulation is also more realistic in terms of total column liquid water and ice.
Distributions of (left) 10-m tangential wind (m s−1) and (right) total column liquid water and ice (TCLWI; kg m−2) for the most intense TCs at the peak of their intensity from (a),(d) the NICAM simulation; (b),(e) the IFS 10-km simulation (T2047); and (c),(f) the IFS 39-km simulation (T159). Radius is 2°. Contour interval is 3 m s−1 for wind. Dashed black contours in (d)–(f) show the radius of maximum winds for each case with respect to the center of the storm determined from the location of maximum vorticity at 925 hPa (1,000 hPa for the IFS cases).
Citation: Bulletin of the American Meteorological Society 94, 2; 10.1175/BAMS-D-11-00043.1
Climate change.
A comparison of the C20C and timeslice runs of IFS at 125- and 16-km resolutions reveals many things about the sensitivity of climate change simulation to model resolution. Keying on Fig. 12.1 in the IPCC AR4 Working Group 2 report (Parry et al. 2007), which shows the change in relative runoff for early and late twenty-first century Europe, based on two different models, we examined the change in growing season precipitation between the recent (late twentieth century simulation) and future (late twenty-first century simulation) climate runs. As shown in Fig. 6, the change is qualitatively similar but quantitatively very different between the lowresolution and high-resolution runs.
Change in April through October mean precipitation from simulations of the 1961–2007 period to the 2071–2117 period over western Eurasia, as a percentage of the 1961–2007 mean precipitation. Observed SST and sea ice were specified for the 1961–2007 simulations. An estimate of the late twenty-first century SST and sea ice, computed by adding the change in mean annual cycle averaged over multiple simulations from phase 3 of the Coupled Model Intercomparison Project (CMIP3) to that specified for the 1961–2007 simulation, was used in the 2071–2117 simulations. Shown are (top) 125-km-resolution simulations and (bottom) 16-kmresolution simulations.
Citation: Bulletin of the American Meteorological Society 94, 2; 10.1175/BAMS-D-11-00043.1
In both simulations, there is a northeast–southwest gradient of rainfall change, with increasing rainfall in the northeast over Scandinavia and western Russia and decreasing rainfall in the southwest over Iberia and the Mediterranean countries. This is very similar to the late twenty-first century results shown in Parry et al. (2007). However, as in the AR4 Working Group 2 (WG2) figure, the magnitude of the gradient is very different in the two simulations. The low-resolution simulation has relatively small changes in rainfall regime throughout Europe with only western Iberia suffering major reductions. The high-resolution simulation, in contrast, has large decreases (>20%) over all of Iberia and parts of southern France, Italy, Greece, and Turkey. Smaller but still severe reductions of 10%–20% are found over the rest of France and Italy and the nations west and north of the Black Sea. In general, the high-resolution simulation has more a spatially coherent and physically plausible pattern of change in precipitation regime. This pattern is related to the change in circulation regime over western and southern Europe (not shown).
Despite the overall similarities between the two patterns, while the changes found in the low-resolution simulation are relatively benign, with only western Iberia potentially threatened with severe water short-ages, the changes in the high-resolution simulation are dire for most of southern Europe. As suggested elsewhere in this article and in Jung et al. (2012), the fidelity of the climate simulation over Europe improves with resolution, lending credence to the high-resolution result. In order to respond appropriately to these potential changes in water availability, it is essential for stakeholders and decision makers in Europe to know which of these simulations is more likely to occur, and this sensitivity of critical details of the pattern of regional climate change to resolution provides strong support for our second guiding assertion.
Diurnal cycle of precipitation.
It is frequently noted that climate models do a relatively poor job of simulating precipitation. This includes the annual mean, mean annual cycle, and variability at all time scales. Climate models, even the latest generation, notably misrepresent the diurnal cycle of rainfall, starting too early in the day and occurring too frequently at reduced intensity (Dai 2006). In many places, particularly in summer, rainfall is observed to peak in mid to late afternoon in association with local destabilization of the atmospheric column or later in the evening or night because of propagating features such as mesoscale convective systems (e.g., Nesbitt and Zipser 2003). Dirmeyer et al. (2011) have evaluated the diurnal behavior of precipitation in the IFS and NICAM simulations, in comparison with various estimates from observations, particularly the Tropical Rainfall Measuring Mission (TRMM). As shown in Fig. 7, the diurnal cycle of precipitation, which is in many places a function or orography and orogenic features, is particularly challenging in South and East Asia with its complex terrain and unique geographic features. As described in Dirmeyer et al. (2011), the amplitude of the diurnal cycle of precipitation is larger in NICAM than in TRMM almost everywhere. On the other hand, it is apparent that in some of the simulations, notably in NICAM in the India–Tibet transect (Fig. 7, line A–B and bottom-left panel), the diurnal phase is quite well reproduced (except possibly for a portion of the Gangetic plain). In contrast, the 125-km IFS simulations have peak precipitation phase within ±3 h of local noon everywhere. The 10-km IFS simulations lie in between the two. On the Tibet to Taiwan Strait transect (Fig. 7, line C–D and bottom-right panel), the NICAM simulation performs well over Tibet and the coastal plain but fails to capture the correct relationship in the highlands between the two. Both the 125- and 10-km IFS simulations are sun locked nearly everywhere along this line except over Sichuan, where the 10-km simulation has an overnight rainfall maximum. We speculate that the above results are indicative of the difficulty in designing convection parameterizations for horizontal resolutions of 10 km and beyond.
Phase of the diurnal cycle of precipitation in observations and models. (top) The June–August mean hour of maximum rainfall estimated from TRMM data over 1998–2009. The colors correspond to local time on the 24-hour clock shown as an inset at the top left. (bottom) The hour of maximum rainfall for transects along lines (left) A–B and (right) C–D. Two transects are shown (A–B) and (C–D) for TRMM (green circles), NICAM (red squares), IFS 125-km simulation (blue dashes), and IFS 10-km simulation (cyan triangles). Data from all sources were first interpolated to the NICAM grid for ease of comparison. Values over ocean points where the amplitude of the diurnal cycle is less than half the seasonal mean (June–August) rainfall, and values over land points where the rainfall rate is less than 0.2 mm day−1 are not shown.
Citation: Bulletin of the American Meteorological Society 94, 2; 10.1175/BAMS-D-11-00043.1
As mentioned by Dirmeyer et al. (2011) in a broader comparison of models also including the CCSM3.0 (Collins et al. 2006) and a superparameterized version of CCSM (SP-CCSM; Stan et a l. 2010), parameterizations that are dependent on atmospheric stability for triggering convection, which is typically locked to local noon in these models, fail to properly simulate the diurnal cycle of precipitation.
Intraseasonal oscillation.
One long-standing issue of global atmospheric circulation models is the ability to realistically simulate intraseasonal oscillations (ISO) whose periodicity lies between a few weeks and one season. It is well recognized that most general circulation models have had difficulty in reproducing the Madden–Julian oscillation (MJO), which is one manifestation of ISO (Slingo et al. 1996; Lin et al. 2006). Significant effort has been made to improve the behavior of ISO/MJO in general circulation models, particularly through the improvement of cumulus parameterization schemes (cf. Bechtold et al. 2008; Chikira and Sugiyama 2010). NWP modeling centers are interested in achieving a better representation of ISO/MJO since these oscillations are directly related to the skill scores of forecasts with lead times of more than a week (Gottschalck et al. 2010; Rashid et al. 2011). It was shown previously that NICAM successfully simulated realistic behavior of an ISO/MJO event (Miura et al. 2007; Oouchi et al. 2009; Taniguchi et al. 2010).
The Athena experiments offered an opportunity to evaluate the systematic behavior of ISO and how it differs between the models and among the different resolutions. The results are mixed. On the one hand, as shown in Jung et al. (2012), the IFS does a relatively unimpressive job of simulating the MJO (see their Fig. 10). On the other hand, Satoh et al. (2012) show that several features of the intraseasonal variability in the tropics are well represented in certain years in both the IFS and NICAM simulations (comparisons available for summers only). The NICAM cases include instances with good representations of the northward propagation of cloud clusters and the 45-day peak in the power spectrum. Figure 8 exemplifies a boreal summer ISO in the Indian Ocean in 2006 in the NICAM 7-km and IFS 10-km simulations. In this case, both models reproduce the two events of the northward propagation of convective systems realistically. The IFS simulation displays a more standing behavior compared to the NICAM and the TRMM data. Since this case includes multiple ISO events, further evaluation of the mechanisms governing ISO, including its onset, such as the roles of convective systems, is worth investigating, as suggested by Satoh et al. (2012).
Time–latitude sections of daily precipitation anomalies averaged over 60°–90°E for (middle) NICAM 7-km and (bottom) IFS 10-km 103-day-long hindcasts for the period 22 May–30 Aug 2006 and (top) for the corresponding observations from TRMM. The annual cycle for the period 2001–09 (2003 omitted) is removed from the daily mean to obtain the daily anomaly.
Citation: Bulletin of the American Meteorological Society 94, 2; 10.1175/BAMS-D-11-00043.1
IMPLICATIONS FOR THE FUTURE.
Resolution and complexity.
There is evidence that current climate models are far from a spatial resolution at which the necessary fidelity to the observed climate is achieved. Controlled experiments conducted at different resolutions have demonstrated qualitative and quantitative sensitivity to model resolution. For the vast majority of fields considered, the impact of increasing horizontal resolution was beneficial. However, the improvement was neither universal nor uniform. The representation of North Atlantic blocking in IFS improved substantially between 125- and 39-km resolutions but showed little change at higher resolutions (Jung et al. 2012). A similar tempered improvement was seen in the simulation of the global hydrologic cycle for IFS (Dirmeyer et al. 2011). In contrast, tropical cyclone structure and intensity in the model improve dramatically as resolution becomes finer than 39 km (Manganello et al. 2012), while the simulation of the mean features of the Indian monsoon shows little qualitative change among all tested resolutions (Achuthavarier et al. 2010). Significant improvements are seen in key aspects of rainfall simulation when clouds and convection are represented explicitly (Dirmeyer et al. 2011); however, the comparisons between IFS and NICAM done here are not wholly satisfying, because proper controlled experiments to quantitatively assess the importance of explicitly resolving deep convection (e.g. running IFS without parameterized convection or running NICAM at different resolutions) were beyond the scope of computational resources available even for Project Athena. The expense is so large that developing new parameterizations suitable for high-resolution models that do not fully resolve processes like deep convection will likely be necessary for the foreseeable future.
The increasing complexity and expense of climate models make it plain that significant resources must be made available not just for research and operations groups but for development as well. In the case of NICAM, hundreds of thousands if not millions of core hours can be required merely to confirm the presence of a bias in a particular field and test changes that can ameliorate its negative effects. This cost represents a serious obstacle to the traditional development methodology of iterative tuning of individual parameters with full model simulations used to evaluate the effects of parameter changes. If we assume that cloud-resolving models will require 1-km or finer grid spacing, then the cost is orders of magnitude higher and the computers capable of such simulations are unlikely to be available for many years. Multicriteria multiparameter calibration methods like those used in other forms of environmental modeling could improve the efficiency of the model development process as we move to high-resolution climate modeling. Also, the transpose-AMIP method (e.g., Phillips et al. 2004; Boyle et al. 2008), in which short model integrations, typically of the same length as forecasts from NWP models, are used to evaluate new climate model formulations, may be applicable for the development and evaluation of such highresolution models, at a considerable cost savings relative to AMIP or other long-term simulation protocols. Finally, it should be noted that model development can be a more demanding scheduling problem than a production-type activity like Project Athena, because the dependence of future experiments on the outcomes of previous runs makes it necessarily a serial endeavor. Moreover, as NWP centers strive toward resolutions with which cloud processes and convection may be explicitly represented, a major research effort is required to achieve such simulations at comparable efficiency when compared to existing hydrostatic simulations.
Observations.
Validation of model output, particularly such highly variable quantities as precipitation, at hourly intervals on sub-20-km global grids represents a significant challenge for the current observational network. Satellite products offer high spatial and temporal resolution, but only for limited swaths of the globe at a given moment. Rain gauge networks are irregularly distributed and many do not offer high temporal resolution. Neither observing system component provides adequate coverage over high or complex topography, where rain rates are frequently high and gradients are sharp. Simulated rainfall is approaching temporal and spatial scales where it is difficult to distinguish between model bias and observational error. The results of Project Athena strongly suggest that a more robust climate observing system is needed. The need becomes ever more urgent for a multifaceted observing network, combining the coverage and high resolution of satellite observing platforms with the accuracy and stability of long-term in situ observing stations. Any lapses in the ongoing coverage of the observational network will be increasingly detrimental to our ability to calibrate and perform forecasts.
Dedicated computing and HEC center design.
Having access to the entire computer as a dedicated resource represented a fantastic opportunity for the scientists involved in the project; the positive impact is difficult to overstate. In the case of Project Athena, it allowed for highly efficient use of the computational resource, as all job sizes and scheduling were under project control. Crucial aspects of the system operation and queues were under the control of the project, which enabled optimization of throughput and accommodated special circumstances as they arose. Full access to the available disk capacity and the undivided attention of system staff were critical as well.
Results that could have taken years to generate using a conventional allocation structure, with attendant changes in model and hardware architecture, were produced in six months. Time can now be devoted to an extended period analyzing the full results of the experiment, rather than a slowly increasing stream spaced out over a much longer time frame.
A general conclusion from the experience of Project Athena is that the balance of investments by a given HEC center in the overall architecture of its facilities—the capability of the computational engine, the disk capacity, the memory bandwidth and network bandwidth of the HEC and postprocessing systems, the path and bandwidth to the archival system, etc.—may be quite different if the facility serves a large number of diverse users with heterogeneous requirements or if the clients are a small number of projects that gain access to the resources in dedicated mode for substantial contiguous periods of time. Data were saved from the simulations at short intervals (6 hours for most variables and hourly for some), which has served as a valuable resource in several of the investigations of the dataset: saving data at high frequency results in very large archives that require substantial data management resources.
Data sharing.
In addition to publication of technical reports and peer-reviewed papers, the Project Athena results will be made public by enabling free and open access to the model output of the simulations. As a first step to make the results accessible to the public, an atlas containing daily- and monthly-mean maps of selected model outputs is available on the project website (http://wxmaps.org/athena/home/index.html). There are several challenges to making the model output available, not least of which is the staggering volume of data, which exceeds 1.2 PB (1.2 × 1015 bytes). After a careful inventory of the data, eliminating duplicate forms of the same data and model restarts, the data volume still exceeds 0.5 PB. Another challenge is the format of the data. The IFS output is in GRIB format, while the NICAM output was generated in a flat binary format that was later converted to compressed network Common Data Form (netCDF) format. The interoperability of data in these disparate formats is very limited [e.g., Grid Analysis and Display System (GrADS; www.iges.org/grads/grads.html) is among the only data analysis programs that can operate on both types of data in native format]. As of this writing, discussions are underway with various data repositories to serve portions of the model output.
CONCLUSIONS.
Overall, the experience in Project Athena confirmed the general expectation of the World Modeling Summit that dedicated computational resources can substantially accelerate progress in climate simulation and prediction. The availability of such resources not only enabled some detailed explorations of issues that were previously considered beyond the scope of computers used for climate but also was an important incentive for the formation of the international team. Participation by experts in Europe, Japan, and the United States, including computational experts at the NICS facility, was an essential element as well, leading to rapid evaluation and solution of issues as they arose in real time and making it possible for objective comparison of radically different models. An important element of this collaboration was the presence of experts from national modeling centers, which argues in favor of another of the summit's recommendations, namely the enhancement of national modeling capabilities in the key centers around the world. While the pilot project was not able to fully confirm (or reject) the assertions that motivated it, the impact of dramatically increased spatial resolution was apparent for numerous important aspects of climate, including such diverse features as North Atlantic blocking, tropical cyclone intensity, and patterns of regional climate change. Considerable more work is needed to carry on the investigation of how best to take advantage of future improvements in high-end computing for higher fidelity climate simulation and insights into future climate change.
ACKNOWLEDGMENTS
The results described herein were obtained in the 2009–10 Athena Project, a computationally intensive project that was carried out using the Athena supercomputer at the University of Tennessee's National Institute for Computational Sciences (NICS), under the auspices of the National Science Foundation (NSF). Support is gratefully acknowledged from grants from the NSF (0957884 and 0830068), the National Oceanic and Atmospheric Administration (NA09OAR4310058), and the National Aeronautics and Space Administration (NNX09AN50G) to support scientists at COLA. The authors gratefully acknowledge support from Cray, Inc.; the European Centre for Medium-Range Weather Forecasts, which provided the IFS code, boundary and initial conditions datasets, and run scripts; the University of Tokyo; and the Japanese Agency for Marine-Earth Science and Technology, which provided the NICAM code and essential datasets as well as assistance with the conversion and running of the code on Athena. Help from the following individuals is also gratefully acknowledged: P. Bechtold and M. Fuentes of ECMWF; K. Hodges of the University of Reading, United Kingdom; and P. Johnsen and P. Nyberg of Cray, Inc.
REFERENCES
Achuthavarier, D., and Coauthors, 2010: Tropical intraseasonal variability in high-resolution climate simulations. AAMP YOTC MJO Task Force Monsoon Intraseasonal Variability Workshop, Busan, South Korea, WWRP-THORPEX and WCRP, P1.1. [Available online at www.ucar.edu/yotc/documents/mjo/korea_poster_presentations/ses1/Achuthavarier_post.pdf.]
Alley, R. B., and Coauthors, 2007: Summary for policy makers. Climate Change 2007: The Physical Science Basis, S. Solomon et al., Eds., Cambridge University Press, 1–18.
Bauer, M., and A. D. Del Genio, 2006: Composite analysis of winter cyclones in a GCM: Influence on climatological humidity. J. Climate, 19, 1652–1672.
Bechtold, P., M. Köhler, T. Jung, F. Doblas-Reyes, M. Leutbecher, M. Rodwell, F. Vitart, and G. Balsamo, 2008: Advances in simulating atmospheric variability with the ECMWF model: From synoptic to decadal time-scales. Quart. J. Roy. Meteor. Soc., 134, 1337–1351.
Boyle, J., S. Klein, G. Zhang, S. Xie, and X. Wei, 2008: Climate model forecast experiments for TOGA COARE. Mon. Wea. Rev., 136, 808–832.
Chikira, M., and M. Sugiyama, 2010: A cumulus parameterization with state-dependent entrainment rate. Part I: Description and sensitivity to temperature and humidity profiles. J. Atmos. Sci., 67, 2171–2193.
Collins, W. D., and Coauthors, 2006: The Community Climate System Model version 3 (CCSM3). J Climate, 19, 2122–2143.
Dai, A., 2006: Precipitation characteristics in eighteen coupled climate models. J. Climate, 19, 4605–4630.
Dawson, A., T. N. Palmer, and S. Corti, 2012: Simulating regime structures in weather and climate prediction models, Geophys. Res. Lett., 39, L21805, doi:10.1029/2012GL053284.
Dirmeyer, P. A., and Coauthors, 2011: Simulating the diurnal cycle of rainfall in global climate models: Resolution versus parameterization. Climate Dyn., 39, 399–418, doi:10.1007/s00382-011-1127-9.
Dirmeyer, P. A., and Coauthors, 2012: Evidence for enhanced land–atmosphere feedback in a warming climate. J. Hydrometeor, 13, 981–995.
Gottschalck, J., and Coauthors, 2010: A framework for assessing operational Madden–Julian oscillation forecasts, a CLIVAR MJO Working Group Project. Bull. Amer. Meteor. Soc., 81, 1247–1258.
Hamilton, K., 2008: Numerical resolution and modeling of the global atmospheric circulation: A review of our current understanding and outstanding issues. High Resolution Numerical Modeling of the Atmosphere and Ocean, K. Hamilton and W. Ohfuchi, Eds., Springer, 7–28, .
Iorio, J. P., P. B. Duffy, B. Govindasamy, S. L. Thompson, M. Khairoutdinov, and D. Randall, 2004: Effects of model resolution and subgrid-scale physics on the simulation of precipitation in the continental United States. Climate Dyn., 23, 243–258.
Jung, T., and P. B. Rhines, 2007: Greenland's pressure drag and the Atlantic storm track. J. Atmos. Sci., 64, 4004–4030.
Jung, T., S. K. Gulev, I. Rudeva, and V. Soloviov, 2006: Sensitivity of extratropical cyclone characteristics to horizontal resolution in the ECMWF model. Quart. J. Roy. Meteor. Soc., 132, 1839–1857.
Jung, T., and Coauthors, 2012: High-resolution global climate simulations with the ECMWF model in Project Athena: Experimental design, model climate and seasonal forecast skill. J. Climate, 25, 3155–3172.
Kimball, S. K., and M. S. Mulekar, 2004: A 15-year climatology of North Atlantic tropical cyclones. Part I: Size parameters. J. Climate, 17, 3555–3575.
Lin, J.-L., and Coauthors, 2006: Tropical intra-seasonal variability in 14 IPCC AR4 climate models. Part I: Convective signals. J. Climate, 19, 2665–2690.
Manganello, J. V., and Coauthors, 2012: Tropical cyclone climatology in a 10-km global atmospheric GCM: Toward weather-resolving climate modeling. J. Climate, 25, 3867–3893.
Miura, H., M. Satoh, T. Nasuno, A. T. Noda, and K. Oouchi, 2007: A Madden-Julian oscillation event simulated using a global cloud-resolving model. Science, 318, 1763–1765.
Mizuta, R., and Coauthors, 2006: 20-km mesh global climate simulations using JMA-GSM model—Mean climate states. J. Meteor. Soc. Japan, 84, 165–185.
Navarra, A., J. L. Kinter, and J. Tribbia, 2010: Crucial experiments in climate science. Bull. Amer. Meteor. Soc., 91, 343–352.
Nesbitt, S. W., and E. J. Zipser, 2003: The diurnal cycle of rainfall and convective intensity according to three years of TRMM measurements. J. Climate, 16, 1456–1475.
Oouchi, K., A. T. Noda, M. Satoh, B. Wang, S.-P. Xie, H. Takahashi, and T. Yasunari, 2009: Asian summer monsoon simulated by a global cloud-system-resolving model: Diurnal to intra-seasonal variability. Geophys. Res. Lett., 36, L11815, doi:10.1029/2009GL038271.
Orlanski, I., 2008: The rationale for why climate models should adequately resolve the mesoscale. High Resolution Numerical Modelling of the Atmosphere and Ocean, K. Hamilton and W. Ohfuchi, Eds., Springer, 29–44.
Palmer, T. N., F. J. Doblas-Reyes, A. Weisheimer, and M. J. Rodwell, 2008: Toward seamless prediction: Calibration of climate change projections using seasonal forecasts. Bull. Amer. Meteor. Soc., 89, 459–470.
Parry, M. L., O. F. Canziani, J. P. Palutikof, P. J. van der Linden, and C. E. Hanson, Eds, 2007: Climate Change 2007: Impacts, Adaptation and Vulnerability. Cambridge University Press, 976 pp.
Phillips, T. J., and Coauthors, 2004: Evaluating parameterizations in general circulation models: Climate simulation meets weather prediction. Bull. Amer. Meteor. Soc., 85, 1903–1915.
Rashid, H. A., H. H. Hendon, M. C. Wheeler, and O. Alves, 2011: Prediction of the Madden-Julian oscillation with the POAMA dynamical prediction system. Climate Dyn., 36, 649–661, doi:10.1007/s00382-010-0754-x.
Reynolds, R. W., T. M. Smith, C. Liu, D. B. Chelton, K. S. Casey, and M. G. Schlax, 2007: Daily high-resolution blended analyses for sea surface temperature. J. Climate, 20, 5473–5496.
Satoh, M., T. Matsuno, H. Tomita, H. Miura, T. Nasuno, and S. Iga, 2008: Nonhydrostatic Icosahedral Atmospheric Model (NICAM) for global cloud resolving simulations. J. Comput. Phys., 227, 3486–3514, doi:10.1016/j.jcp.2007.02.006.
Satoh, M., and Coauthors, 2012: Intra-seasonal oscillation and its control of tropical cyclones simulated by high-resolution global atmospheric models. Climate Dyn., 39, 2185–2206, doi:10.1007/s00382-011-1235-6.
Shapiro, M., and Coauthors, 2010: An Earth-system prediction initiative for the twenty-first century. Bull. Amer. Meteor. Soc., 91, 1377–1388.
Shukla, J., R. R. Hagedorn, M. Miller, T. N. Palmer, B. Hoskins, J. Kinter, J. Marotzke, J. Slingo, 2009: Strategies: Revolution in climate prediction is both necessary and possible: A declaration at the World Modeling Summit for Climate Prediction. Bull. Amer. Meteor. Soc., 90, 16–19.
Shukla, J., T. N. Palmer, R. Hagedorn, B. Hoskins, J. Kinter, J. Marotzke, M. Miller, and J. Slingo, 2010: Toward a new generation of world climate research and computing facilities. Bull. Amer. Meteor. Soc., 91, 1407–1412.
Slingo, J. M., and Coauthors, 1996: Intraseasonal oscillation in 15 atmospheric general circulation models: Results from an AMIP diagnostic subproject. Climate Dyn., 12, 325– 357, doi:10.1007/BF00231106.
Solomon, S., D. Qin, M. Manning, M. Marquis, K. Averyt, M. M. B. Tignor, H. L. Miller Jr., and Z. Chen, Eds., 2007: Climate Change 2007: The Physical Science Basis. Cambridge University Press, 996 pp.
Stan, C., M. Khairoutdinov, C. A. DeMott, V. Krishnamurthy, D. M. Straus, D. A. Randall, J. L. Kinter III, and J. Shukla, 2010: An ocean-atmosphere climate simulation with an embedded cloud resolving model. Geophys. Res. Lett., 37, L01702, doi:10.1029/2009GL040822.
Taniguchi, H., W. Yanase, and M. Satoh, 2010: Ensemble simulation of cyclone Nargis by a global cloud-system-resolving model—Modulation of cyclogenesis by the Madden-Julian oscillation. J. Meteor. Soc. Japan, 88, 571–591.
Tibaldi, S., and F. Molteni, 1990: On the operational predictability of blocking. Tellus, 42A, 343–365.
Wang, Y., and C.-C. Wu, 2004: Current understanding of tropical cyclone structure and intensity changes— A review. Meteor. Atmos. Phys., 87, 257–278.
Zhao, M., I. M. Held, S.-J. Lin, and G. A. Vecchi, 2009: Simulations of global hurricane climatology, interannual variability, and response to global warming using a 50-km resolution GCM. J. Climate, 33, 6653–6678.
Throughout this article, model resolution will be measured in terms of the average spacing between grid points.
Here, T indicates triangular spectral truncation of atmospheric states represented as a spherical harmonics series expansion, which consists of Fourier transforms in longitudinal and Legendre transforms in the meridional direction.
