a. Grid generation

The icosahedral–hexagonal grid, first introduced in meteorological modeling by Sadourny et al. (1968) and Williamson (1968), has been gaining increasing interest in recent years, for example, Masuda and Ohnishi (1986), Heikes and Randall (1995a,b), Giraldo (1997), and Thuburn (1997). The approach described here closely follows the work of Baumgardner (1983), who has applied this grid to the problem of planetary mantle convection.

To generate the grid, a regular icosahedron (Fig. 1) is constructed inside the sphere such that 2 of its 12 vertices coincide with the North and South Poles. Five of the other 10 vertices are spaced at equal longitudinal intervals of 72° (=360°/5) along a latitude circle at 26.565°N, the other 5 along a latitude circle at 26.565°S.

Connecting nearest neighbors among these 12 points with great circle arcs divides the spherical surface into 20 equal spherical triangles (Fig. 2a). Beginning from this grid of icosahedral triangles, a new finer grid of triangles is generated by connecting midpoints of the spherical triangle sides by an additional set of great circle arcs (Fig. 2b). This process may be repeated until a grid of the desired resolution is obtained (Figs. 2c and 2d). This construction procedure yields a grid consisting of 10 n²_i + 2 grid points (nodes) and 20 n²_i elementary spherical triangles, where n_i is the number of equal intervals into which each side of the original icosahedral triangles is divided. Each of these 10 n²_i + 2 grid points is surrounded by 6 nearest neighbors except for the original 12 icosahedral vertices, which are surrounded by only 5. We therefore refer to these 12 special points as pentagonal points. If we place all variables at the vertices of the triangles (Arakawa A grid), the dual mesh consists exclusively of hexagons except for the 12 pentagons at the pentagonal points.

The number n_i is a natural parameter for specifying the resolution of the grid. It can be shown that there is a close numerical equivalence between n_i and the maximum harmonic degree in a spherical harmonic representation (Yang 1997). The (minimum) spacing between grid points is then the length of a side of the original icosahedral triangles (about 7054 km for the earth) divided by n_i. For example, with n_i = 128 we obtain a spacing between grid points of about 55 km.

The icosahedral–hexagonal grid provides a nearly uniform coverage of the sphere even though the hexagonal cells vary somewhat in their exact shape and size (Table 1a), especially those close to the pentagons. The pentagons, however, are perfectly regular. To increase the available choice of grid resolution, an initial trisection of the main triangles edges followed by bisections may be performed. Specifications for these grids are summarized in Table 1b.

By combining the areas of pairs of the original adjacent icosahedral triangles (Fig. 3), the global grid can logically also be viewed as comprising 10 rhombuses or diamonds, each of which has n_i × n_i unique grid points. The diamonds are indexed as shown in Fig. 3. Five diamonds share the North Pole and five the South Pole. The indexing on a diamond is based on the convention that those n_i × n_i grid points that are unique to each diamond are numbered from 1 to n_i in the rows and columns of the data arrays. The grid points on the diamonds edges, (0, 1) to (0, n_i + 1) and (0, n_i + 1) to (n_i, n_i + 1), are shared between adjacent diamonds and their data values must be exchanged at each time step. The polar points (0, 1) are each shared by five diamonds. Diamonds 1–5 share the North Pole and diamonds 6–10 share the South Pole.

From the computational point of view the icosahedral–hexagonal grid offers the major advantage that no indirect addressing is required. The data structure is regular and has the dimensions (0: n_i, n_i + 1, 10), that is, consists of 10 logically square arrays of points. Discrete differential operators have the form of seven-point stencils, involving the home point and the six nearest neighbor points. The indices of the neighbor points are given by fixed offsets from the index of the home point. These operations can be coded to obtain high efficiencies on both vector- and cache-based computer architectures. Furthermore, the square arrays of points are readily partitioned in a domain decomposition strategy for distributed memory parallel architectures.

b. Horizontal finite-difference operators

The derivation of finite-difference operators is not based on Gauss's theorem as in Masuda and Ohnishi (1986) or Heikes and Randall (1995a) but follows a strategy similar to that of Stuhne and Peltier (1996, 1999). Our approach utilizes local basis functions that are orthogonal and conform perfectly to the spherical surface. These basis functions are the longitude and latitude of a locally defined spherical coordinate system whose equator and zero meridian intersect at the grid point. We generate such a local spherical coordinate system at each grid point with coordinates (η, χ) and align the local east direction to coincide with the global east direction and the local north with the global north direction. The local spherical coordinate system is specified by three orthogonal unit vectors [x₀, (e_λ)₀, (e_φ)₀], where x₀ is the gridpoint location on the unit sphere, (e_λ)₀ is orthogonal to x₀ and aligned with the global east, and (e_φ)₀ is orthogonal to x₀ and aligned with the global north direction (Fig. 4).

The advantages of this local coordinate system are the following:

• within the local neighborhood of the grid point the coordinate system is nearly Cartesian; that is, the coordinate singularities are far removed from this gridpoint neighborhood; and

• only two (tangential) velocity components are needed to describe the horizontal velocities.

However, there is one disadvantage, namely, transformations are required between the local coordinate systems of neighboring grid points when operators are applied to vector fields.

The meteorological equations are formulated and solved in the local spherical system (η, χ), where the horizontal distances (dx, dy) on the earth with radius a are given by dx = a cosχ dη and dy = adχ.

Discrete horizontal operators must be derived for this system. The analytical form of the operators is the usual form, as given, for example, by Dutton (1976), taking into account that η = χ = 0 at the center node.

1) Derivation of the discrete gradient and Laplace operators

To obtain a second-order-accurate representation of the partial differential equations we desire to solve, we approximate an arbitrary global function ψ* in the neighborhood of each grid point by a quadratic polynomial ψ in the local coordinates (η, χ) as

View Expanded

Equation (2.1) may be interpreted as a Taylor series of the form

View Expanded

The finite-difference form of the gradient and Laplace operators for a scalar field ψ may be written in terms of a stencil operation involving the values of ψ at the center node and the nearest five or six neighbors. (For simplicity in what follows, we shall let the neighbor index m range from 1 to 6 also in the case of pentagonal nodes where it will be understood that the stencil coefficients for the nonexistent node are identically zero.)

The gradient operator at each grid point may be expressed as

View Expanded

The coefficients G_η,m, G_χ,m(m = 1, … , 6) are associated with the neighboring nodes and depend only on the geometric locations of the nodes expressed in terms of the local coordinates (η, χ).

Similar to the approach for the gradient operator, the Laplacian operator is expressed in terms of the neighboring nodes as

View Expanded

To obtain the stencil coefficients G_η,m, G_χ,m, and L_m we apply the quadratic polynomial approximation (2.1). For the case of the six nearest neighbors, we have six constraints for the five coefficients α₁, α₂, … , α₅ that specify the function ψ in the local neighborhood, namely,

View Expanded

for m = 1, … , 6. A least squares procedure is used to solve for the five unknown coefficients from a system of the form

α_jβ_j,mψ_mψ₀(2.7)

where j = 1, … , 5, and the summation is over m = 1, … , 6.

Using Eqs. (2.1)–(2.6) the coefficients G_η,m, G_χ,m are then given by

_η,mβ_1,mχ,mβ_2,m(2.8)

and the coefficients L_m by

L_mβ_3,mβ_5,m(2.9)

where m = 1, … , 6.

Due to the symmetry of the icosahedral–hexagonal grid, the coefficients G_η,m, G_χ,m, and L_m may be precomputed and stored only for diamond 1. The same gradient coefficients are used to compute the velocity divergence. Care has to be taken, however, to rotate the wind components (u_m, υ_m) of the surrounding nodes into the local spherical coordinate system of the central node before the divergence operator is applied.

c. Interpolation in the icosahedral–hexagonal grid

GME uses semi-Lagrangian advection for water vapor and cloud water. Semi-Lagrangian methods require the interpolation of fields from neighboring grid points to the departure and midpoints of the parcel trajectory. Our approach involves two types of interpolation, namely bilinear and biquadratic. Bilinear interpolation is used in the calculation of the trajectory itself to derive the wind components (u, υ) at the trajectory midpoint. Biquadratic interpolation is then applied to find the prognostic fields at the departure point of the trajectory. Both methods are performed on grid triangles.

1) Bilinear interpolation

Bilinear interpolation of an arbitrary function ψ(η, χ) at a point P(η, χ) uses the values (ψ₀, ψ₁, ψ₂) at the three grid points (P₀, P₁, P₂) having position vectors (p₀, p₁, p₂) that are the vertices of the spherical triangle containing the point as indicated in Fig. 5. To derive the value ψ(η, χ) at P(η, χ) we introduce barycentric coordinates. Each point within the triangle is uniquely defined by the vector

View Expanded

where (γ₀, γ₁, γ₂) are called the barycentric coordinates of the point P. To calculate these coordinates the following linear system has to be solved (note that at the central node P₀, η = χ = 0):

View Expanded

The bilinear interpolation of ψ(η, χ) within the triangle is then obtained by weighting the values of ψ at the triangle vertices by the corresponding barycentric coordinates:

View Expanded

2) Biquadratic interpolation

The standard biquadratic interpolation formula (Zienkiewicz 1979) for a triangle in terms of values at the triangle vertices and midpoints of the edges (Fig. 6) is applied to obtain the value of ψ at an arbitrary point P(η, χ) in the triangle:

View Expanded

where (γ₀, γ₁, γ₂) are again the barycentric coordinates of the point P. The values of the function ψ at the midpoints of the triangle edges, ψ₄, ψ₅, ψ₆, are obtained by approximating ψ along these edges with a cubic Hermite polynomial using the gradients at the end points, that is, at the triangle vertices (P₀, P₁, P₂). When the stencil of the gradient operator is taken into account, the biquadratic interpolation is based on a stencil that involves 12 grid points (Fig. 7).

Monotonicity may be enforced by simply demanding that the interpolated value not be higher or lower than the values at the three corner points (P₀, P₁, P₂). In the same way, positive definiteness may be enforced by the condition that the interpolated value be greater than or equal to zero.

If the Courant numbers are restricted to values less than unity, it is fairly easy to determine which of the surrounding triangles contains the departure or midpoint of the trajectory. Without this restriction the search algorithm is more complicated and uses a binary search procedure to accelerate convergence.

d. Accuracy of the gradient and Laplace operators

Heikes and Randall (1995b) introduced the following function to test the accuracy of their finite-difference operators on the icosahedral–hexagonal grid:

β_m,nλ,a²⁴mnλ(2.14)

where λ is the longitude, φ is the latitude, a is the radius of the earth, and m and n are integers set to 1 or 3. For different resolutions n_i of the grid the analytical solution x^true is compared to the finite-difference one x^fd, and some error norms are evaluated. The one norm is defined by

View Expanded

where the summation is over all N grid points of the icosahedral–hexagonal grid, A_i is the area of a particular hexagon (pentagon), and A is the area of the globe:

View Expanded

The two norm is defined by

View Expanded

and the infinity norm is defined by

View Expanded

For the gradient operator, the norms of course include both components in the summations and in the evaluation of the maximum over the grid.

A finite-difference operator is said to be consistent if the infinity norm converges to zero for decreasing mesh sizes. Figure 8 summarizes the results for the GME gradient and Laplace operators. Both operators satisfy the consistency requirement. Their overall accuracy as characterized by their one and two norms is second order because the norms drop close to a factor of 4 when the resolution n_i is doubled. The GME operators constructed from the unaltered icosahedral–hexagonal grid thus display an accuracy similar to that of the operators derived by Heikes and Randall (1995b, Fig. 4) on their twisted icosahedral–hexagonal grid where a special optimization of the gridpoint distribution has been performed. Note that in Fig. 8, we include results not only for grids constructed with an initial bisection of the sides of the icosahedral triangles but an initial trisection as well. We also show results for grids with n_i values up to 768, corresponding to a horizontal resolution of about 10 km.

It should be noted that the slope of the infinity norm changed from −2 to −1 for the Laplace operator. This occurs where errors due to local grid nonuniformity begin to dominate those due to the inherent inability of the finite grid to represent the function exactly. These maximum absolute errors captured by the infinity norm due to grid nonuniformity occur along arcs corresponding to the sides of the original 20 spherical triangles. The magnitude of these errors decreases by a factor of 2 as n_i is doubled while the area associated with such points also decreases by a factor of 2. The fact that the one and two norms involve an area weighting factor explains why these norms maintain a slope close to −2 where the infinity norm switches to a slope of −1.

a. Differential form of model equations

The prognostic equations for the three-dimensional version of the model are expressed in differential form in terms of a local spherical coordinates (η, χ) and a hybrid (sigma pressure) vertical coordinate ξ as follows:

View Expanded

where (u, υ) are the zonal (meridional) wind components; T is the temperature; p_s is the surface pressure; q_υ is the specific water vapor content and q_c is the specific cloud liquid water content; t is the time and a is the mean radius of the earth (a = 6 371 229 m); ζ is the vorticity and f is the Coriolis parameter; is the vertical velocity in the hybrid system and ω is the vertical velocity in the pressure system; α is the density of air; Φ is the geopotential and K is the specific kinetic energy; p is the pressure and T_υ is the virtual temperature; T_ref is a reference temperature depending only on height; L_υ is the latent heat of condensation; C_υc is the condensation rate and ( )_sub is the subgrid-scale tendency due to parameterized processes like radiation, convection, or turbulence; and K₄ is the constant coefficient of linear fourth-order diffusion.

b. Numerical solution of the three-dimensional equation set

The shallow water test bed for GME included a semi-Lagrangian and an Eulerian version of the code. The semi-Lagrangian version was restricted to Courant numbers less than 1. For larger Courant numbers the communication pattern become more complicated (“communication on demand”), and the extensive code changes required were therefore not implemented in the short period of time available for the project. In the framework of the shallow water model both schemes produced very similar results but the Eulerian code was about 20% faster. Therefore, the dry part of the three-dimensional version of GME, that is the prognostic equations for u, υ, T, p_s, is solved by the semi-implicit Eulerian method. Only the two prognostic moisture equations (q_υ, q_c) use semi-Lagrangian advection (Staniforth and Côté 1991) in the horizontal direction to ensure monotonicity and positive definiteness. An Arakawa A-grid staggering (Mesinger 2000) is employed because of the semi-Lagrangian advection and ease of implementation. In the vertical, the energy and angular momentum conserving finite-difference scheme of Simmons and Burridge (1981) is applied to all prognostic equations.

The semi-implicit treatment of gravity waves (Robert 1981) leads to a three-dimensional Helmholtz equation for the second temporal derivative of the divergence of the horizontal wind field. The eigenvectors of the vertical structure matrix are used to diagonalize this 3D equation into a set of 2D Helmholtz equations corresponding to the number of layers in the model. A split semi-implicit scheme (Burridge 1975) is employed to solve the Helmholtz equations for only the external mode plus the first four internal ones. Currently, these five 2D equations are solved by successive overrelaxation. About 20 iterations are needed to solve for the external mode, and only 3–11 for the internal ones. A slight off-centering of the implicit terms is necessary to damp the gravity waves and to stabilize the solution. Part of the subgrid-scale tendencies is also treated implicitly for stability reasons.

c. Physical parameterizations

Unresolved atmospheric processes interact with the large-scale flow but contain also essential forecast information (e.g., cloudiness or precipitation), which cannot be generated by the adiabatic part of the model. The simulation of such processes in GME is handled by a set of dedicated parameterization modules. The simulation of diabatic processes in the icosahedral–hexagonal grid of GME generally employs the same methods and procedures applied in other NWP grid schemes. However, the uniformity of the GME grid avoids unnecessary physics calculations in overresolved high-latitude zones that commonly occur in grids with polar singularities (e.g., regular latitude–longitude grids). In contrast to such grids, where the area represented by each grid node varies strongly with latitude, the distinction between resolved and unresolved atmospheric scales does not depend on the geographical position in the GME grid because the area of grid nodes varies only moderately. The following physical phenomena are simulated by parameterization modules:

• radiative transfer of solar and thermal radiation in clear and cloudy atmospheres (Ritter and Geleyn 1992) (a full radiation step is performed every 2 h at all grid points, solar fluxes are computed each time step taking the actual zenith angle but the atmospheric transmission from the previous full radiation step is used);

• grid-scale precipitation scheme including parameterized cloud microphysics (Doms and Schättler 1997);

• deep and shallow convection based on a mass flux approach (Tiedtke 1989);

• vertical turbulent fluxes (Müller 1981), based on Louis (1979) in the Prandtl layer, and a diagnostic level-two scheme based on Mellor and Yamada (1974) for the boundary layer and the free atmosphere;

• subgrid-scale orographic effects (Lott and Miller 1997);

• soil model (Jacobsen and Heise 1982); and

• cloudiness derived from specific cloud liquid water content, relative humidity, convective activity, and stability.

With the exception of the subgrid-scale orographic effects scheme that was adapted from the operational ECMWF forecast model, the parameterization modules are nearly identical to those in the previous NWP system of the DWD, where they underwent extensive testing and evaluation both in global and limited area model applications. In the framework of GME, some of the parameterization schemes required some adjustments of the free parameters utilizing available validation and verification data.

d. External parameters

We refer to time-invariant gridpoint properties, such as mean orographic height, land–sea fraction, roughness length, and soil type, as external parameters. They are computed for each grid node area from high-resolution supplementary data. Table 2 summarizes the datasets used in the generation of external parameters for GME.

For each icosahedral–hexagonal grid node high-resolution raw data values associated with the corresponding geographical location are combined to form an area average. At the current operational resolution of 59.9 km, the average grid node area is 3100 km². For some parameters processing of the data also includes conversion from the basic information available, for example, soil texture, to the model parameter required, for example, soil type. In geographical regions where more than one raw dataset is available for the same external parameter, a priority rule is applied using quality assessment of the raw datasets. For most grid elements in Europe land-use-dependent parameters are based on the Coordination of Information on the Environment (CORINE) database information, for example, rather than on the lower-resolution Global Land Cover Characterization (GLCC) dataset.

a. Intermittent 4D data assimilation suite

The data assimilation scheme of the GME is based on a traditional intermittent 6-hourly analysis–forecast cycle. Analyses are performed at 0000, 0600, 1200, and 1800 UTC based on all observations valid in a 1.5-h window around the analysis times. A 6-h forecast of the GME provides the first guess for the analysis scheme.

Table 3a outlines the salient features of the upper air analysis. A multivariate optimum interpolation scheme provides the analysis of the mass (surface pressure and geopotential) and wind (zonal and meridional wind components) fields simultaneously. The correlation functions employed until now are the ones of the former global spectral model (T106, L19) of the DWD. They will be replaced by functions properly describing the error statistics of GME by the end of the year 2001.

The only surface fields analyzed so far are the sea surface temperature and the snow depth (Table 3b). No analysis of soil temperatures or water content is performed, but the 6-h first guess fields from GME are applied.

b. Incremental digital filtering initialization

Initialization schemes are designed to remove noise from the forecast while introducing acceptably small changes to the analysis and forecasts. Furthermore, if the initialization can achieve a better balance between humidity and dynamic fields, the spinup problem is alleviated. For GME we apply the digital filtering initialization (DFI) of Lynch (1997) that involves a 3-h adiabatic backward integration and a 3-h diabatic forward one centered around the initial time. The incremental approach employed avoids unwanted smoothing of the first guess fields due to the DFI in regions without observations by applying the filter only to the analysis increments.

a. Daily schedule of analyses and forecasts

Since 1 December 1999 GME has been the operational global NWP model of the DWD and currently provides the meteorological database for many follow-up products and systems. GME and its data assimilation scheme are implemented on the Cray T3E1200 of the DWD. The gridded binary (GRIB) code analysis and forecast data are stored in huge ORACLE data bases on an SGI Origin cluster.

The operational schedule is structured by data assimilation steps every 6 h, that is, at 0000, 0600, 1200, and 1800 UTC, with a data cutoff between 7 and 12 h. An early run with a data cutoff of 2 h, 14 min, and forecasts up to 78 h allows early numerical guidance, and provides lateral boundary conditions for the nonhydrostatic regional model (LM; 7-km mesh size, 35 layers) of the DWD as well as the regional models of 11 other national meteorological services (see section 5c). The early runs are based on 0000, 1200, and 1800 UTC analyses and use 13 × 13 processors. To complete the 78-h forecast takes about 50 min of wallclock time. The main run with a data cutoff of 3 h, 30 min, and forecasts up to 174 h is based on 0000 and 1200 UTC analyses. With 15 × 15 processors the whole 174-h forecast takes 1 h, 35 min.

b. Available products

GME data are mostly stored on the icosahedral–hexagonal grid (Arakawa A, 163 842 grid points, 31 hybrid layers). More than 80% (11 GB) of the data of a 174-h forecast are given in this original spatial representation. Software is available to extract the GME forecast at single grid points anywhere on the globe to derive meteographs. To ease the data visualisation and as an interface to applications like wave modeling, selected forecast fields are interpolated horizontally from the icosahedral–hexagonal grid to a regular latitude–longitude one (0.75° × 0.75°). In addition, some multilevel fields are interpolated vertically from the 31 model layers to selected pressure levels.

c. GME data as lateral boundary conditions at other NMS

Forecast data from the early run of GME are sent via the Internet to other national meteorological services (NMSs). These data serve as initial and lateral boundary conditions for regional NWP models, which are based on either the high-resolution regional model of the DWD or the nonhydrostatic LM. Only those GME grid points within the domain of interest of the NMS in question are transmitted to reduce the amount of data. In this way real distributed computing is realized, with GME at the DWD and the regional models at the NMS running in parallel. Currently, the following 11 NMSs are receiving the GME data twice daily based on 0000 and 1200 UTC data out to 48 (78) h at 3-hourly (for some, even at hourly) intervals:

• Directorate of Hydrography and Navigation (Brazil),

• Instituto Nacional de Meteorologia (Brazil),

• Guangzhou Regional Meteorological Centre (China),

• National Meteorological and Hydrological Service (Greece),

• Israel Meteorological Service (Israel),

• Regional Service SMR-ARPA (Italy),

• National Meteorological Service, DGCAM (Oman),

• National Meteorological Service, IMGW (Poland),

• National Meteorological and Hydrological Service (Romania),

• MeteoSwiss (Switzerland), and

• National Meteorological and Hydrological Service (Vietnam).

Most of the regional models running at the NMSs have horizontal resolutions between 30 and 7 km; they are able to add valuable details to the GME forecast because the local topographical forcing generally modifies the larger-scale flow being provided by GME.

a. Systematic trends in 60-day forecasts

To detect systematic trends in GME forecasts a 60-day (=1440 h) run was performed at the operational resolution of 60 km and 31 layers. The forecast was initialized on 15 December 1999 at 0000 UTC and used constant sea surface temperature. Global diagnostics were produced each day to monitor the model evolution. The results are summarized in Fig. 9. No obvious trends are visible. The volume average of the kinetic energy (Fig. 9a) varies slowly within the range of 150–175 m² s⁻², while the maximum wind speed (usually found at the top level, i.e., at 10 hPa) fluctuates on a much shorter timescale between 80 and 140 m s⁻¹. Although the mass (Fig. 9b) is not formally conserved, the mean deviation from the initial state never exceeds 0.14 hPa. The hydrological quantities (Figs. 9c and 9d) seem to be balanced rather well throughout the 60-day period. Only a slight tendency is visible to shift the precipitation from the convective to the grid-scale regime.

b. Verification of precipitation forecasts

The Global Precipitation Climatology Centre (GPCC; Rudolf et al. 1996) provides an objective analysis of monthly precipitation. This analysis is based on measurements at about 6000 surface stations over land and estimated amounts derived from brightness temperature observations from geostationary satellites over the oceans. The spatial resolution of the combined product is 2.5° × 2.5° (Fig. 10b). For February 2000, the daily precipitation forecasts of GME for the 24-h period 6–30 h were accumulated to derive a monthly value (Fig. 10a). There is close correspondence between observation and simulation of the main features such as the precipitation extremes at the ITCZ and the storm tracks of both hemispheres. Even the heavy flooding that struck Mozambique in February 2000 was forecast by GME remarkably well. Regarding the distribution over the continents, the model is able to simulate the topographical modification of the precipitation field (e.g., in South America) in more detail than can be obtained from the GPCC analysis because of the coarse resolution of the observing network and analysis grid.

c. Kinetic energy spectra

Based on the horizontal wind components at icosahedral–hexagonal grid points of GME we compute the kinetic energy spectrum by replacing integrals by summations over grid nodes. At the operational resolution n_i = 128 with 163 842 grid points, a triangular truncation of up to T340 is possible. Figure 11 shows the resulting spectrum of the eddy kinetic energy (after summation over the zonal index m for m > 0) at 250 hPa on 25 May 2000 at 0000 UTC. The full spectrum is shown on the left, the higher end between wavenumbers 150 and 340 is shown to the right. No “wavenumber 5 problem” can be detected in the GME spectrum. For comparison the spectrum of the ECMWF model (T_L319, 60 layers) is also displayed. Both models show close agreement up to wavenumber 100, especially the n⁻³ drop of energy in the wavenumber range between n = 10 to n = 100. At higher wavenumbers, the GME spectrum falls off at a rate close to n^−5/3, while the ECMWF one drops off at a much higher rate indicating a stronger (and more effective) horizontal diffusion of the model. Integrated over the full spectrum, both models have almost the same level of eddy kinetic energy (133.8 m² s⁻² for the GME, and 135.8 m² s⁻² for the ECMWF model).

d. Christmas 1999 storm in France and Germany or “the flap of a butterfly's wing”

On 25 December 1999, a strong winter storm with peak gusts of more than 200 km h⁻¹ resulted in widespread damages in France, Belgium, Switzerland, and Germany, and caused the deaths of more than 80 people. Afterward, the media blamed the meteorological services of the countries hit by the storm for not having warned the public early enough.

This storm developed from a cyclone that appeared on 26 December 1999 at 0000 UTC west of Brittany with a central pressure of about 980 hPa. While moving rapidly east-northeastward the storm deepened by about 20 hPa in only 6 h. Mean winds of about 50 kt and peak gusts of more than 90 kt were reported. The translation speed of the storm center exceeded 50 kt (∼90 km h⁻¹). At 1200 UTC the system had reached Frankfurt (Main) (see Fig. 12a). At the southwestern flank of the cyclone very stormy winds occurred that caused severe damage in southern Germany. In the mountainous region of the Black Forest, large areas of forest were completely destroyed.

The operational GME severely underestimated the strong development of this system even in the 24-h forecast based on the 25 December 1200 UTC analysis (Fig. 12b). Peak gusts of less than 30 kt were predicted for southern Germany, and no closed low pressure system was generated over Germany by the model. What was the reason for this serious failure?

A first hint can be identified by comparing the 48-h forecasts of GME based on the early and main runs, which started at 1200 UTC on 24 December. The early run (Fig. 12c) has a data cutoff around 2 h, 14 min, past the analysis time, whereas for the main run (Fig. 12d) the cutoff is 3 h, 30 min. The two forecasts differ dramatically! The early run shows 48 h in advance a clear signal of a strong cyclone with peak gusts around 30 m s⁻¹ (∼60 kt) in southern Germany, while the main run predicts a remarkably different weather situation, namely, a well-developed storm over southern England.

The only difference between the early and the main runs of GME is the initial state, that is, numerical analysis valid at 1200 UTC on 24 December 1999. There are small differences between both analyses in the region of the initial disturbance, a shallow low pressure system east of North America at 38°N, 55°W, which later developed into the Christmas storm. The additional observations used for the main run, for example, a restarted radiosonde at Sable Island and some geostationary satellite observations of winds derived from cloud drift, resulted in a reduction of the speed of the upper-tropospheric jet by some 6 m s⁻¹. This relatively small difference in the initial state (or “flap of a butterfly's wing”) caused a serious failure in the forecast 48 h later (and more than 4000 km to the east) over Germany. Thus there is reason to believe that the weather situation was strongly chaotic during this period. A small change of the initial conditions lead to drastic changes of the forecast only 48 h later.

This obvious dependence of the forecast quality on the initial state prompted detailed investigations by the DWD data assimilation section regarding the optimal use of all available observations, for example, from ships, planes, and satellites. Figure 13 shows as an example the 24-h forecast of GME, started at 1200 UTC on 25 December 1999, based on an experimental data assimilation where the observation window has been reduced from ±3 to ±1.5 h around the analysis times 0000, 0600, 1200, and 1800 UTC. This reduction avoids the “smearing out” of the information in the rapidly changing flow. With a mesh size of 60 km, GME is of course not able to simulate the rapid deepening (and filling) of the storm over France in all the mesoscale details, but comparison (Fig. 14) between the observed and simulated temporal evolution of the surface pressure at Paris (Orly Airport) reveals reasonable correspondence. Even the nonhydrostatic high-resolution regional numerical weather prediction model LM with its 7-km grid mesh is unable to capture the rapid deepening and filling.

The reduction of the observation window improved the forecast quality of GME not only in this case, but also for the second French storm (28 Dec 1999) and for the Danish storm (3 Dec 1999) as well. Since May 2000, this shortened observation window has been adopted for all operational forecasts.

a. Parallelization on MPP systems

Since the design of the GME included a domain decomposition from the beginning, it took only 3 months to parallelize the program using Message Passing Interface (MPI) software for message passing. The code is written in standard FORTRAN90 and is fully portable. For the two dimensional domain decomposition (Fig. 15) the (n_i + 1)² grid points of each diamond are divided among n₁ × n₂ processing elements (PEs). Thus each PE computes the forecast in a subdomain of all 10 diamonds. This approach improves the chance of achieving a better load balancing for the physical parameterizations, for example, between day and night, land and sea, or rain and no rain. For example, on 13 × 13 PEs of a Cray T3E1200 the physical parameterizations for a 24-h real data forecast consume between 220 and 298 s of wallclock time; the average time is 267 s. In the current version of GME, each computational subdomain has a halo of just two rows and columns of grid points that have to be exchanged via MPI with those PEs that compute the forecast in the neighboring subdomains. There are only seven synchronization points during one complete forecast step. It should be noted that good load balancing requires that all processors have nearly the same workload; thus, the difference between the mean and the maximum number of grid points in the computational subdomains should be small.

b. Performance of the GME on the Cray T3E

The GME has been ported successfully to several parallel platforms based on vector or reduced instruction set computing (RISC) processors. However, the effort devoted to optimizing the GME code for different computer systems has been rather limited so far; thus, there is substantial potential for further improvement of the performance of the model.

Figure 16 shows the speedup of the GME (60 km, 31 layers) on a Cray T3E1200 for a 24-h real data forecast without postprocessing. Between 5 × 5 and 13 × 13 PEs, an almost linear speedup is obtained. About 60 PEs are necessary to perform a 24-h forecast in less than 30 min.

The distribution of the cost for the different parts of GME expressed as a percentage of the total number of floating point operations on a Cray T3E1200 is highlighted in Fig. 17.

c. Global forecasts at resolutions ranging from 160 km down to 15 km

Current global models of major NWP centers employ horizontal mesh sizes of about 60 km and require about 5 × 10¹² floating point operations (Flops) for a 24-h forecast. A few years from now, global models with mesh sizes in the range between 10 and 20 km will be feasible. We have recently tested GME on the Fujitsu VPP5000 of the ECMWF for mesh sizes of 160, 120, 80, 60, 40, 30, 20, and 15 km. The initial state for the test runs was derived from a rather coarse resolution analysis (T106L19, i.e., a mesh size of about 120 km, 19 layers) with interpolation to the GME grids. Thus these runs cannot show the full potential of high-resolution global modeling because the data assimilation part is missing. Here, the main goal is to test the behavior of the GME at different resolutions from the computing point of view. The case chosen is the Christmas storm (see section 6d) with the initial date 25 December 1999 at 1200 UTC. A 24-h forecast with postprocessing only at 18 and 24 h was performed for each resolution. The results are summarized in Table 4 and Fig. 18.

A halving of the mesh size Δ, that is, a doubling of the resolution, is normally associated with a factor of 8 increase of the computational cost of the forecast. This is due to the fact that the time step of the model usually has to be halved as well, due to the Courant–Friedrichs–Lewy criterion. Looking at the results presented in Table 4, in practice this factor is much less for GME, for example, for a reduction of the mesh size from 160 to 80 km, the factor is only 5.65. This smaller increase in the computational workload is a consequence of model processes that do not depend on the time step directly, namely, the parameterization of radiation and postprocessing. A full radiation step is performed every 2 h at each GME grid point. At a coarse resolution of 160 km, radiation contributes about 40% of the total computational cost, but at high resolutions like 20 km this cost falls to 11%. Moreover, on vector machines like the Fujitsu VPP5000 the execution time of the model run may not increase as much as expected because of the greater vector length at higher resolutions. For example the speed per processor increases from 0.9 GFlops at 160-km resolution (with an average vector length of 62 elements) to 2.3 GFlops at 20 km (with an average vector length of 348 elements).

Of course, at very high resolutions like 10–15 km the number of layers should be much higher than 31. Moreover, a full radiation step may be necessary every 15–30 min for a proper cloud–radiation feedback. Thus the true computational cost of GME at 10-km resolution will be surely more than twice the numbers given in Table 4.

From the meteorological point of view, the surface pressure forecasts at the different mesh sizes (Fig. 18, isolines) differ only in the details for resolutions finer than 120 km. At coarser resolutions, the storm moves too slowly to the east. On the other hand, prediction of peak gusts clearly benefits from higher horizontal resolution (Fig. 18, shading). Simulation of the observed gusts of up to 32 m s⁻¹ over France, Germany, and Switzerland requires a high-resolution description of the topography.