a. Tracer conservation and freshwater forcing

Many ocean models employ a constant volume. For example, the rigid-lid approximation sets ∂_tη = 0, thence eliminating fast barotropic waves. With this assumption, volume conservation via Eq. (1) leads to the balance q_w = ∇ · U, which forms the basis for the work of Huang (1993). The corresponding barotropic dynamics require both a streamfunction and velocity potential, thus introducing two elliptic problems. Elliptic problems are generally inefficient to solve, especially in the presence of realistic geometry, topography, and surface forcing, and moreso on parallel computers (see section 5). Bryan (1969) made the further assumption that the vertically integrated velocity has zero divergence, resulting in a zero vertical velocity at the ocean surface: w(z = 0) = −∇ · U = 0. This assumption eliminates the velocity potential, yet at the cost of precluding a direct freshwater forcing.

In principle, implementation of tracer forcing in flux form is sufficient to ensure that tracer is globally conserved. However, with time-dependent cell thicknesses, details related to the discrete time stepping scheme, including the use of time filters necessary with the three-time-level leapfrog scheme, serve to preclude strict conservation. Additionally, global conservation is neither necessary nor sufficient to ensure locally conservative behavior. For example, consider an ocean at rest with globally constant tracer concentration. Then allow a wind to blow so to initiate velocity and surface height fluctuations, yet do not introduce any tracer surface fluxes. In addition to global tracer content remaining fixed, the tracer concentration locally at every point in the ocean will maintain the same constant value. This behavior provides an example of “local” tracer conservation, which is not a necessary result of global conservation. Instead, it arises so long as there is compatibility between tracer and volume budgets.

b. Splitting between fast and slow dynamics

For computational efficiency, primitive equation ocean models aim to exploit the timescale split between the fast barotropic gravity waves, which can propagate at some 200 m s⁻¹, and the much slower (some 100 times slower) remaining dynamics. Such models approximate the barotropic mode by the depth-averaged motion, and the baroclinic mode is approximated by the deviation from the depth average. In a rigid-lid ocean model, vertical averaging provides a clean split between the fast and slow modes. In contrast, vertical averaging does not completely separate out the fast dynamics in a free surface model, since in this case the barotropic mode is weakly depth dependent. The papers by KSWP, Dukowicz and Smith (1994), Higdon and Bennett (1996), Higdon and de Szoeke (1997), and Hallberg (1997) provide discussions of these issues, which can be quite subtle yet important for the purpose of providing a stable split between the fast and slow modes in realistic ocean models. Additionally, details of this split have crucial implications for the tracer conservation properties of the model.

c. Modeling context and summary of key results

MOM has traditionally been a rigid-lid ocean model, where the streamfunction algorithm of Bryan (1969) was used to solve for the barotropic component of the dynamics. In the 1990s, important alternatives have been added, including the explicit free surface of KSWP, the rigid lid–surface pressure of Smith et al. (1992) and Dukowicz et al. (1993), and the implicit free surface of Dukowicz and Smith (1994). Version 3 of MOM (MOM 3) includes all of these algorithms within a single model framework. There are numerous additional developments using a free surface in z-coordinate ocean models, such as that used in the Ocean Circulation and Climate Advanced Modelling project (OCCAM; Webb et al. 1998), where the KSWP scheme is refined, as well as the Hamburg Ocean Primitive Equation (Wolff et al. 1997), Massachusetts Institute of Technology (Marshall et al. 1997), and Océan Parallélisé (OPA; Roullet and Madec 2000) ocean models, each of which employ an implicit time stepping algorithm. The present paper is the result of an effort to assess each of these free surface schemes, and possible alternatives such as those of Blumberg and Mellor (1987) and Mellor (1996), who use a terrain-following coordinate model, and Bleck and Smith (1990) and Hallberg (1997) who employ isopycnal layered models.

When testing the various free surface methods, we kept the following goals in mind: 1) MOM’s dynamical core should provide a faithful representation of large-scale ocean dynamics in an efficient manner for use in both coarse (>1°) and fine (<1°) resolution global and regional experiments. 2) Because of the increasing importance of parallel computation, the model should scale well as the number of computer processors increases. 3) The input of freshwater should occur naturally and allow tracer conservation. As described in this paper, our preferred approach is to use an explicit free surface method, which is shown here to satisfy these three goals.

There are important differences between the free surface method as described in this paper and the other free surface methods currently in use with z-coordinate models. First, the KSWP, Dukowicz and Smith (1993), Wolff et al. (1997), and Marshall et al. (1997) approaches assume the top model box to have fixed volume for purposes of the tracer and baroclinic momentum budgets, and so do not satisfy the third goal above. Second, the OCCAM model (Webb et al. 1998) relaxes this constraint, although details of their method are not documented. As shown in this paper, allowing the top box to change in time is necessary, but not sufficient, to globally and locally conserve tracer when using a free surface method. Details of the time stepping scheme are crucial and warrant careful analysis. Third, the OPA model also relaxes the fixed top cell volume constraint, yet it employs an implicit approach that is arguably less straightforward than an explicit approach, in addition does not tend to perform as well on parallel machines [though advances are available, as documented by Guyon et al. (2000)].

d. Contents of this paper

This paper consists of the following sections. Section 2 describes the model’s discrete tracer equations. It highlights points concerning tracer conservation in the presence of a time-dependent surface cell. Section 3 describes the model’s discrete equations of motion and details the time stepping algorithm for the barotropic and baroclinic systems. Section 4 exhibits results from idealized numerical examples that provide some practical illustrations of the method. Section 5 discusses issues related to using the method in a parallel computational environment. Section 6 finishes the paper with conclusions.

a. Continuous time tracer budget

Specification of the surface tracer flux involves details of how the air–sea interface is modeled. A simple “closure” for the turbulence term, which is often used for ocean-only simulations, is a restoring condition Q^turb_T = γh^t(T₁ − T∗), where T∗ is the reference tracer, γ is an inverse damping time, and T₁ is the time-lagged surface tracer concentration. For a more refined coupling of ocean and atmosphere, the flux Q^turb_T and the tracer T_w have to be prescribed, for example, by a boundary layer model. For salt, which does not pass the sea surface and is conserved exactly, Q^turb_T = 0 and T_w = 0 is appropriate (salt is considered in detail in section 2g). For other tracers, approximations depend on the details of the boundary layer model [a simple approximation is given in section 2f; see also Pacanowski and Griffies (2000) or Large and Pond (1981) for more discussion].

b. Time discretization

The question arises whether it is preferable to time step the thickness-weighted tracer concentration h^tT or the tracer concentration T. Tests with both approaches were performed and showed negligible differences. Notably, as shown below, both approaches have the same temporal order of accuracy, and so choosing one is a matter of convenience. As time stepping T involved the least changes to the existing MOM code, we detail that approach in the following. A similar reasoning applies to the momentum equation time discretization discussed in section 3.

c. Compatibility between volume and tracer budgets

Recall the example considered in section 1a, in which an ocean at rest with an initially constant tracer concentration is perturbed via a wind stress yet without tracer sources. Again, the solution should maintain the same constant tracer concentration locally at each grid cell, even as the surface cells undulate. To do so, it is necessary for there to be compatibility between the volume and tracer budgets. In particular, upon setting the tracer concentration to a constant, the tracer budget in the surface grid cells must reduce to the discretized surface height equation (1). Compatibility between the budgets thus imposes a constraint on how to time step the surface height, and such is detailed in section 3c.

d. Total tracer content in the discrete model

More generally, manipulation of the tracer concentration budget (6), again assuming that both the tracer concentration and surface height are time stepped via a leapfrog, leads, in the absence of boundary fluxes, to conservation of the total tracer content as measured by

View Expanded

where A_ij is the horizontal area of a tracer cell. Again, this measure of total tracer content is the same, to second-order accuracy in Δτ, as the analogous measure realized via time stepping hT instead of T.

e. Local compatibility versus global conservation

The compatibility condition, which leads to a locally conservative scheme, and global tracer conservation cannot be simultaneously maintained with a stable three-time-level discretization. For example, use of a time filter on either the tracer (via the Robert filter) or the surface height (via a time averaging procedure described in section 3), preclude strict conservation of the total tracer content given by Eq. (10). However, these filters do not preclude local compatibility between volume and tracer budgets. Alternatively, omitting the time filters on the surface height and tracer in the tracer budget (11) allows for exact global conservation, yet such breaks the local compatibility so long as a time filter is used in the surface height equation (11). We provide two examples in section 4 that aim to quantify these points. Given the inability to provide exact discrete local and global conservation properties with an undulating free surface and the leapfrog scheme, we generally believe it prudent that conservation properties should be assessed on a case by case basis.

Alternative time stepping schemes exist, such as two-time-level schemes (e.g., Hallberg 1995, 1997), which provide potential remedies to the issues raised here. However, pursuit of such alternatives goes beyond our present scope.

f. Comments on surface fluxes

Assuming that the tracer concentration in the freshwater is the same as in the surface cell, T_w ≈ T₁, makes the tracer time tendency independent of any explicit freshwater forcing. Instead, the dependence is restricted to the affects that freshwater has on the convergence −∇ · U. This approximation simplifies the setup of boundary conditions for the tracer flux. However, in general T_w and T₁ are distinct and so T_w may need to be specified explicitly from data or another component model.

g. The special case of salt

a. Continuous time momentum budget

On the B grid used in MOM, the offset in the horizontal between velocity and tracer/density points means that the hydrostatic pressure at the western face of a surface velocity cell is given by

View Expanded

where α^y is a meridional grid average and η_i is on the tracer point. The first piece of this pressure, p_b ≡ (g|z₁|/2)ρ_i^y, represents a discretization of the baroclinic pressure in the surface cell, whose horizontal gradients arise from baroclinicity in the density field. Note that z₁ has been assumed independent of horizontal position, hence its removal from the meridional average operator. The second contribution, p_s ≡ gρ_iη_i^y, represents a discretization of the surface pressure, whose gradients arise from those in the density-weighted free surface height. Here, p^atm represents an atmospheric pressure contribution, which is dropped in the following for brevity, yet is maintained in the numerical code when coupling to an atmospheric model.

For Boussinesq models, it is common to approximate the surface pressure as p_s ≈ gρ₀η_i^y, rather than the hydrostatic form gρ_iη_i^y. With this approximation, surface pressure gradients arise solely from gradients in the free surface height. To maintain self-consistency with the hydrostatic baroclinic pressure field, while incurring only trivial computational expense, we maintain the hydrostatic form p_s = gρ_iη_i^y in the model.

b. General strategy for the time stepping

Manipulations of the budget (14) lead to the zonal velocity equation

_tu_kfυ_kxp_sρ_oG̃_k(18)

where G̃_k represents the forcing

View Expanded

Notably, the cell thicknesses h^u used for computing the terms in G_k are taken at baroclinic time τ, as are the other inviscid contributions such as pressure and advection.

For the barotropic mode, MOM time steps the vertically integrated transport U = Σ_k h_ku_k, as is common in shallow water models. Its evolution takes the form

View Expanded

where Eq. (18) was used for ∂_tu_k, the vertically integrated forcing is given by G = Σ_k h_kG_k with G_k defined in Eq. (19), and D = Σ_k h_k is the time-dependent ocean depth. Once the transport and ocean depth are updated, the updated barotropic velocity can be diagnosed through u(τ + Δτ) = U(τ + Δτ)/D(τ + Δτ).

c. Time stepping algorithm

The focus in this section is on time and depth discretization, with Fig. 2 summarizing the following algorithm. For purposes of brevity, the horizontal spatial discretization discussed in the previous subsection will not be exposed. Discrete baroclinic times and time steps will be denoted by the Greek τ and Δτ, respectively, whereas the barotropic analogs will use the Latin t and Δt.

1) Algorithm basics

Allowing the grid cell thicknesses to evolve introduces a fundamentally new element to the traditional algorithms relevant for constant cells in z-coordinate models (e.g., Bryan 1969; Semtner 1974; Cox 1984; Killworth et al. 1991; Dukowicz and Smith 1994). However, since we impose positive cell thicknesses for all cells, including the surface, modifications to the constant cell approach should be relatively modest. That is a goal of the proposed algorithm.

The basic idea is directly analogous to the approach of Bryan (1969), Semtner (1974), and Cox (1984). That is, the velocity at an arbitrary depth level k and baroclinic time τ′ = τ + Δτ is split into two components:

u_kτB_kmτu_mτδ_kmB_kmτu_mτû_kττuττ(22)

This equation is an identity valid for any baroclinic times τ′ and τ, with τ′ = τ + Δτ chosen in the following. The utility of the identity relies on the form of the “baroclinicity operator” used to split the velocity field

B_kmτδ_kmDτ⁻¹h^u_mτ(23)

where δ_km is the Kronecker delta, summation over the repeated vertical level index m is implied, and

View Expanded

is the ocean depth at baroclinic time τ over a column of velocity points, with D_o the resting ocean depth.

This baroclinicity operator projects out the depth-independent part of a field and keeps its approximate baroclinic portion, where the projection is based on the distribution of cell thicknesses at time τ. Introduction of two baroclinic time labels τ and τ′ to Eq. (22) is necessitated by the freedom afforded the ocean depth to change in time. That is, τ′ represents the baroclinic time of the full velocity field u_k(τ′), whereas τ represents the time used to define the baroclinicity operator B_km(τ) and which defines the split.

If the split introduced in Eq. (22) is successful, û_k(τ, τ′) will evolve on a slow timescale, Δτ, and u(τ, τ′) will evolve on the fast timescale, Δt = 2N⁻¹Δτ, with N determined by the ratio of external to internal gravity wave speeds. The method therefore proceeds by separately updating û_k(τ, τ′) and u(τ, τ′) by exploiting the timescale split. Upon doing so, the right-hand side of the identity (22) will be specified, hence allowing for an update of the full velocity field via

u_kττû_kτ, ττuτ, ττ(25)

Upon updating the velocity cell thickness to h^u(τ + Δτ), the baroclinic and barotropic velocities û_k(τ + Δτ) and u(τ + Δτ) can then be diagnosed. The following subsections detail this approach.

3) Computing u(τ, τ + Δτ)

The computation of u(τ, τ + Δτ) constitutes a most crucial part of any free surface algorithm, since details largely determine the stability and conservative aspects of the scheme. The following approach, which uses a time averaging over barotropic time steps integrated from τ to τ + 2Δτ, is commonly used in explicit free surface methods. However, its details differ somewhat from other approaches, and so prompts a reasonably thorough presentation.

The barotropic “subcycling” uses a forward–backward time stepping scheme, also used by KSWP (see also, e.g., Haltiner and Williams 1980). In this approach, surface height is first time stepped using the small barotropic time step Δt:

View Expanded

The asterisk is used to denote intermediate values of the barotropic fields, each of which are updated on the barotropic time steps. The time label τ on η∗ and U∗ denotes the baroclinic time at which the vertically integrated forcing G(τ), the tracer fields, and the total depth of the ocean D(τ) are held for the duration of the barotropic time stepping from τ to τ + 2Δτ. This is also the time that sets the barotropic time steps via

t_nτnt,(29)

with n ∈ [0, N]. For stability purposes, we found it important to take the initial condition η∗(τ, t_n=0) as the time average η∗(τ) computed from the previous barotropic integration taking place over τ − Δτ to τ + Δτ [time averaging is defined by Eq. (32) discussed below]. Note that it is assumed that the freshwater flux q_w is constant over the small barotropic time steps, since the hydrological fluxes are typically updated on each baroclinic time step.

Having the surface height η∗ updated to the new barotropic time step allows for an update of the transport

View Expanded

where p̃^∗_s(τ, t_n+1) = gη∗(τ, t_n+1)ρ(τ)/ρ_o is the surface pressure normalized by the Boussinesq density. The Coriolis force is computed using a Crank–Nicholson semi-implicit time stepping scheme, which was also used by KSWP (see also Haltiner and Williams 1980).

After N barotropic time steps, the vertical transport and surface height are time averaged to produce

View Expanded

The time-averaged fields are centered on the baroclinic time step τ + Δτ, so long as N is an even integer. The time-averaged surface height η∗(τ + Δτ) is used to initialize the next suite of barotropic integrations from τ + Δτ to τ + 3Δτ, and U∗(τ + Δτ) is used to update u(τ, τ + Δτ) via

uτ, ττUττDτ(33)

Having u(τ, τ + Δτ) then allows for the full velocity field u_k(τ + Δτ) to be updated according to Eq. (25).

4) Computing η(τ + Δτ) and u(τ + Δτ)

Although the updated full velocity field u_k(τ + Δτ) is now known, the updated surface height η(τ + Δτ) and barotropic velocity u(τ + Δτ) = U(τ + Δτ)/D(τ + Δτ) remain to be determined. A number of options were tried, with the following two approaches initially attempted. (a) Stop the barotropic subcycling at τ + Δτ and use the unaveraged barotropic fields:

View Expanded

This choice was originally used by KSWP, yet we, and others such as the OCCAM group (see Webb et al. 1998), have found it to result in a very unstable algorithm and so it is not suitable. (b) Use of the time-averaged barotropic fields given by Eqs. (31) and (32),

View Expanded

allows for a substantial improvement in stability, and it is for this reason that OCCAM chooses this approach. However, this choice is not self-consistent in the sense that U(τ + Δτ) so defined will not generally be equivalent to Σ_k h^u_k(τ + Δτ)u_k(τ + Δτ) due to time-dependent thicknesses. Additionally, use of the time average for the surface height precludes compatibility between the tracer and volume budgets as discussed in section 2c, and it also does not allow for a specification of a discretely conserved total tracer as described in section 2d. Therefore, we propose the following alternative approach that satisfies our goals.

We first update the thicknesses, and then diagnose the integrated transport. To do so, compatibility as discussed in section 2c indicates that since the tracer concentration is time stepped with a leapfrog scheme, so too must the surface height budget:

View Expanded

As the surface height generally evolves on a fast timescale, use of this proposed baroclinic leapfrog time step is generally unstable. To maintain stability, as well as general smoothness of the free surface field, it has been found necessary to use the time-averaged surface height η∗(τ − Δτ) rather than a more traditional Robert filtered height. Doing so does not effect the compatibility condition, and so has been found suitable. Using this “big leapfrog” step to define η(τ + Δτ) then allows for an update of the surface tracer and velocity cell thicknesses h^t_k(τ + Δτ) and h^u_k(τ + Δτ).

Given the updated cell thicknesses, the updated transport U(τ + Δτ) can be diagnosed

View Expanded

thus ensuring self-consistency between the updated full velocity u_k(τ + Δτ) and the updated vertically integrated velocity. Finally, to complete the time stepping, the updated barotropic velocity is diagnosed through

uττUττDττ(40)

where D(τ + Δτ) = Σ h^u_k(τ + Δτ) is the new depth of a velocity cell column. The updated baroclinic velocity follows from its definition û_k(τ + Δτ) = u_k(τ + Δτ) − u(τ + Δτ). Note that a similar reconciliation or readjustment of the velocities is discussed by Hallberg (1995, p. 221) for use in his isopycnal model.

d. Comments on the algorithm

7) Vertical velocities

Vertical velocities are diagnosed at baroclinic time steps using volume conservation within a grid cell. In particular, volume conservation over a surface cell indicates that the vertical velocity at the bottom face of this cell arises from the horizontal convergence of volume in this cell, time tendencies in the thickness of this cell, and volume passing across the top face from freshwater fluxes.

It is useful to see precisely how these velocities are determined. For this purpose, consider a vertical integral of the continuity equation ∇ · u + ∂_zw = 0 over a rectangular top model grid cell:

View Expanded

Use of the surface kinematic boundary condition

_tuη∇ηwηq_w(42)

and Leibnitz’s rule with ∇z₁ = 0, renders

View Expanded

where the last step used the vertically integrated continuity equation (1). This exact expression for w_k=1 is approximated in the model with a discretized version of

w_k=1∇U∇hu(44)

where h = η + |z₁| is the surface cell thickness and the horizontal velocity u is that in the surface cell k = 1. Notably, unlike most z-coordinate free surface models, there has been no linearization of the surface kinematic boundary condition made by dropping the surface height advection u(η) · ∇η.

Given an expression for the convergence −∇ · U, diagnosing w_k=1 in this manner allows for the remaining interior vertical velocities to be successively found through further integration of the continuity equation downward through a vertical column. On a B grid, −∇ · U is centered on a tracer point. Hence, Eq. (44) yields the vertical velocity on the bottom face of the surface tracer cell. The vertical velocity on the surrounding velocity cells is constructed as a volume conserving average of the surrounding tracer cell vertical velocities.

In MOM, the bottom of the ocean on tracer cells is a flat surface, representing the “lopped off” surfaces of topography.² Hence, the vertical velocity must vanish at this location. A self-consistency check on how accurately the model’s numerics conserve volume amounts to testing how well this property is satisfied when integrating downward from the ocean surface, starting from the vertical velocity given by Eq. (44). Adding freshwater to the model provides a nontrivial test of these properties. The present scheme produces zero vertical velocities at the ocean bottom on tracer cells, to within computer roundoff, regardless of the topography or surface forcing.

a. Free surface compared to rigid lid

We start by providing a direct comparison between the explicit free surface and the rigid-lid streamfunction method in the sector model. Each experiment used the same tracer time step of 1 day and the same baroclinic time step of 1 h. The rigid-lid experiment used a 1 h time step for the streamfunction, whereas the free surface used 300 s for the barotropic time step. With the time averaging used with the free surface method, there are a total of 24 barotropic time steps integrated for each baroclinic time step. Both the free surface and rigid-lid methods used a virtual salt flux and zero freshwater forcing.

The experiments were run for 4000 tracer years, at which point both reached a quasi-equilibrium. When run on a single processor on GFDL’s Cray T90, the free surface method took a few percent longer than the rigid-lid method. Although this comparison is a function of model configuration and computer details, as well as elliptic solver algorithm, the key point is that such a comparison is impressive for the free surface model, since the rigid-lid model is ideally suited for such a simple configuration. That is, the flat-bottom, time-independent forcing, and absence of mesoscale eddies means that the number of elliptic solver scans can be quite low, often requiring only a single scan to update the streamfunction.

More realistic bottom topography, surface forcing, and mesoscale eddies will greatly increase the number of elliptic solver scans required by the rigid-lid model. In practice, more realistic situations often prompt one to greatly reduce or loosen the criteria used for determining a solution to the elliptic problem. Indeed, in some cases it is difficult to argue that a “solution” has even been found. In contrast, the ratio of barotropic to baroclinic wave speeds is independent of model resolution and surface forcing. Rather, it is determined by the bottom topography and stratification. Consequently, for the explicit free surface method, there is no issue regarding convergence to a solution, as convergence is guaranteed by construction.

As with KSWP and Dukowicz and Smith (1994), we check numerical integrity of the free surface solution by comparing with the more highly tested rigid-lid results. For example, Fig. 3 provides a sample of the free surface solution; the rigid-lid results are nearly identical. Since the vertical velocity at the ocean surface vanishes at equilibrium in a constant forced, coarse-resolution free surface model without freshwater forcing, the two methods should indeed provide nearly identical answers. A detailed comparison (not shown) reveals negligible differences for all aspects of the simulations. Other model configurations also have been run, with similar comparisons.

We conclude from these tests that the explicit free surface method maintains the convenience of the rigid lid for purposes of idealized coarse-resolution modeling. The solutions are likewise nearly the same when run under the same forcing. Both of these conclusions have remained valid with more realistic climate model experiments now routinely run at GFDL.

b. Goldsbrough–Stommel circulation

As discussed by Huang (1993), the Goldsbrough–Stommel circulation arises from freshwater forcing, and this circulation is absent in traditional rigid-lid models that use the virtual salt flux. Figure 4 illustrates this circulation when run with the free surface for 4000 yr. The freshwater forcing, as used by Huang, is given by

View Expanded

where q⁰_w = 1 m yr⁻¹ and ϕ_N and ϕ_S are the northern and southern latitudinal boundaries, respectively. The domain integral of this water flux vanishes. There is no wind forcing nor temperature forcing, and temperature is uniform. The barotropic circulation is quite weak and is comparable to that obtained by Huang. The baroclinic circulation, as documented through the overturning streamfunction, is strong and saline direct, yet somewhat weaker than that of Huang, perhaps because of our use of half the vertical diffusivity. It remains unclear how important this circulation is for realistic climate simulations, since it is often easily masked in the presence of wind and heat forcing. Regardless, its presence is a natural consequence of the use of freshwater forcing with the free surface method.

When integrating the model to reach the solution in Fig. 4, we used 1-day tracer and 1-h baroclinic velocity time steps as in the previous experiment. Although neither salt nor water were added to the model, the total salt does not remain precisely constant during the integration. The reason, as discussed in section 2, is that the tracer and baroclinic momentum time steps were unequal. The consequences on the equilibrium solution reached at 4000 yr have not been assessed. Doing so requires rerunning the experiment with equal tracer and baroclinic time steps.

c. Marsigli problem

The purpose of this section is to directly compare the new free surface method to that of KSWP. For this purpose, we consider an unforced baroclinic adjustment problem motivated from the system first discussed for the Bosporus by Marsigli in 1681 (see Gill 1982, p. 96).

The experimental details are given in the caption to Fig. 5. A strong baroclinic pressure gradient arising from a salinity front rapidly piles a sea level difference of about 40 cm between the two basins. After about two model days, barotropic and average baroclinic pressure gradients balance each other, and the sea level difference becomes roughly time independent. A weak cross-channel geostrophic circulation persists that prevents a salinity exchange between both basins. Although the setup appears artificial, a similar balance can be observed in the Baltic Sea during periods of weak wind forcing. Because the sea level changes rapidly and there is a strong interbasin salinity gradient, this adjustment problem provides a severe test for how well the model conserves total salt.

Figure 5 shows the sea level and barotropic current after 1 day using the nonlinear free surface method. The current between the basins is deflected to the right by the Coriolis force. There is a strong dipole sitting on top of the sill, with a high to the southwest and a low to the northeast. The barotropic flow at the outer edges of the subbasins is directed northward in both basins, which indicates that this current is the result of barotropic Kelvin waves encircling the subbasins. Such waves, and the remaining currents, cannot be represented in a rigid-lid model.

In general, the solution for the sea level, velocity, temperature, and salinity differ only slightly between the KSWP and the new approach during the adjustment. The close agreement of these features provides an important positive assessment of the new method. In particular, it indicates that the time averaging approach used here does not overly damp the barotropic waves, at least as compared to the KSWP algorithm.

As there are no surface water fluxes, the total volume of the ocean domain should remain constant, as indeed it does to within computer roundoff (not shown). Likewise, total salt and domain-averaged salinity (total salt divided by total volume) should remain constant, where total salt is computed via Eq. (10). An accounting of the domain-averaged salinity is shown in Fig. 6. Shown are four time series over a 3-month period that track how salinity deviates from its initial value.

The time series l₁ results from running the free surface of KSWP, with a constant cell thickness in the tracer and baroclinic velocity equations. The tracer and baroclinic time steps are both 240 s and the barotropic time step is 60 s. During the first two model days the sea level in the western basin is decreasing rapidly but the sea level in the eastern basin is increasing. Since the upper model boundary with the KSWP scheme is at z = 0, saline water is entering the model area in the western basin but brackish water leaves the model area through the level z = 0 in the eastern basin. Consequently, the domain-averaged salinity in the model area is increasing. As long as the sea surface salinity is uniform, the salt gain in the western basin and the salt loss in the eastern basin would be canceled exactly by a sea level variation in the opposite direction. However, such cancellation is not general, and so nonconservation of total salt is the norm.

Since the sea surface height is known, the salt between the upper model boundary at z = 0 and the sea level z = η can be included in the diagnostic for domain tracer content. As an example, the time series l₂ employs exactly the same linearized free surface scheme as l₁ but uses this improved tracer diagnostic. Compared with l₁, the conservation of salt is better; however, there is a clear trend in the domain-averaged salinity. This gain of salt stems from undulations of the sea level in correspondence with a varying sea surface salinity.

The time series n₁, which shows minor deviation from zero, uses the new free surface discussed in this paper with tracer and baroclinic time steps both equal to 240 s, with a 60-s barotropic time step. The conservation of tracer is improved substantially when compared with the linearized free surface results. The deviation from zero is less than 10⁻⁵ psu.

The time series n₂ uses the same free surface, yet with unequal baroclinic and tracer time steps, with a tracer time step of 720 s, and baroclinic time step of 240 s. As discussed in section 2, tracer conservation is not ensured in this case, as is indeed shown by the roughly 0.04 psu salinity drift.

d. Quantifying local versus global conservation

The Marsigli example provided an illustration of the global conservation properties of the new algorithm when using a discretization that maintains local compatibility between the volume and tracer budgets. As mentioned in section 2e, there is an alternative approach that is available for maintaining more exact global conservation properties, yet at the cost of sacrificing the local compatibility. We present here an example that compares the two approaches and concludes that the scheme that performs superior for global conservation is preferable for climate modeling purposes.

We consider a coarse-resolution global model run with two idealized conditions and using two different forms for the discretization of the T∂_tη term appearing in the tracer concentration budget (6). Discretization (a) uses T(τ)[η(τ + Δτ) − η∗(τ − Δτ)], as used in the Marsigli example just presented. This form provides for local self-consistency between the tracer and volume budgets as discussed when deriving Eq. (38). However, as discussed in section 2d, it will not provide for exact global conservation. Discretization (b) drops the time average on the lagged surface height and instead uses T(τ)[η(τ + Δτ) − η(τ − Δτ)]. This expression leads to exact global conservation in the case where there is no Robert filtering on the tracer, yet it is not locally self-consistent with a volume budget that uses a time-filtered surface height η∗(τ − Δτ) for stability.

The global model has roughly three degrees of horizontal resolution, uses 15 vertical levels down to 5000-m depth, and is run with a 30-min time step for the tracer and baroclinic momentum and 30-s barotropic time step. The first experiment initialized the ocean at rest with constant 25°C water and then added a climatological wind stress yet no heat fluxes. As presented in section 2c, the solution should maintain constant 25°C water everywhere. For an indefinite integration period, discretization a indeed remains precisely at 25°C. In contrast, discretization b shows water after 30 days with a temperature of 24.999 95°C. Although not exact, this is quite a small deviation even when extrapolated to integrations of centuries to millennia.

The second experiment initialized the temperature field with a zonally averaged version of the Levitus (1982) analysis. The surface flux consisted of the same climatological wind stress as the previous example as well as a uniform and constant 10 W m⁻² surface heat flux. We assess here whether the heat input through the ocean surface is equivalent to the heat absorbed by the ocean as deduced by measuring the ocean’s total heat content via Eq. (10). We focus on a single model day since this is a typical time period over which winds and surface tracer flux are held fixed in coupled climate models.

After 1 day for discretization a, the heat input through the ocean surface minus the change in ocean heat content was equivalent to a spatially averaged error over the globe of −0.5 W m⁻². Note that over the first half of the day, the error was −1.6 W m⁻², which diminished to −0.37 W m⁻² after 2 days. These are nontrivial errors, upward of 10%–20% of the imposed heating from the atmosphere. In contrast, discretization b, along with Robert filtering on the tracer, reduced the error from −0.5 W m⁻² to a negligible −0.003 W m⁻². Removing the Robert filter brings the error to 10⁻⁹ W m⁻², which arises from computer roundoff.

Both solution methods showed negligible qualitative differences for the structure of the surface height over the 30-day integration period. However, given the small errors incurred by discretization b for both the first and second set of experiments, we conclude that it is a preferable approach for longer integrations where conservation is key.

a. Fundamental considerations

The dominant feature of current supercomputing technology is the large and rapidly growing disparity between processor clock speeds and the memory bandwidth and latency. The rate at which processors are able to execute their instructions upon data far outstrips the ability of memory to deliver data to the processor. All current research into supercomputing architecture (see, e.g., Culler and Singh 1998) is directed at resolving this problem.

Traditional supercomputers, such as the Cray vector machines, are largely dependent on prohibitively expensive custom memory to deliver data at a rate sufficient to keep vector pipelines busy. The microprocessor-based computers currently being produced rely instead on commodity memory, but implement a deep memory hierarchy, where multiple levels of smaller and faster data caches are placed between the main memory and the processor. For performance comparable to vector systems, scalable cache-based microprocessor supercomputers achieve speedup through massive parallel partitioning of the problem, where the problem is distributed among many processors working concurrently and exchanging data on a network. This architectural model leads to measures of algorithm performance substantially different from the vector model. Notably, the following issues become central.

1. Temporal locality. Given the relatively high cost of memory access, algorithms that have a high rate of data reuse are preferable. The computational intensity, defined as the ratio of floating-point operations per memory request, is required to be as high as possible. The match between the size of the data subset allotted to a processor, and the size of its cache, is a key element in this aspect of performance.

2. Spatial locality. In the interests of minimizing communication among processors, algorithms that maintain spatial locality of data references are most useful.

3. Communication profile. The communication performance of the interprocessor hardware interconnect is measured in terms of its latency (startup time per message) and bandwidth (the inverse of the transmission rate). Communication latency can be a cause for concern on most commercially available machines today, barring a few exceptions such as the Cray T3E. The amount of data that needs to be transferred between processors is a function of the spatial locality of the algorithm and the problem size. Algorithms that can accomplish the maximum data transfer in the fewest possible messages are preferable.

b. Algorithms available in MOM 3

As mentioned in the introduction, there are three basic algorithms in MOM 3 for solving the barotropic mode:the rigid-lid streamfunction of Bryan (1969), with modifications to the rigid-lid surface pressure method of Smith et al. (1992) and Dukowicz et al. (1993); the implicit free surface method of Dukowicz and Smith (1994); and the explicit free surface method of KSWP as well as that discussed here.

The rigid-lid model, although parallelized (Redler et al. 1998), has not been the focus of recent algorithm research at GFDL, largely because of the physical issues previously raised in this paper as well as problems with global data dependencies related to island integrals [see Bryan (1969) as well as Smith et al. 1992 for discussion].

The implicit free surface method, in contrast, has a relatively long history of use on parallel machines. This algorithm solves the barotropic system implicitly in time, hence allowing for the barotropic mode time step to be lengthened to that of the baroclinic mode. As such, there is no subcycling of the barotropic mode as required with the explicit approach. In so doing, however, the algorithm requires the solution of an elliptic problem.

The elliptic solvers used in the implicit free surface method are variants of the basic iterative Jacobi solver, where methods are used to accelerate the rate of convergence. For example, the conjugate gradient (CG) solver computes the optimal vector along which to converge. Computation of this vector involves a global sum across the entire domain; hence, the data dependencies are global, and involve global communication at each step in the solver iteration. It is observed in MOM that the number of iterations for the CG method is ∼100 for a 1° model with static boundary conditions, and goes up with increasing resolution. Additionally, the iteration count is highly dependent on the model’s representation of bottom topography and surface forcing, with more realistic topography and time-dependent surface forcing generally requiring much larger iteration counts. Parallel implementation of the CG requires communication at each iteration. Other, faster methods, with better parallel performance, such as multigrid methods, are known to exist, but also require increased steps as the resolution goes up.

The explicit free surface has improved data locality, since the computations involve only local derivatives and no elliptic problem. The ratio of the baroclinic to the barotropic time step is given by the ratio of the phase speeds of the barotropic gravity wave to the first baroclinic wave. This ratio is determined by ocean depth and stratification and, so, is independent of grid resolution. Hence, the method’s computational performance is generally independent of grid resolution.

c. Examples of performance on scalable systems

MOM 3 is parallelized in one dimension, along latitude rows only. Scalability is thus measured in terms of latitude rows per processor. We show scaling results from a Cray T3E at two different resolutions using the explicit free surface algorithm documented here. Table 1 shows scaling results for a Southern Hemisphere configuration using a 4° Mercator grid. The superlinear scaling above seven rows per processing element (PE) is a result of arrays becoming small enough to fit in cache. Table 2 shows results for the same domain at 1° resolution.

The results for the 1° model are more appropriate for current and future needs of parallel ocean modeling. They indicate that one can increase the number of processors in MOM 3 until 8 rows/PE without significant loss of scalability. Note that with 4 rows/PE, scalability is 80%, whereas it has been found to be about 60% for MOM’s version of the implicit free surface method. Furthermore, as noted previously, absolute times for the implicit free surface method are highly dependent on the conjugate gradient iteration count needed to achieve convergence to the subjectively chosen tolerance level.