The Spectral Element Atmosphere Model (SEAM): High-Resolution Parallel Computation and Localized Resolution of Regional Dynamics

Aimé Fournier Department of Meteorology, College of Computer, Mathematical, and Physical Sciences, University of Maryland, College Park, College Park, Maryland

Search for other papers by Aimé Fournier in
Current site
Google Scholar
PubMed
Close
,
Mark A. Taylor Los Alamos National Laboratory, Los Alamos, New Mexico

Search for other papers by Mark A. Taylor in
Current site
Google Scholar
PubMed
Close
, and
Joseph J. Tribbia National Center for Atmospheric Research,* Boulder, Colorado

Search for other papers by Joseph J. Tribbia in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Fast, accurate computation of geophysical fluid dynamics is often very challenging. This is due to the complexity of the PDEs themselves and their initial and boundary conditions. There are several practical advantages to using a relatively new numerical method, the spectral-element method (SEM), over standard methods. SEM combines spectral-method high accuracy with the geometric flexibility and computational efficiency of finite-element methods.

This paper is intended to augment the few descriptions of SEM that aim at audiences besides numerical-methods specialists. Advantages of SEM with regard to flexibility, accuracy, and efficient parallel performance are explained, including sufficient details that readers may estimate the benefit of applying SEM to their own computations.

The spectral element atmosphere model (SEAM) is an application of SEM to solving the spherical shallow-water or primitive equations. SEAM simulated decaying Jovian atmospheric shallow-water turbulence up to resolution T1067, producing jets and vortices consistent with Rhines theory. SEAM validates the Held–Suarez primitive equations test case and exhibits excellent parallel performance. At T171L20, SEAM scales up to 292 million floating-point operations per second (Mflops) per processor (29% of supercomputer peak) on 32 Compaq ES40 processors (93% efficiency over using 1 processor), allocating 49 spectral elements/processor. At T533L20, SEAM scales up to 130 billion floating-point operations per second (Gflops) (8% of peak) and 9 wall clock minutes per model day on 1024 IBM POWER3 processors (48% efficiency over 16 processors), allocating 17 spectral elements per processor. Local element-mesh refinement with 300% stretching enables conformally embedding T480 within T53 resolution, inside a region containing 73% of the forcing but 6% of the area. Thereby the authors virtually reproduced a uniform-mesh T363 shallow-water computation, at 94% lower cost.

Corresponding author address: Dr. Aimé Fournier, NCAR, P.O. Box 3000, Boulder, CO 80307-3000. Email: fournier@ucar.edu

Abstract

Fast, accurate computation of geophysical fluid dynamics is often very challenging. This is due to the complexity of the PDEs themselves and their initial and boundary conditions. There are several practical advantages to using a relatively new numerical method, the spectral-element method (SEM), over standard methods. SEM combines spectral-method high accuracy with the geometric flexibility and computational efficiency of finite-element methods.

This paper is intended to augment the few descriptions of SEM that aim at audiences besides numerical-methods specialists. Advantages of SEM with regard to flexibility, accuracy, and efficient parallel performance are explained, including sufficient details that readers may estimate the benefit of applying SEM to their own computations.

The spectral element atmosphere model (SEAM) is an application of SEM to solving the spherical shallow-water or primitive equations. SEAM simulated decaying Jovian atmospheric shallow-water turbulence up to resolution T1067, producing jets and vortices consistent with Rhines theory. SEAM validates the Held–Suarez primitive equations test case and exhibits excellent parallel performance. At T171L20, SEAM scales up to 292 million floating-point operations per second (Mflops) per processor (29% of supercomputer peak) on 32 Compaq ES40 processors (93% efficiency over using 1 processor), allocating 49 spectral elements/processor. At T533L20, SEAM scales up to 130 billion floating-point operations per second (Gflops) (8% of peak) and 9 wall clock minutes per model day on 1024 IBM POWER3 processors (48% efficiency over 16 processors), allocating 17 spectral elements per processor. Local element-mesh refinement with 300% stretching enables conformally embedding T480 within T53 resolution, inside a region containing 73% of the forcing but 6% of the area. Thereby the authors virtually reproduced a uniform-mesh T363 shallow-water computation, at 94% lower cost.

Corresponding author address: Dr. Aimé Fournier, NCAR, P.O. Box 3000, Boulder, CO 80307-3000. Email: fournier@ucar.edu

Save
  • Alpert, B., G. Beylkin, D. Gines, and L. Vozovoi, 2002: Adaptive solution of partial differential equations in multiwavelet bases. J. Comput. Phys., 182 , 149190.

    • Search Google Scholar
    • Export Citation
  • Anderson, W. K., W. D. Gropp, D. K. Kaushik, D. E. Keyes, and B. F. Smith, 1999: Achieving high sustained performance in an unstructured mesh CFD application. Proc. Supercomputing, Portland, OR, IEEE, 1–13. [Available online at http://www.supercomp.org/sc99/proceedings/papers/anderson.pdf.].

    • Search Google Scholar
    • Export Citation
  • Baer, F., J. J. Tribbia, and M. Taylor, 2001: Global and regional atmospheric modelling using spectral elements. Fluid Mech. Appl., 61 , 8186.

    • Search Google Scholar
    • Export Citation
  • Barosan, I., F. N. van de Vosse, P. D. Anderson, and H. E. H. Meijer, cited 2001: hN mesh-refinement techniques 2D. Eindhoven University of Technology Internal Poster. [Available online at http://www.mate.tue.nl/mate/showemp.php/54.].

    • Search Google Scholar
    • Export Citation
  • Beylkin, G., J. M. Keiser, and L. Vozovoi, 1998: A new class of time discretization schemes for the solution of nonlinear PDES. J. Comput. Phys., 147 , 362387.

    • Search Google Scholar
    • Export Citation
  • Blackburn, H. M., 1998: Channel flow LES with spectral elements. Proc. 13th Australasian Fluid Mechanics Conf., Melbourne, Australia, Monash University, 989–992.

    • Search Google Scholar
    • Export Citation
  • Boyd, J. P., 2000: Chebyshev and Fourier Spectral Methods. 2d ed. Dover, 688 pp.

  • Canuto, C., M. Y. Hussaini, A. Quarteroni, and T. A. Zang, 1988: Spectral Methods in Fluid Dynamics. Springer-Verlag, 557 pp.

  • Canuto, C., A. Tabacco, and K. Urban, 2000: The wavelet element method. Part II: Realization and additional features in 2D and 3D. Appl. Comput. Harm. Anal., 8 () 123165.

    • Search Google Scholar
    • Export Citation
  • Capdeville, Y., C. Larmat, J. P. Vilotte, and J. P. Montagner, 2002: A new coupled spectral element and modal solution method for global seismology: A first application to the scattering induced by a plume-like anomaly. Geophys. Res. Lett.,29, 1318, doi:10.1029/2001GL013747.

    • Search Google Scholar
    • Export Citation
  • Chang, H. R., and H. N. Shirer, 1985: Compact spatial differencing techniques in numerical modeling. Mon. Wea. Rev., 113 , 409423.

  • Dahmen, W., P. Oswald, and A. J. Kurdila, 1997: Multiscale Wavelet Methods for Partial Differential Equations. Academic Press, 570 pp.

  • DAO, 2000: Algorithm theoretical basis document, Version 2.00. Data Assimilation Office Tech. Rep., NASA Goddard Space Flight Center, 33 pp.

    • Search Google Scholar
    • Export Citation
  • de Frutos, J., and J. Novo, 2000: A spectral element method for the Navier–Stokes equations with improved accuracy. SIAM J. Numer. Anal., 38 , 799819.

    • Search Google Scholar
    • Export Citation
  • Dixon, M., and K. Tan, 2003: Distributed solution of high-order compact difference schemes for multidimensional convection–diffusion equations. ICCSA03 Conference Proceedings, V. Kumar, M. L. Gavrilova, C. J. K. Tan, and P. L'Ecuyer, Eds., Lecture Notes in Computer Science, Vol. 2669, Springer-Verlag, 226–235.

    • Search Google Scholar
    • Export Citation
  • Feng, H., and C. Mavriplis, 2002: Adaptive spectral element simulations of thin premixed flame sheet deformations. J. Sci. Comput., 17 , 385395.

    • Search Google Scholar
    • Export Citation
  • Foster, I. T., and P. H. Worley, 1997: Parallel algorithms for the spectral transform method. SIAM J. Sci. Comput., 18 , 806837.

  • Foster, I. T., W. Gropp, and R. Stevens, 1992: The parallel scalability of the spectral transform method. Mon. Wea. Rev., 120 , 835850.

    • Search Google Scholar
    • Export Citation
  • Fournier, A., cited 2001: Adaptive spectral-element method for the shallow-water equations. [Available online at http://www.asp.ucar.edu/gtp/fournier/files/preprint/ccsm2001mini.pdf.].

    • Search Google Scholar
    • Export Citation
  • Fournier, A., M. A. Taylor, L. M. Polvani, and R. Saravanan, 2000a: Spectral element method. Part 2: Numerical simulations. Proc. Eighth Annual Conf., Montreal, QC, Canada, CFD Society of Canada, 181–188. [Available online at http://www.asp.ucar.edu/gtp/fournier/files/publications/AINu.pdf.].

    • Search Google Scholar
    • Export Citation
  • Fournier, A., M. A. Taylor, and J. Tribbia, 2000b: Spectral element method. Part 1: Numerical algorithms. Proc. Eighth Annual Conf., Montreal, QC, Canada, CFD Society of Canada, 173–180. [Available online at http://www.asp.ucar.edu/gtp/fournier/files/publications/NuSi.pdf.].

    • Search Google Scholar
    • Export Citation
  • Fournier, A., B. Beylkin, and V. Cheruvu, 2003: Multiresolution adaptive space refinement in geophysical fluid dynamics simulation. Adaptive Mesh Refinement—Theory and Application, T. Plewa, T. Linde, and V. G. Weirs, Eds., Lecture Notes in Computational Sciences and Engineering, Springer-Verlag, in press.

    • Search Google Scholar
    • Export Citation
  • Fox-Rabinovitz, M. S., L. L. Takacs, R. C. Govindaraju, and M. J. Suarez, 2001: A variable-resolution stretched-grid general circulation model: Regional climate simulation. Mon. Wea. Rev., 129 , 453469.

    • Search Google Scholar
    • Export Citation
  • Funaro, D., 1997: Spectral Elements for Transport-Dominated Equations. Springer-Verlag, 211 pp.

  • Giraldo, F. X., and T. E. Rosmond, 2004: A scalable spectral element Eulerian atmospheric model (SEE-AM) for NWP: Dynamical core tests. Mon. Wea. Rev., 132 , 133153.

    • Search Google Scholar
    • Export Citation
  • Gottlieb, D., and S. A. Orszag, 1977: Numerical Analysis of Spectral Methods: Theory and Applications. SIAM, 170 pp.

  • Gropp, W. D., D. K. Kaushik, D. E. Keyes, and B. F. Smith, 2001: High-performance parallel implicit CFD. Parallel Comput., 27 , 337362.

    • Search Google Scholar
    • Export Citation
  • Haidvogel, D. B., and A. Beckmann, 1999: Numerical Ocean Circulation Modeling. Imperial College Press, 300 pp.

  • Hamrud, M., S. Saarinen, and D. Salmond, 2003: Implementation of the IFS on a highly parallel scaler system. Realizing TeraComputing—Tenth Workshop on the Use of High Performance Computing in Meteorology, W. Zwieflhofer and N. Kreitz, Eds., World Scientific, 74– 87. [Available online at http://www.ecmwf.int/publications/library/ecpublications/proceedings/high_performance_computing_2002.].

    • Search Google Scholar
    • Export Citation
  • Held, I. M., and M. J. Suarez, 1994: A proposal for the intercomparison of the dynamical cores of atmospheric general circulation models. Bull. Amer. Meteor. Soc., 75 , 18251830.

    • Search Google Scholar
    • Export Citation
  • Henderson, R. D., 1999: Dynamic refinement algorithms for spectral element methods. Comput. Methods Appl. Mech. Eng., 175 , 395411.

  • Hoppe, R. H. W., and E. M. Nash, 2002: A combined spectral element/finite element approach to the numerical solution of a nonlinear evolution equation describing amorphous surface growth of thin films. J. Numer. Math., 10 () 127136.

    • Search Google Scholar
    • Export Citation
  • Iacono, R., M. V. Struglia, C. Ronchi, and S. Nicastro, 1999: High-resolution simulations of freely decaying shallow-water turbulence on a rotating sphere. Nuovo Cimento, 22C , 813821.

    • Search Google Scholar
    • Export Citation
  • Isaacson, E., and H. B. Keller, 1994: Analysis of Numerical Methods. Dover, 541 pp.

  • Iselin, J. P., J. M. Prusa, and W. J. Gutowski, 2002: Dynamic grid adaptation using the MPDATA scheme. Mon. Wea. Rev., 130 , 10261039.

    • Search Google Scholar
    • Export Citation
  • Iskandarani, M., D. B. Haidvogel, J. C. Levin, E. Curchitser, and C. A. Edwards, 2002: Multiscale geophysical modeling using the spectral element method. Comput. Sci. Eng., 4 (5) 4248.

    • Search Google Scholar
    • Export Citation
  • Karniadakis, G. E., and S. J. Sherwin, 1999: Spectral/hp Element Methods for Computational Fluid Dynamics. Oxford University Press, 404 pp.

    • Search Google Scholar
    • Export Citation
  • Komatitsch, D., J. Ritsema, and J. Tromp, 2002: The spectral-element method, Beowulf computing, and global seismology. Science, 298 , 17371742.

    • Search Google Scholar
    • Export Citation
  • Lin, S. J., and R. B. Rood, 1998: A flux-form semi-Lagrangian general circulation model with a Lagrangian control-volume vertical coordinate. Proc. Rossby-100 Symp., Stockholm, Sweden, Dept. of Meteorology, Stockholm University.

    • Search Google Scholar
    • Export Citation
  • Lin, S. J., and R. B. Rood, 1999: Development of the joint NASA/NCAR general circulation model. Preprints, 13th Conf. on Numerical Weather Prediction, Denver, CO, Amer. Meteor. Soc., 115–119.

    • Search Google Scholar
    • Export Citation
  • Maday, Y., and A. T. Patera, 1989: Spectral element methods for the incompressible Navier–Stokes equations. State-of-the-Art Surveys on Computational Mechanics, A. K. Noor and J. T. Oden, Eds., American Society of Mechanical Engineers, 71–143.

    • Search Google Scholar
    • Export Citation
  • Mashkovich, S. A., 1994: Application of a continuous dynamic grid adaptation technique to numerical weather forecasting. Russ. Meteor. Hydrol., 11 , 16.

    • Search Google Scholar
    • Export Citation
  • Mathews, J., and R. L. Walker, 1971: Mathematical Methods of Physics. 2d ed. Addison-Wesley, 501 pp.

  • McGregor, J. L., 1999: Regional modeling at CAR: Recent developments. Parallel computing in meteorology and oceanography, BMRC Research Rep. 75, Bureau of Meteorology Research Centre, 43–48.

    • Search Google Scholar
    • Export Citation
  • Morton, K. W., and D. F. Mayers, 1994: Numerical Solution of Partial Differential Equations: An Introduction. Cambridge University Press, 239 pp.

    • Search Google Scholar
    • Export Citation
  • Patera, A. T., 1984: A spectral element method for fluid dynamics: Laminar flow in a channel expansion. J. Comput. Phys., 54 , 468488.

    • Search Google Scholar
    • Export Citation
  • Pedlosky, J., 1987: Geophysical Fluid Dynamics. 2d ed. Springer-Verlag, 710 pp.

  • Prusa, J. M., P. K. Smolarkiewicz, and A. A. Wyszogrodzki, 2001: Simulations of gravity wave induced turbulence using 512 pe Cray T3E. Int. J. Appl. Math. Comput. Sci., 11 , 883897.

    • Search Google Scholar
    • Export Citation
  • Purser, R. J., and M. Rančić, 1998: Smooth quasi-homogeneous gridding of the sphere. Quart. J. Roy. Meteor. Soc., 124 , 637647.

  • Randall, D. A., T. D. Ringler, R. P. Heikes, P. Jones, and J. Baumgardner, 2002: Climate modeling with spherical geodesic grids. Comput. Sci. Eng., 4 (5) 3241.

    • Search Google Scholar
    • Export Citation
  • Sawyer, W., 2001: Challenges in the dynamics of atmospheric modelling. Seminar at the Institut für Angewandte Mathematik II, Freiberg, Germany, TU Bergakademie Freiberg, 1–37. [Available online at http://www.iac.ethz.ch/staff/sawyer/freibergJun2001/sld001.htm.].

    • Search Google Scholar
    • Export Citation
  • Schaffer, D. S., and M. J. Suárez, 2000: Design and performance analysis of a massively parallel atmospheric general circulation model. Sci. Programm., 8 , 4957.

    • Search Google Scholar
    • Export Citation
  • Shingu, S., and Coauthors, 2002: A 26.58 Tflops global atmospheric simulation with the spectral transform method on the Earth Simulator. Supercomputing 2002: From Terabytes to Insights, R. Lucas, Ed., IEEE Computer Society and ACM SIGARCH. [Available online at http://sc-2002.org/paperpdfs/pap.pap331.pdf.].

    • Search Google Scholar
    • Export Citation
  • Spotz, W. F., M. A. Taylor, and P. N. Swarztrauber, 1998: Fast shallow-water equation solvers in latitude–longitude coordinates. J. Comput. Phys., 145 , 432444.

    • Search Google Scholar
    • Export Citation
  • Stevens, D. E., A. S. Almgren, and J. B. Bell, 1999: Adaptive simulations of trade cumulus convection. University of California Tech. Rep. UCRL-JC-133201, 23 pp.

    • Search Google Scholar
    • Export Citation
  • Tanguay, M., A. Simard, and A. Staniforth, 1989: A three-dimensional semi-Lagrangian scheme for the Canadian regional finite-element forecast model. Mon. Wea. Rev., 117 , 18611871.

    • Search Google Scholar
    • Export Citation
  • Taylor, M., J. Tribbia, and M. Iskandarani, 1997: The spectral element method for the shallow water equations on the sphere. J. Comput. Phys., 130 , 92108.

    • Search Google Scholar
    • Export Citation
  • Taylor, M., R. Loft, and J. Tribbia, 1998: Performance of a spectral element atmospheric model (SEAM) on the HP Exemplar SPP2000. NCAR Tech. Rep. TN-439 + EDD, 16 pp.

    • Search Google Scholar
    • Export Citation
  • Thomas, S. J., and R. D. Loft, 2002: Semi-implicit spectral element atmospheric model. J. Sci. Comput., 17 , 339350.

  • Thomas, S. J., R. Loft, W. F. Spotz, and A. Fournier, 2000: Semi-implicit scheme for the Spectral Element Atmospheric Model. Proc. Eighth Annual Conf., Montreal, QC, Canada, CFD Society of Canada, 231–238.

    • Search Google Scholar
    • Export Citation
  • Thomas, S. J., R. Loft, A. Fournier, and J. Tribbia, 2001: Parallel spectral element dynamical core for atmospheric general circulation models. Proc. Ninth Annual Conf., Waterloo, ON, Canada, CFD Society of Canada, 69–74.

    • Search Google Scholar
    • Export Citation
  • Wang, G., and N. W. Wereley, 2002: Spectral finite element analysis of sandwich beams with passive constrained layer damping. Trans. ASME, J. Vib. Acoust., 124 , 376386.

    • Search Google Scholar
    • Export Citation
  • Williamson, D. L., and J. M. Rosinski, 2000: Accuracy of reduced grid calculations. Quart. J. Roy. Meteor. Soc., 126 , 16191640.

  • Williamson, D. L., J. T. Kiehl, V. Ramanathan, R. E. Dickinson, and J. J. Hack, 1987: Description of the NCAR community climate model (CCM1). NCAR Tech. Rep. 285, 112 pp.

    • Search Google Scholar
    • Export Citation
  • Williamson, D. L., J. B. Drake, J. J. Hack, R. Jakob, and P. N. Swarztrauber, 1992: A standard test set for numerical approximations to the shallow water equations in spherical geometry. J. Comput. Phys., 102 , 211224.

    • Search Google Scholar
    • Export Citation
  • Worley, P. H., 2002: Performance studies using CCM/MP-2D. Oak Ridge National Laboratory Tech. Rep. [Available online at http://www.csm.ornl.gov/~worley/studies/ccm-mp-2d-platforms.html.].

    • Search Google Scholar
    • Export Citation
  • Yoden, S., K. Ishioka, Y-Y. Hayashi, and M. Yamada, 1999: A further experiment on two-dimensional decaying turbulence on a rotating sphere. Nuovo Cimento, 22C , 803812.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1160 764 50
PDF Downloads 296 61 3