Editorial Type:
Article
Article Type:
Research Article

A Dynamic Parallel Data-Computing Environment for Cross-Sensor Satellite Data Merger and Scientific Analysis

Andrew S. Jones Center for Geosciences/Atmospheric Research, Cooperative Institute for Research in the Atmosphere, Colorado State University, Fort Collins, Colorado

Search for other papers by Andrew S. Jones in
Current site
Google Scholar
PubMed Close
and
Thomas H. Vonder Haar Center for Geosciences/Atmospheric Research, Cooperative Institute for Research in the Atmosphere, Colorado State University, Fort Collins, Colorado

Search for other papers by Thomas H. Vonder Haar in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Sep 2002
DOI:
https://doi.org/10.1175/1520-0426(2002)019<1307:ADPDCE>2.0.CO;2
Page(s):
1307–1317
Restricted access

Abstract

The Data Processing and Error Analysis System (DPEAS) is a dynamic parallel data processing system for cross-sensor satellite data merger and analysis. Using a peer-to-peer methodology, DPEAS is able to distribute and simplify the near-real-time merger of multiple datasets from several sensors and satellite platforms. The organization of the system and its practical benefits for research and operational use are discussed. Using several simplifying constructs, the technical requirements placed upon the end user are minimized, while maintaining high scalability and reliability of the system. Thus, the system encourages the flow of research into operations by making the system environment mobile, rather than placing the transferability burden on the research development effort. Several reliability enhancements, such as cluster failover and automatic file replication services, are incorporated into the system. The technical aspects of the various feature implementations are discussed in detail. Future adaptation of this processing approach to new Internet standards is also discussed.

Corresponding author address: Dr. Andrew S. Jones, CSU/CIRA, Foothills Campus, Fort Collins, CO 80523. Email: jones@cira.colostate.edu

Abstract

The Data Processing and Error Analysis System (DPEAS) is a dynamic parallel data processing system for cross-sensor satellite data merger and analysis. Using a peer-to-peer methodology, DPEAS is able to distribute and simplify the near-real-time merger of multiple datasets from several sensors and satellite platforms. The organization of the system and its practical benefits for research and operational use are discussed. Using several simplifying constructs, the technical requirements placed upon the end user are minimized, while maintaining high scalability and reliability of the system. Thus, the system encourages the flow of research into operations by making the system environment mobile, rather than placing the transferability burden on the research development effort. Several reliability enhancements, such as cluster failover and automatic file replication services, are incorporated into the system. The technical aspects of the various feature implementations are discussed in detail. Future adaptation of this processing approach to new Internet standards is also discussed.

Corresponding author address: Dr. Andrew S. Jones, CSU/CIRA, Foothills Campus, Fort Collins, CO 80523. Email: jones@cira.colostate.edu

Save
Cover Journal of Atmospheric and Oceanic Technology
Journal of Atmospheric and Oceanic Technology

  • Adams, J. C., Brainerd W. S. , Martin J. T. , Smith B. T. , and Wagener J. L. , 1992: Fortran 90 Handbook: Complete ANSI/ISO Reference. McGraw-Hill, 740 pp.

    • Search Google Scholar
    • Export Citation

  • Asrar, G., and Ramapriyan H. K. , 1995: Data and information system for Mission to Planet Earth. Remote Sens. Rev., 13 , 1–25.

  • Avresky, D. R., and Kaeli D. R. , Eds.,. 1998: Fault-Tolerant Parallel and Distributed Systems. Kluwer Academic, 416 pp.

  • Botts, M. E., 1992: The state of scientific visualization with regard to the NASA EOS Mission to Planet Earth. Proc. Applied Information Systems Research Program (AISRP) Workshop II Meeting, Boulder, CO, NASA and Laboratory for Atmospheric and Space Physics, University of Colorado, E132–E139.

    • Search Google Scholar
    • Export Citation

  • Dongarra, J., Kacsuk P. , and Podhorszki N. , Eds.,. 2000: Lecture Notes in Computer Science, 1908: Recent Advances in Parallel Virtual Machine and Message Passing Interface. Springer Verlag, 364 pp.

    • Search Google Scholar
    • Export Citation

  • Eckel, B., 2000: Thinking in Java. Prentice Hall, 1128 pp.

  • Fortner, B., and Ilg D. , 1995: HDF-EOS primer for version 1 EOSDIS. Tech. Rep. 175-WP-001-001, 90 pp.

  • Gropp, W., Lusk E. , and Thakur R. , 2000: Using MPI-2: Advanced Features of the Message-Passing Interface. MIT Press, 382 pp.

  • Hibbard, W., and Santek D. , 1989: Visualizing large data sets in the earth sciences. Computer, 22 , 53–57.

  • Ilg, D., Wynne D. , Gejjagaraguppe R. , and Klein L. , 1999: HDF-EOS library user's guide for the ECS Project. Vol. 1: Overview and examples. Tech. Rep. 170-TP-500-001, Raytheon Systems Company, Maryland, 156 pp.

    • Search Google Scholar
    • Export Citation

  • Jones, A. S., Eis K. E. , and Vonder Haar T. H. , 1995: A method for multisensor–multispectral satellite data fusion. J. Atmos. Oceanic Technol., 12 , 739–754.

    • Crossref
    • Search Google Scholar
    • Export Citation

  • Kidder, S. Q., Goldberg M. D. , Zehr R. M. , DeMaria M. , Purdom J. F. W. , Velden C. S. , Grody N. C. , and Kusselson S. J. , 2000: Satellite analysis of tropical cyclones using the Advanced Microwave Sounding Unit (AMSU). Bull. Amer. Meteor. Soc., 81 , 1241–1260.

    • Crossref
    • Search Google Scholar
    • Export Citation

  • May, J. M., 2000: Parallel I/O for High Performance Computing. Morgan Kaufmann, 325 pp.

  • NCSA, 2000: HDF user's guide version 4.1r4. National Center for Supercomputing Applications, University of Illinois, Urbana–Champaign, 501 pp.

    • Search Google Scholar
    • Export Citation

  • NSSDC, 2001: CDF user's guide, version 2.7. National Space Science Data Center, NASA GSFC, Greenbelt, MD, 204 pp.

  • Oram, A., Ed.,. 2001: Peer-to-Peer: Harnessing the Power of Disruptive Technologies. O'Reilley and Associates, 445 pp.

  • Reinke, D. L., 2001: CloudSat data processing center system description and data processing plan. Cooperative Institute for Research in the Atmosphere (CIRA), Colorado State University, Fort Collins, CO, 62 pp.

    • Search Google Scholar
    • Export Citation

  • Rew, R., Davis G. , Emmerson S. , and Davies H. , 1997: NetCDF user's guide for C: An access interface for self-describing, portable data, version 3. Unidata Program Center, Boulder, CO, 132 pp.

    • Search Google Scholar
    • Export Citation

  • Rosen, K. H., Rosinski R. R. , Farber J. M. , and Host D. A. , 1996: Unix System V Release 4: An Introduction. McGraw-Hill, 1175 pp.

  • Space Systems Loral, 1996: GOES I-M DataBook. Tech. Rep. DRL-101-08, Space Systems Loral, 196 pp.

  • St. Laurent, S., Dumbill E. , and Johnston J. , 2001: Programming Web Services with XML-RPC. O'Reilly Internet Series, O'Reilly and Associates, 230 pp.

    • Search Google Scholar
    • Export Citation

  • Suomi, V. E., Fox R. , Limaye S. S. , and Smith W. L. , 1983: McIDAS III: A modern data access and analysis system. J. Climate Appl. Meteor., 22 , 766–778.

    • Crossref
    • Search Google Scholar
    • Export Citation

  • Thai, T. L., and Lam H. , 2001: NET Framework Essentials. O'Reilly Programming Series, O'Reilly and Associates, 304 pp.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 889 628 273
PDF Downloads 77 13 1