Editorial Type:
Article
Article Type:
Research Article

Fast Playback Framework for Analysis of Ground-Based Doppler Radar Observations Using MapReduce Technology

Jingyin Tang Department of Geography, University of Florida, Gainesville, Florida

Search for other papers by Jingyin Tang in
Current site
Google Scholar
PubMed Close
and
Corene J. Matyas Department of Geography, University of Florida, Gainesville, Florida

Search for other papers by Corene J. Matyas in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Apr 2016
DOI:
https://doi.org/10.1175/JTECH-D-15-0118.1
Page(s):
621–634
Restricted access

Abstract

The creation of a 3D mosaic is often the first step when using the high-spatial- and temporal-resolution data produced by ground-based radars. Efficient yet accurate methods are needed to mosaic data from dozens of radar to better understand the precipitation processes in synoptic-scale systems such as tropical cyclones. Research-grade radar mosaic methods of analyzing historical weather events should utilize data from both sides of a moving temporal window and process them in a flexible data architecture that is not available in most stand-alone software tools or real-time systems. Thus, these historical analyses require a different strategy for optimizing flexibility and scalability by removing time constraints from the design. This paper presents a MapReduce-based playback framework using Apache Spark’s computational engine to interpolate large volumes of radar reflectivity and velocity data onto 3D grids. Designed as being friendly to use on a high-performance computing cluster, these methods may also be executed on a low-end configured machine. A protocol is designed to enable interoperability with GIS and spatial analysis functions in this framework. Open-source software is utilized to enhance radar usability in the nonspecialist community. Case studies during a tropical cyclone landfall shows this framework’s capability of efficiently creating a large-scale high-resolution 3D radar mosaic with the integration of GIS functions for spatial analysis.

Corresponding author address: Jingyin Tang, Department of Geography, University of Florida, 3141 Turlington Hall, P.O. Box 117315, Gainesville, FL 32611-7315. E-mail: jtang8756@ufl.edu

Abstract

The creation of a 3D mosaic is often the first step when using the high-spatial- and temporal-resolution data produced by ground-based radars. Efficient yet accurate methods are needed to mosaic data from dozens of radar to better understand the precipitation processes in synoptic-scale systems such as tropical cyclones. Research-grade radar mosaic methods of analyzing historical weather events should utilize data from both sides of a moving temporal window and process them in a flexible data architecture that is not available in most stand-alone software tools or real-time systems. Thus, these historical analyses require a different strategy for optimizing flexibility and scalability by removing time constraints from the design. This paper presents a MapReduce-based playback framework using Apache Spark’s computational engine to interpolate large volumes of radar reflectivity and velocity data onto 3D grids. Designed as being friendly to use on a high-performance computing cluster, these methods may also be executed on a low-end configured machine. A protocol is designed to enable interoperability with GIS and spatial analysis functions in this framework. Open-source software is utilized to enhance radar usability in the nonspecialist community. Case studies during a tropical cyclone landfall shows this framework’s capability of efficiently creating a large-scale high-resolution 3D radar mosaic with the integration of GIS functions for spatial analysis.

Corresponding author address: Jingyin Tang, Department of Geography, University of Florida, 3141 Turlington Hall, P.O. Box 117315, Gainesville, FL 32611-7315. E-mail: jtang8756@ufl.edu
Save
Cover Journal of Atmospheric and Oceanic Technology
Journal of Atmospheric and Oceanic Technology

  • Ansari, S., Del Greco S. , and Hankins B. , 2010: The weather and climate toolkit. 2010 Fall Meeting, San Francisco, CA, Amer. Geophys. Union, Abstract IN32A-06.

  • Apache, 2015: Spark programming guide. Accessed 23 January 2015. [Available online at http://spark.apache.org/docs/latest/programming-guide.]

  • Bentley, J. L., 1975: Multidimensional binary search trees used for associative searching. Commun. ACM, 18, 509–517, doi:10.1145/361002.361007.

    • Search Google Scholar
    • Export Citation

  • Bluestein, H. B., and Coauthors, 2014: Radar in atmospheric sciences and related research: Current systems, emerging technology, and future needs. Bull. Amer. Meteor. Soc., 95, 1850–1861, doi:10.1175/BAMS-D-13-00079.1.

    • Search Google Scholar
    • Export Citation

  • Carbone, R., Carpenter M. , and Burghart C. , 1985: Doppler radar sampling limitations in convective storms. J. Atmos. Oceanic Technol., 2, 357–361, doi:10.1175/1520-0426(1985)002<0357:DRSLIC>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Chandrasekar, V., Cho Y.-G. , Brunkow D. , and Jayasumana A. , 2005: Virtual CSU-CHILL radar: The VCHILL. J. Atmos. Oceanic Technol., 22, 979–987, doi:10.1175/JTECH1745.1.

    • Search Google Scholar
    • Export Citation

  • Crockford, D., 2006: The application/json media type for JavaScript Object Notation (JSON). Internet Requests for Comments RFC 4627. [Available online at http://www.rfc-editor.org/rfc/rfc4627.txt.]

  • Crum, T. D., and Alberty R. L. , 1993: The WSR-88D and the WSR-88D operational support facility. Bull. Amer. Meteor. Soc., 74, 1669–1687, doi:10.1175/1520-0477(1993)074<1669:TWATWO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Crum, T. D., Alberty R. L. , and Burgess D. W. , 1993: Recording, archiving, and using WSR-88D data. Bull. Amer. Meteor. Soc., 74, 645–653, doi:10.1175/1520-0477(1993)074<0645:RAAUWD>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Dean, J., and Ghemawat S. , 2008: MapReduce: Simplified data processing on large clusters. Commun. ACM, 51, 107–113, doi:10.1145/1327452.1327492.

    • Search Google Scholar
    • Export Citation

  • Dixon, M., and Wiener G. , 1993: TITAN: Thunderstorm Identification, Tracking, Analysis, and Nowcasting—A radar-based methodology. J. Atmos. Oceanic Technol., 10, 785–797, doi:10.1175/1520-0426(1993)010<0785:TTITAA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Franklin, J. L., Pasch R. J. , Avila L. A. , Beven J. L. , Lawrence M. B. , Stewart S. R. , and Blake E. S. , 2006: Atlantic hurricane season of 2004. Mon. Wea. Rev., 134, 981–1025, doi:10.1175/MWR3096.1.

    • Search Google Scholar
    • Export Citation

  • Gao, J., Droegemeier K. K. , Gong J. , and Xu Q. , 2004a: A method for retrieving mean horizontal wind profiles from single-Doppler radar observations contaminated by aliasing. Mon. Wea. Rev., 132, 1399–1409, doi:10.1175/1520-0493(2004)132<1399:AMFRMH>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Gao, J., Xue M. , Brewster K. , and Droegemeier K. K. , 2004b: A three-dimensional variational data analysis method with recursive filter for Doppler radars. J. Atmos. Oceanic Technol., 21, 457–469, doi:10.1175/1520-0426(2004)021<0457:ATVDAM>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Gao, J., and Coauthors, 2013: A real-time weather-adaptive 3DVAR analysis system for severe weather detections and warnings. Wea. Forecasting, 28, 727–745, doi:10.1175/WAF-D-12-00093.1.

    • Search Google Scholar
    • Export Citation

  • Germann, U., and Zawadzki I. , 2002: Scale-dependence of the predictability of precipitation from continental radar images. Part I: Description of the methodology. Mon. Wea. Rev., 130, 2859–2873, doi:10.1175/1520-0493(2002)130<2859:SDOTPO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Heistermann, M., Jacobi S. , and Pfaff T. , 2013: Technical note: An open source library for processing weather radar data (wradlib). Hydrol. Earth Syst. Sci., 17, 863–871, doi:10.5194/hess-17-863-2013.

    • Search Google Scholar
    • Export Citation

  • Helmus, J., Collis S. , Johnson K. L. , North K. , Giangrande S. E. , and Jensen M. , 2013: The Python-ARM Radar Toolkit (Py-ART), an open source package for weather radar. 36th Conf. on Radar Meteorology, Breckenridge, CO, Amer. Meteor. Soc., 392. [Available online at https://ams.confex.com/ams/36Radar/webprogram/36RADAR.html.]

  • Hu, H., 2014: An algorithm for converting weather radar data into GIS polygons and its application in severe weather warning systems. Int. J. Geogr. Inf. Sci., 28, 1765–1780, doi:10.1080/13658816.2014.898767.

    • Search Google Scholar
    • Export Citation

  • Lakshmanan, V., and Humphrey T. W. , 2014: A MapReduce technique to mosaic continental-scale weather radar data in real-time. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 7, 721–732, doi:10.1109/JSTARS.2013.2282040.

    • Search Google Scholar
    • Export Citation

  • Lakshmanan, V., Smith T. , Hondl K. , Stumpf G. J. , and Witt A. , 2006: A real-time, three-dimensional, rapidly updating, heterogeneous radar merger technique for reflectivity, velocity, and derived products. Wea. Forecasting, 21, 802–823, doi:10.1175/WAF942.1.

    • Search Google Scholar
    • Export Citation

  • Lakshmanan, V., Fritz A. , Smith T. , Hondl K. , and Stumpf G. J. , 2007a: An automated technique to quality control radar reflectivity data. J. Appl. Meteor. Climatol., 46, 288–305, doi:10.1175/JAM2460.1.

    • Search Google Scholar
    • Export Citation

  • Lakshmanan, V., Smith T. , Stumpf G. , and Hondl K. , 2007b: The Warning Decision Support System–Integrated Information. Wea. Forecasting, 22, 596–612, doi:10.1175/WAF1009.1.

    • Search Google Scholar
    • Export Citation

  • Lakshmanan, V., Karstens C. , Krause J. , and Tang L. , 2014: Quality control of weather radar data using polarimetric variables. J. Atmos. Oceanic Technol., 31, 1234–1249, doi:10.1175/JTECH-D-13-00073.1.

    • Search Google Scholar
    • Export Citation

  • Lee, W.-C., and Bell M. M. , 2007: Rapid intensification, eyewall contraction, and breakdown of Hurricane Charley (2004) near landfall. Geophys. Res. Lett., 34, L02802, doi:10.1029/2006GL027889.

    • Search Google Scholar
    • Export Citation

  • Li, X., and Mecikalski J. R. , 2012: Impact of the dual-polarization Doppler radar data on two convective storms with a warm-rain radar forward operator. Mon. Wea. Rev., 140, 2147–2167, doi:10.1175/MWR-D-11-00090.1.

    • Search Google Scholar
    • Export Citation

  • Lin, Y.-L., Ensley D. B. , Chiao S. , and Huang C.-Y. , 2002: Orographic influences on rainfall and track deflection associated with the passage of a tropical cyclone. Mon. Wea. Rev., 130, 2929–2950, doi:10.1175/1520-0493(2002)130<2929:OIORAT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Matejka, T., and Srivastava R. C. , 1991: An improved version of the extended velocity-azimuth display analysis of single-Doppler radar data. J. Atmos. Oceanic Technol., 8, 453–466, doi:10.1175/1520-0426(1991)008<0453:AIVOTE>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Matyas, C. J., 2007: Quantifying the shapes of U.S. landfalling tropical cyclone rain shields. Prof. Geogr., 59, 158–172, doi:10.1111/j.1467-9272.2007.00604.x.

    • Search Google Scholar
    • Export Citation

  • Matyas, C. J., 2009: A spatial analysis of radar reflectivity regions within Hurricane Charley (2004). J. Appl. Meteor. Climatol., 48, 130–142, doi:10.1175/2008JAMC1910.1.

    • Search Google Scholar
    • Export Citation

  • Matyas, C. J., 2010: Use of ground-based radar for climate-scale studies of weather and rainfall. Geogr. Compass, 4, 1218–1237, doi:10.1111/j.1749-8198.2010.00370.x.

    • Search Google Scholar
    • Export Citation

  • Michalakes, J., Dudhia J. , Gill D. , Klemp J. , Skamarock W. , and Wang W. , 2004: The Weather Research and Forecast Model version 2.0. Proc. 11th Workshop on the Use of High Performance Computing in Meteorology, Reading, United Kingdom, ECMWF, 156–158. [Available online at http://www.ecmwf.int/sites/default/files/elibrary/2004/14144-weather-research-and-forecast-model-version-20.pdf.]

  • Mohr, C. G., Jay Miller L. , Vaughan R. L. , and Frank H. W. , 1986: The merger of mesoscale datasets into a common Cartesian format for efficient and systematic analyses. J. Atmos. Oceanic Technol., 3, 143–161, doi:10.1175/1520-0426(1986)003<0143:TMOMDI>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Montmerle, T., Caya A. , and Zawadzki I. , 2001: Simulation of a midlatitude convective storm initialized with bistatic Doppler radar data. Mon. Wea. Rev., 129, 1949–1967, doi:10.1175/1520-0493(2001)129<1949:SOAMCS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Nettleton, L., Daud S. , Neitzel R. , Burghart C. , Lee W. , and Hildebrand P. , 1993: SOLO: A program to peruse and edit radar data. Preprints, 26th Conf. on Radar Meteorology, Norman, OK, Amer. Meteor. Soc., 338–339.

  • NOAA/ROC, 2008: RPG SW build 10.0—Includes reporting for SW 41 RDA software note 41/43. NOAA/Radar Operations Center. Accessed 8 May 2015. [Available online at http://www.roc.noaa.gov/ssb/cm/csw_notes/Completion.aspx?ID=2689.]

  • Oye, D., and Case M. , 1995: REORDER: A program for gridding radar data; Installation and use manual for the Unix version. NCAR ATD, 44 pp. [Available online at https://www.eol.ucar.edu/system/files/unixreorder.pdf.]

  • Rew, R., and Davis G. , 1990: NetCDF: An interface for scientific data access. IEEE Comput. Graphics Appl., 10, 76–82, doi:10.1109/38.56302.

    • Search Google Scholar
    • Export Citation

  • Steiniger, S., and Bocher E. , 2009: An overview on current free and open source desktop GIS developments. Int. J. Geogr. Inf. Sci., 23, 1345–1370, doi:10.1080/13658810802634956.

    • Search Google Scholar
    • Export Citation

  • Tiranti, D., Cremonini R. , Marco F. , Gaeta A. R. , and Barbero S. , 2014: The DEFENSE (debris Flows triggEred by storms—Nowcasting system): An early warning system for torrential processes by radar storm tracking using a Geographic Information System (GIS). Comput. Geosci., 70, 96–109, doi:10.1016/j.cageo.2014.05.004.

    • Search Google Scholar
    • Export Citation

  • Vasiloff, S. V., and Coauthors, 2007: Improving QPE and very short term QPF: An initiative for a community-wide integrated approach. Bull. Amer. Meteor. Soc., 88, 1899–1911, doi:10.1175/BAMS-88-12-1899.

    • Search Google Scholar
    • Export Citation

  • Villarini, G., Smith J. A. , Baeck M. L. , Marchok T. , and Vecchi G. A. , 2011: Characterization of rainfall distribution and flooding associated with U.S. landfalling tropical cyclones: Analyses of Hurricanes Frances, Ivan, and Jeanne (2004). J. Geophys. Res., 116, D23116, doi:10.1029/2011JD016175.

    • Search Google Scholar
    • Export Citation

  • Vulpiani, G., Montopoli M. , Passeri L. D. , Gioia A. G. , Giordano P. , and Marzano F. S. , 2012: On the use of dual-polarized C-band radar for operational rainfall retrieval in mountainous areas. J. Appl. Meteor. Climatol., 51, 405–425, doi:10.1175/JAMC-D-10-05024.1.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 785 480 89
PDF Downloads 232 54 5