• Abousleman, G. P., 1999: Adaptive coding of hyperspectral imagery. Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Process., Phoenix, AZ, Institute of Electrical and Electronic Engineers, 2243–2246.

  • Abousleman, G. P., , Marcellin M. W. , , and Hunt B. R. , 1997: Hyperspectral image compression using entropy-constrained predictive trellis coded quantization. IEEE Trans. Image Process., 6 , 566573.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ahuja, A., , Huang B. , , Huang H-L. , , and Goldberg M. D. , 2006: Comparison of minimum spanning tree reordering with bias-adjusted reordering for lossless compression of 3D ultraspectral sounder data. Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XII, S. S. Shen and P. E. Lewis, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 6233), D1–D8.

    • Search Google Scholar
    • Export Citation
  • Aumann, H. H., , and Strow L. , 2001: AIRS, the first ultraspectral infrared sounder for operational weather forecasting. Proc. 2001 IEEE Aerospace Conf., Big Sky, MT, Institute of Electrical and Electronic Engineers, 1683–1692.

  • Bloom, H. J., 2001: The Cross-track Infrared Sounder (CrIS): A sensor for operational meteorological remote sensing. Proc. IEEE Int. Geoscience and Remote Sensing Symp., Sydney, Australia, Institute of Electrical and Electronic Engineers, 1341–1343.

  • CCSDS, 1997: Lossless data compression. Tech. Doc. 121.0-B-1, Blue Book, Issue 1, 33 pp.

  • CCSDS, 2005: Image data compression. Tech. Doc. 122.0-B-1, Blue Book, Issue 1, 63 pp.

  • Cuperman, V., , and Gersho A. , 1982: Adaptive differential vector coding of speech. Proc. IEEE Global Telecommunications Conf., Miami, FL, Institute of Electrical and Electronic Engineers, 1092–1096.

  • Eckstein, B. A., , Peters R. , , Irvine J. M. , , Ritzel R. , , and Hummel R. , 2000: Assessing the performance effects of data compression for SAR imagery. Proc. Applied Imagery Pattern Recognition Workshop, Washington, DC, Institute of Electrical and Electronic Engineers, 102–108.

  • Edmonds, J., 1967: Optimum branchings. J. Res. Natl. Bur. Stand. (U.S.), 71B , 233240.

  • Gersho, A., , and Gray R. M. , 1991: Vector Quantization and Signal Compression. Springer-Verlag, 760 pp.

  • Gladkova, I., , and Grossberg M. , 2006: A lossless compression algorithm for hyperspectral data. Satellite Data Compression, Communications, and Archiving II, R. W. Heymann, C. C. Wang, and T. J. Schmit, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 6300), 1–11.

    • Search Google Scholar
    • Export Citation
  • Gladkova, I., , Roytman L. , , and Goldberg M. , 2005: Adaptive clustering for hyperspectral sounder data compression. Satellite Data Compression, Communications, and Archiving, B. Huang, R. W. Heymann, and C. C. Wang, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 5889), 49–58.

    • Search Google Scholar
    • Export Citation
  • Gray, R. M., 1984: Vector quantization. IEEE Acoustic Speech and Signal Processing Magazine, Vol. 1, 4–29.

  • Huang, B., , Smith W. L. , , Huang H-L. , , and Woolf H. M. , 2002: Comparison of linear forms of the radiative transfer equation with analytic Jacobians. Appl. Opt., 41 , 42094219.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Huang, B., , Huang H-L. , , Chen H. , , Ahuja A. , , Baggett K. , , Schmit T. J. , , and Heymann R. W. , 2004a: Data compression studies for NOAA hyperspectral environmental suite using 3D integer wavelet transforms with 3D set partitioning in hierarchical trees. Image and Signal Processing for Remote Sensing IX, L. Bruzzone, Ed., International Society for Optical Engineering (SPIE Proceedings, Vol. 5238), 255–265.

    • Search Google Scholar
    • Export Citation
  • Huang, B., , Ahuja A. , , Huang H-L. , , Schmit T. J. , , and Heymann R. W. , 2004b: Predictive partitioned vector quantization for hyperspectral sounder data compression. Atmospheric and Environmental Remote Sensing Data Processing and Utilization: An End-to-End System Perspective, H.-L. A. Huang and H. J. Bloom, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 5548), 70–77.

    • Search Google Scholar
    • Export Citation
  • Huang, B., , Ahuja A. , , Huang H-L. , , Schmit T. J. , , and Heymann R. W. , 2004c: Lossless compression of 3D hyperspectral sounding data using context-based adaptive lossless image codec with Bias-Adjusted Reordering. Opt. Eng., 43 , 20712079.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Huang, B., , Ahuja A. , , Huang H-L. , , Schmit T. J. , , and Heymann R. W. , 2004d: Effects of the starting channel for spectral reordering on the lossless compression of ultraspectral sounder data. Multispectral and Hyperspectral Remote Sensing Instruments and Applications II, A. M. Larar, M. Suzuki, and Q. Tong, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 5655), 353–363.

    • Search Google Scholar
    • Export Citation
  • Huang, B., , Ahuja A. , , Huang H-L. , , Schmit T. J. , , and Heymann R. W. , 2005a: Fast precomputed VQ with optimal bit allocation for lossless compression of ultraspectral sounder data. Proc. IEEE Data Compression Conf., Snowbird, UT, Institute of Electrical and Electronic Engineers, 408–417.

  • Huang, B., , Ahuja A. , , Huang H-L. , , Schmit T. J. , , and Heymann R. W. , 2005b: Ultraspectral sounder data compression using error-correcting reversible variable-length coding. Satellite Data Compression, Communications, and Archiving, B. Huang, R. W. Heymann, and C. C. Wang, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 5889), 167–176.

    • Search Google Scholar
    • Export Citation
  • Huang, B., , Ahuja A. , , and Huang H-L. , 2006: Lossless compression of ultraspectral sounder data. Hyperspectral Data Compression, G. Motta and J. Storer, Eds., Springer-Verlag, 75–106.

    • Search Google Scholar
    • Export Citation
  • ISO/IEC, 1999: Information technology—Lossless and near-lossless compression of continuous-tone still images: Baseline. ISO/IEC 14495-1 and ITU Recommendation T.87, 66 pp.

  • ISO/IEC, 2004a: Information technology—JPEG2000 image coding system: Core coding system. ISO/IEC 15444-1, 194 pp.

  • ISO/IEC, 2004b: Information technology—JPEG2000 image coding system: Extensions. ISO/IEC 15444-2, 321 pp.

  • Linde, Y., , Buzo A. , , and Gray R. M. , 1980: An algorithm for vector quantizer design. IEEE Trans. Commun., 28 , 8495.

  • Lloyd, S. P., 1982: Least squares quantization in PCM. IEEE Trans. Inf. Theory, 28 , 129136.

  • Max, J., 1960: Quantizing for minimum distortion. IRE Trans. Inf. Theory, 6 , 712.

  • Motta, G., , Rizzo F. , , and Storer J. A. , 2003: Compression of hyperspectral imagery. Proc. IEEE Data Compression Conf., Snowbird, UT, Institute of Electrical and Electronic Engineers, 333–342.

  • Penna, B., , Tillo T. , , Magli E. , , and Olmo G. , 2006: Progressive 3-D coding of hyperspectral images based on JPEG2000. IEEE Geosci. Remote Sens. Lett., 3 , 125129.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Phulpin, T., , Cayla F. , , Chalon G. , , Diebel D. , , and Schlüssel D. , 2002: IASI onboard Metop: Project status and scientific preparation. Proc. 12th Int. TOVS Study Conf., International Association of Meteorology and Atmospheric Sciences, 234–243.

  • Qian, S-E., , Hollinger A. B. , , Dutkiewicz M. , , Tsang H. , , Zwick H. , , and Freemantle J. R. , 2001: Effect of lossy vector quantization hyperspectral data compression on retrieval of red-edge indices. IEEE Trans. Geosci. Remote Sens., 39 , 14591470.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ryan, M. J., , and Arnold J. F. , 1998: A suitable distortion measure for the lossy compression of hyperspectral data. Proc. IEEE Int. Geoscience and Remote Sensing Symp., Vol. 4, Seattle, WA, Institute of Electrical and Electronic Engineers, 2056–2058.

  • Saghri, J. A., , Tescher A. G. , , and Reagan J. T. , 1995: Practical transform coding of multispectral imagery. Signal Processing, Sensor Fusion, and Target Recognition IV, I. Kadar and V. Libby, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 2484), 354–357.

    • Search Google Scholar
    • Export Citation
  • Said, A., 2004: Introduction to arithmetic coding theory and practice. H. P. Labs Tech. Rep. HPL-2004-76, 67 pp.

  • Schmit, T. J., , Walsh T. , , Heymann R. W. , , and Gurka J. , 2006: GOES-R data dissemination. Second Symp.: Toward a Global Earth Observation System of Systems—Future National Operational Environmental Satellite Systems, Atlanta, GA, Amer. Meteor. Soc, 4.4.

  • Serra-Sagrista, J., , Garcia F. , , Minguillon J. , , Megias D. , , Huang B. , , and Ahuja A. , 2005: Wavelet lossless compression of ultraspectral sounder data. Proc. IEEE Int. Geoscience and Remote Sensing Symp., Vol. 1, Seoul, South Korea, Institute of Electrical and Electronic Engineers, 148–151.

  • Shaw, G. A., , and Burke H-H. K. , 2003: Spectral imaging for remote sensing. Lincoln Lab. J., 14 , 328.

  • Shen, S. S., , Lindgren J. E. , , and Payton P. M. , 1993: Effects of multispectral compression on machine exploitation. Proc. IEEE Conf. Record of the Twenty-Seventh Asimolar Conf. on Signals, Systems, and Computers, Vol. 2, Pacific Grove, CA, Institute of Electrical and Electronic Engineers, 1352–1356.

  • Smith, W. L., , Harrison F. W. , , Hinton D. E. , , Revercomb H. E. , , Bingham G. E. , , Petersen R. , , and Dodge J. C. , 2002: GIFTS—The precursor geostationary satellite component of the future earth observing system. Proc. IEEE Int. Geoscience and Remote Sensing Symp., Toronto, ON, Canada, Institute of Electrical and Electronic Engineers, 357–361.

  • Witten, I. H., , Neal R. M. , , and Cleary J. C. , 1987: Arithmetic coding for data compression. Commun. ACM, 30 , 520541.

  • Wu, X., 1997: Lossless compression of continuous-tone images via context selection, quantization, and modeling. IEEE Trans. Image Process., 6 , 656664.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • View in gallery

    AIRS digital counts at wavenumber 800.01 cm−1 for the 10 selected granules on 2 Mar 2004.

  • View in gallery

    Block diagram of the AVQLP method.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 23 23 7
PDF Downloads 5 5 1

Optimal Compression of High Spectral Resolution Satellite Data via Adaptive Vector Quantization with Linear Prediction

View More View Less
  • 1 Space Science and Engineering Center, Cooperative Institute for Meteorological Satellite Studies, University of Wisconsin—Madison, Madison, Wisconsin
© Get Permissions
Full access

Abstract

Contemporary and future high spectral resolution sounders represent a significant technical advancement for environmental and meteorological prediction and monitoring. Given their large volume of spectral observations, the use of robust data compression techniques will be beneficial to data transmission and storage. In this paper, a novel adaptive vector quantization (VQ)-based linear prediction (AVQLP) method for lossless compression of high spectral resolution sounder data is proposed. The AVQLP method optimally adjusts the quantization codebook sizes to yield the maximum compression on prediction residuals and side information. The method outperforms the state-of-the-art compression methods [Joint Photographic Experts Group (JPEG)-LS, JPEG2000 Parts 1 and 2, Consultative Committee for Space Data Systems (CCSDS) Image Data Compression (IDC) 5/3, Context-Based Adaptive Lossless Image Coding (CALIC), and 3D Set Partitioning in Hierarchical Trees (SPIHT)] and achieves a new high in lossless compression for the standard test set of 10 NASA Atmospheric Infrared Sounder (AIRS) granules. It also compares favorably in terms of computational efficiency and compression gain to recently reported adaptive clustering methods for lossless compression of high spectral resolution data. Given its superior compression performance, the AVQLP method is well suited to ground operation of high spectral resolution satellite data compression for rebroadcast and archiving purposes.

Corresponding author address: Dr. Bormin Huang, Space Science and Engineering Center, 1225 W. Dayton St., Madison, WI 53706. Email: bormin@ssec.wisc.edu

Abstract

Contemporary and future high spectral resolution sounders represent a significant technical advancement for environmental and meteorological prediction and monitoring. Given their large volume of spectral observations, the use of robust data compression techniques will be beneficial to data transmission and storage. In this paper, a novel adaptive vector quantization (VQ)-based linear prediction (AVQLP) method for lossless compression of high spectral resolution sounder data is proposed. The AVQLP method optimally adjusts the quantization codebook sizes to yield the maximum compression on prediction residuals and side information. The method outperforms the state-of-the-art compression methods [Joint Photographic Experts Group (JPEG)-LS, JPEG2000 Parts 1 and 2, Consultative Committee for Space Data Systems (CCSDS) Image Data Compression (IDC) 5/3, Context-Based Adaptive Lossless Image Coding (CALIC), and 3D Set Partitioning in Hierarchical Trees (SPIHT)] and achieves a new high in lossless compression for the standard test set of 10 NASA Atmospheric Infrared Sounder (AIRS) granules. It also compares favorably in terms of computational efficiency and compression gain to recently reported adaptive clustering methods for lossless compression of high spectral resolution data. Given its superior compression performance, the AVQLP method is well suited to ground operation of high spectral resolution satellite data compression for rebroadcast and archiving purposes.

Corresponding author address: Dr. Bormin Huang, Space Science and Engineering Center, 1225 W. Dayton St., Madison, WI 53706. Email: bormin@ssec.wisc.edu

1. Introduction

In the era of contemporary and future high spectral resolution sounders, such as the Atmospheric Infrared Sounder (AIRS; Aumann and Strow 2001), Cross-track Infrared Sounder (CrIS; Bloom 2001), Infrared Atmospheric Sounding Interferometer (IASI; Phulpin et al. 2002), Geosynchronous Imaging Fourier Transform Spectrometer (GIFTS; Smith et al. 2002), and Hyperspectral Environmental Suite (HES; Huang et al. 2004a; Schmit et al. 2006), better inference of atmospheric, cloud, and surface parameters is feasible. An unprecedented amount of three-dimensional (3D) data, consisting of two spatial and one spectral dimension, is produced by the ultraspectral sounders. Given the large volume of 3D data that will be generated by an ultraspectral sounder each day, the use of robust data compression techniques will be beneficial to data rebroadcast, transfer, and storage.

There are differences between ultraspectral sounder data and hyperspectral imager data in terms of application areas and subsequent user constraints on the data compression. The hyperspectral imager data [e.g., the well-known Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) data (Abousleman 1999)] is in the visible or near-infrared regions with major application categories of anomaly detection, target recognition, and background characterization (Shaw and Burke 2003). Lossy compression is usually acceptable for imager data as long as the tolerance limits in application-specific metrics are met (Saghri et al. 1995). These metrics include those that signify scientific loss for end users (Qian et al. 2001; Ryan and Arnold 1998), content-independent metrics (Shen et al. 1993), and even visual comparisons (Eckstein et al. 2000). On the other hand, the ultraspectral sounder data are in the infrared region with the main purpose of retrieving atmospheric temperature and absorbing gas profiles, as well as surface properties, for better weather and climate prediction. The physical retrieval of these geophysical parameters involves the inverse solution of the radiative transfer equation, and it is a mathematically ill-posed problem (Huang et al. 2002); that is, the solution is sensitive to the error or noise in the data. Therefore, there is a need for lossless or near-lossless compression of ultraspectral sounder data to avoid potential retrieval degradation due to lossy compression.

In the literature, state-of-the-art lossless compression methods investigated for ultraspectral sounder data have been categorized as transform-, prediction-, projection-, and clustering-based methods (Huang et al. 2006). Serra-Sagrista et al. (2005) provide a comparison of various transform-based lossless compression methods for ultraspectral sounder data. The ultraspectral sounder data feature strong correlations in disjoint spectral regions affected by the same type of absorbing gases at various altitudes. To take advantage of this feature, a bias-adjusted reordering (BAR) data preprocessing scheme (Huang et al. 2004c, d) is used to improve compression gains of state-of-the-art transform- and prediction-based methods for ultraspectral sounder data. The minimum spanning tree (MST; Edmonds 1967) reordering can also be used as a data preprocessing scheme (Ahuja et al. 2006) for ultraspectral sounder data compression.

Vector quantization (VQ; Gray 1984) is a clustering-based method that has long been used for hyperspectral imager data compression (Abousleman et al. 1997; Motta et al. 2003). To reduce the computational burden for ultraspectral sounder data compression, predictive partitioned VQ (PPVQ) was proposed by Huang et al. (2004b), which falls under the category of predictive vector quantization (Cuperman and Gersho 1982; Gersho and Gray 1991). Huang et al. (2005a) also developed a fast precomputed vector quantization (FPVQ) scheme with optimal bit allocation. In this paper, we present a new coding method, adaptive VQ-based linear prediction (AVQLP), for lossless compression of ultraspectral sounder data. The method combines vector quantization and linear prediction in an adaptive way. The size of the VQ codebook is adaptively increased. The VQ code words are used as linear predictors for estimating each spectral channel. Thus, the size of the linear predictors is also adaptive. This method also introduces adaptive uniform quantization to reduce the VQ codebooks and linear prediction coefficients as side information. The compression ratios obtained by AVQLP are compared with those produced by conventional VQ, as well as several state-of-the-art methods, such as Context-Based Adaptive Lossless Image Coding (CALIC; Wu 1997), Joint Photographic Experts Group (JPEG)-LS (ISO/IEC 1999), JPEG2000 (ISO/IEC 2004a), and the latest lossless compression recommendation from the Consultative Committee for Space Data Systems (CCSDS 1997; CCSDS 2005). Given its superior compression performance, the AVQLP method is well suited to ground operation of ultraspectral sounder data compression for rebroadcast and archiving purposes.

The rest of the paper is arranged as follows. Section 2 describes the ultraspectral sounder data used in this study. Section 3 highlights the AVQLP compression method, while section 4 presents the compression results of the AVQLP method. Section 5 summarizes the paper.

2. Ultraspectral sounder data

The ultraspectral sounder dataset with 10 AIRS (available online at http://www-airs.jpl.nasa.gov) digital count granules was prepared at the direction of the National Oceanic and Atmospheric Administration (NOAA) to serve as a standard test set for ultraspectral compression studies in support of the NOAA next-generation geostationary operational environmental satellite. The data are publicly available via anonymous ftp (ftp://ftp.ssec.wisc.edu/pub/bormin/Count/). It consists of 10 digital count granules, five daytime and five nighttime, selected from representative geographical regions of the earth on 2 March 2004. Their locations, UTC times, and local time adjustments are listed in Table 1.

This standard ultraspectral sounder dataset is obtained from the National Aeronautics and Space Administration (NASA) AIRS digital counts collected on 2 March 2004. The AIRS data include 2378 infrared channels in the 3.74- to 15.4-μm region of the spectrum. A day’s worth of AIRS data is divided into 240 granules, each of 6-min durations. The AIRS digital count data range from 12 to 14 bits for different channels. To make the selected data more generic to other ultraspectral sounders, 271 AIRS-specific bad channels are excluded. Each resulting granule is saved as a binary file, arranged as 2107 channels, 135 scan lines, and 90 cross-track footprints per scan line; that is, there are a total of 135 × 90 = 12 150 footprints per channel. For this study, the data represent a 2D matrix with size 2107 × 12 150. Figure 1 shows the 10 AIRS digital count granules at wavenumber 800.01 cm−1. In these granules, coast lines are depicted by solid curves, and multiple clouds at various altitudes are shown as different levels of grayscale.

3. Adaptive VQ-based linear prediction method

The AVQLP method consists of four stages: adaptive VQ, adaptive LP, adaptive scalar quantization (SQ), and adaptive arithmetic coding (AC), as depicted in Fig. 2. Each step is briefly described.

a. Adaptive vector quantization

The 3D spectral data cube of size nx × ny × nc is reshaped into a 2D matrix 𝗔 of size ns pixels by nc channels, where ns = nx × ny. In this step, each 2D spatial frame with ns pixels is considered a vector; that is, 𝗔 = {X1, . . . , Xnc}. The Linde–Buzo–Gray (LBG) algorithm (Linde et al. 1980), also known as the generalized Lloyd algorithm (Gray 1984; Gersho and Gray 1991), can be used to generate a VQ codebook from the input granule. The algorithm starts with an initial codebook and iterates until the process converges. An iteration of the algorithm is decomposed into a clustering step, where the vectors are partitioned into clusters, and a codebook step, where new code vectors (also known as code words) are calculated. The algorithm can be described as follows.

  • Step 1: Begin with an initial codebook, V = {Vi; i = 1, . . . , m}, where m is the number of code words.
  • Step 2: Assign each input vector, X ∈ 𝗔, to its nearest cluster 𝗦i, given by
    i1520-0426-25-6-1041-e1
    and compute the distortion by
    i1520-0426-25-6-1041-e2
  • Step 3: Update the code words as the centroids of the clusters, given by
    i1520-0426-25-6-1041-e3
    where n(𝗦i) is the cardinality of the set 𝗦i.
  • Step 4: Update the distortion by
    i1520-0426-25-6-1041-e4
    If |d0d| < δ, stop, otherwise go to step 2.

The adaptive VQ starts with a codebook of two random code words for input to the LBG algorithm, and the codebook size is doubled until the best estimated compression gain is reached. Doubling of the codebook is achieved by splitting each codeword Vi, (i = 1, . . . , m) into two code words, Viɛ and Vi + ɛ, where ɛ is a fixed perturbation vector.

b. Adaptive linear prediction

The VQ code words, V, obtained from the previous stage are used as predictors for the input granule 𝗔. The problem is formulated as 𝗔 = 𝗩 𝗖, where 𝗖 is the matrix of the LP coefficients obtained by 𝗖 = (𝗩T 𝗩)−1 𝗩T 𝗔. The prediction residuals are rounded. For the cases where the predicted channel residual norm is greater than the predicted channel norm, the predicted channel replaces the residual.

c. Adaptive scalar quantization

To reduce side information, adaptive SQ is first applied to the VQ code words and then to the prediction coefficients. The adaptive SQ is based on the well-known Lloyd–Max scalar quantization algorithm (Lloyd 1982; Max 1960). The Lloyd–Max scalar quantization algorithm is used to find the quantizers and partition endpoints that minimize the mean squared distortion, given a probability density function (PDF). Each obtained quantizer is the centroid of a PDF between the neighboring partition endpoints, whereas each obtained endpoint is at the midpoint of two neighboring quantizers. Both the sizes of the SQ codebooks for VQ code words and prediction coefficients are doubled until the best estimated compression gain is adaptively reached. Steps 1–3 are repeated until there is no improvement in the estimated compression gain.

d. Adaptive arithmetic coding

The arithmetic coding is an entropy coder that approaches the optimal compression bound. It utilizes the concept of interval subdivision, where successive input symbols are encoded as intervals on the range [0, 1) based on their probability of occurrence (Said 2004). Given a set of source symbols T = {t1, . . . , tn} with probabilities P = {p1, . . . , pn}, the cumulative probabilities are calculated as ck = Σk−1i=1pi, with c1 = 0. The range [0, 1) is initially represented by a base l0 = 0 and a length r0 = 1. The kth input symbol tk corresponds to an interval with the lower bound lk and the range rk satisfying lk = lk−1 + ck rk−1 and rk = pk rk−1. A symbol with a higher occurrence probability will posses a larger interval, which requires fewer bits to represent, whereas symbols with a lower occurrence probability will have a smaller interval, which requires more bits to represent. During the interval reduction process, the arithmetic encoder outputs the leading bits, which are the same between the upper and lower bounds of the current interval. Using the leading bits and the statistics of the received symbols, the arithmetic decoder can adaptively duplicate the same interval reduction process and locate the correct symbol. In the context-based adaptive AC (Witten et al. 1987) a symbol is encoded based on the occurrence probabilities of its previous symbols, denoted as the context of the symbol. The context-based adaptive AC generally yields better compression gain than the context-free static AC, since it exploits the higher-order statistical dependencies among neighboring symbols. For the proposed AVQLP compression method, the previous three symbols in the same residual spatial frame are chosen as context for the current residual symbol, and the context-based adaptive AC is used to encode the prediction residuals and the SQ indexes for both VQ codebook and prediction coefficients.

4. Results

In this study the aforementioned NASA AIRS ultraspectral sounder dataset with 2107 channels is used for lossless compression. Table 2 shows the achieved compression ratios using the AVQLP compression method. For comparison, the compression ratios using state-of-the-art methods, such as CALIC, JPEG-LS, JPEG2000 Parts 1 and 2 (ISO/IEC 2004a, b; Huang et al. 2005b; Penna et al. 2006), 3D Set Partitioning in Hierarchical Trees (SPIHT; Huang et al. 2004a, 2006), and the CCSDS recommendation for Image Data Compression (IDC) 5/3 method (Serra-Sagrista et al. 2005) are also shown. The recently released JPEG2000 Part 2 supports compression of 3D data by allowing a 1D wavelet transform along one dimension followed by the 2D spatial wavelet transform on the other two dimensions. As seen from Table 2, the AVQLP method significantly outperforms all these methods in terms of compression ratios.

Conventionally, the VQ uses code words to approximate the data, and then the VQ codebooks and residuals are compressed. In the proposed AVQLP, the VQ code words are used as linear predictors to approximate the data, and then the VQ codebooks, prediction coefficients, and prediction residuals are compressed. The superior compression gain of AVQLP over its adaptive VQ counterpart (without linear prediction) for the aforementioned AIRS dataset is shown in Table 3.

In some literature on AIRS ultraspectral sounder data compression, compression ratios have been reported based on a partial set (with 1502 lower noise channels and increased bit depth) instead of the full set of AIRS ultraspectral sounder test data. In the real world all the channels in the spaceborne ultraspectral sounder are expected to be transmitted, although not all are expected to be used. Getting rid of the higher noise channels and assuming larger data bit depth to boost compression ratios will lead to an overoptimistic estimate of future sounders’ spectral design and data transmission feasibility. Therefore, this practice of using the partial set of the data with increased bit depth is not encouraged. However, for a fair comparison the compression ratios of the AVQLP method for the 1502-channel subset are listed in Table 4, along with those from a previously reported adaptive clustering method (Gladkova et al. 2005). As seen, the average compression ratio of 4.30 from AVQLP outperforms the value of 3.50 from the adaptive clustering method. It also compares favorably to the average compression ratio of 3.72 over 235 global AIRS granules produced by a more recent method also based on adaptive clustering (Gladkova and Grossberg 2006).

5. Summary

Contemporary and future hyperspectral and ultraspectral infrared sounders generate a huge amount of data. The compression of high spectral resolution sounder data is better to be lossless or near lossless to avoid potential degradation of the geophysical retrieval in the associated ill-posed problem. In this paper, we propose the adaptive VQ-based linear prediction (AVQLP) method for high spectral resolution data compression. The method is compared favorably with other state-of-the-art methods, including JPEG-LS and JPEG2000, and the two recent International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) standards for image compression. The significantly higher compression ratios show the advantage of the AVQLP method for lossless compression of hyperspectral and ultraspectral data. The AVQLP compression method is optimal in the sense that both the VQ and SQ codebook sizes are adaptively adjusted to yield the maximum compression gains on prediction residuals and side information. For the real-time implementation of the AVQLP method, the VQ and SQ codebook sizes can be predetermined from the experiments on the test data.

Acknowledgments

This work is prepared in support of NOAA next-generation GOES data compression research under Grant NA07EC0676. NOAA co-leads are Roger Heymann of NOAA/NESDIS/OSD and Tim Schmit of NOAA/NESDIS/ORA.

REFERENCES

  • Abousleman, G. P., 1999: Adaptive coding of hyperspectral imagery. Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Process., Phoenix, AZ, Institute of Electrical and Electronic Engineers, 2243–2246.

  • Abousleman, G. P., , Marcellin M. W. , , and Hunt B. R. , 1997: Hyperspectral image compression using entropy-constrained predictive trellis coded quantization. IEEE Trans. Image Process., 6 , 566573.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ahuja, A., , Huang B. , , Huang H-L. , , and Goldberg M. D. , 2006: Comparison of minimum spanning tree reordering with bias-adjusted reordering for lossless compression of 3D ultraspectral sounder data. Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XII, S. S. Shen and P. E. Lewis, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 6233), D1–D8.

    • Search Google Scholar
    • Export Citation
  • Aumann, H. H., , and Strow L. , 2001: AIRS, the first ultraspectral infrared sounder for operational weather forecasting. Proc. 2001 IEEE Aerospace Conf., Big Sky, MT, Institute of Electrical and Electronic Engineers, 1683–1692.

  • Bloom, H. J., 2001: The Cross-track Infrared Sounder (CrIS): A sensor for operational meteorological remote sensing. Proc. IEEE Int. Geoscience and Remote Sensing Symp., Sydney, Australia, Institute of Electrical and Electronic Engineers, 1341–1343.

  • CCSDS, 1997: Lossless data compression. Tech. Doc. 121.0-B-1, Blue Book, Issue 1, 33 pp.

  • CCSDS, 2005: Image data compression. Tech. Doc. 122.0-B-1, Blue Book, Issue 1, 63 pp.

  • Cuperman, V., , and Gersho A. , 1982: Adaptive differential vector coding of speech. Proc. IEEE Global Telecommunications Conf., Miami, FL, Institute of Electrical and Electronic Engineers, 1092–1096.

  • Eckstein, B. A., , Peters R. , , Irvine J. M. , , Ritzel R. , , and Hummel R. , 2000: Assessing the performance effects of data compression for SAR imagery. Proc. Applied Imagery Pattern Recognition Workshop, Washington, DC, Institute of Electrical and Electronic Engineers, 102–108.

  • Edmonds, J., 1967: Optimum branchings. J. Res. Natl. Bur. Stand. (U.S.), 71B , 233240.

  • Gersho, A., , and Gray R. M. , 1991: Vector Quantization and Signal Compression. Springer-Verlag, 760 pp.

  • Gladkova, I., , and Grossberg M. , 2006: A lossless compression algorithm for hyperspectral data. Satellite Data Compression, Communications, and Archiving II, R. W. Heymann, C. C. Wang, and T. J. Schmit, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 6300), 1–11.

    • Search Google Scholar
    • Export Citation
  • Gladkova, I., , Roytman L. , , and Goldberg M. , 2005: Adaptive clustering for hyperspectral sounder data compression. Satellite Data Compression, Communications, and Archiving, B. Huang, R. W. Heymann, and C. C. Wang, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 5889), 49–58.

    • Search Google Scholar
    • Export Citation
  • Gray, R. M., 1984: Vector quantization. IEEE Acoustic Speech and Signal Processing Magazine, Vol. 1, 4–29.

  • Huang, B., , Smith W. L. , , Huang H-L. , , and Woolf H. M. , 2002: Comparison of linear forms of the radiative transfer equation with analytic Jacobians. Appl. Opt., 41 , 42094219.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Huang, B., , Huang H-L. , , Chen H. , , Ahuja A. , , Baggett K. , , Schmit T. J. , , and Heymann R. W. , 2004a: Data compression studies for NOAA hyperspectral environmental suite using 3D integer wavelet transforms with 3D set partitioning in hierarchical trees. Image and Signal Processing for Remote Sensing IX, L. Bruzzone, Ed., International Society for Optical Engineering (SPIE Proceedings, Vol. 5238), 255–265.

    • Search Google Scholar
    • Export Citation
  • Huang, B., , Ahuja A. , , Huang H-L. , , Schmit T. J. , , and Heymann R. W. , 2004b: Predictive partitioned vector quantization for hyperspectral sounder data compression. Atmospheric and Environmental Remote Sensing Data Processing and Utilization: An End-to-End System Perspective, H.-L. A. Huang and H. J. Bloom, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 5548), 70–77.

    • Search Google Scholar
    • Export Citation
  • Huang, B., , Ahuja A. , , Huang H-L. , , Schmit T. J. , , and Heymann R. W. , 2004c: Lossless compression of 3D hyperspectral sounding data using context-based adaptive lossless image codec with Bias-Adjusted Reordering. Opt. Eng., 43 , 20712079.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Huang, B., , Ahuja A. , , Huang H-L. , , Schmit T. J. , , and Heymann R. W. , 2004d: Effects of the starting channel for spectral reordering on the lossless compression of ultraspectral sounder data. Multispectral and Hyperspectral Remote Sensing Instruments and Applications II, A. M. Larar, M. Suzuki, and Q. Tong, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 5655), 353–363.

    • Search Google Scholar
    • Export Citation
  • Huang, B., , Ahuja A. , , Huang H-L. , , Schmit T. J. , , and Heymann R. W. , 2005a: Fast precomputed VQ with optimal bit allocation for lossless compression of ultraspectral sounder data. Proc. IEEE Data Compression Conf., Snowbird, UT, Institute of Electrical and Electronic Engineers, 408–417.

  • Huang, B., , Ahuja A. , , Huang H-L. , , Schmit T. J. , , and Heymann R. W. , 2005b: Ultraspectral sounder data compression using error-correcting reversible variable-length coding. Satellite Data Compression, Communications, and Archiving, B. Huang, R. W. Heymann, and C. C. Wang, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 5889), 167–176.

    • Search Google Scholar
    • Export Citation
  • Huang, B., , Ahuja A. , , and Huang H-L. , 2006: Lossless compression of ultraspectral sounder data. Hyperspectral Data Compression, G. Motta and J. Storer, Eds., Springer-Verlag, 75–106.

    • Search Google Scholar
    • Export Citation
  • ISO/IEC, 1999: Information technology—Lossless and near-lossless compression of continuous-tone still images: Baseline. ISO/IEC 14495-1 and ITU Recommendation T.87, 66 pp.

  • ISO/IEC, 2004a: Information technology—JPEG2000 image coding system: Core coding system. ISO/IEC 15444-1, 194 pp.

  • ISO/IEC, 2004b: Information technology—JPEG2000 image coding system: Extensions. ISO/IEC 15444-2, 321 pp.

  • Linde, Y., , Buzo A. , , and Gray R. M. , 1980: An algorithm for vector quantizer design. IEEE Trans. Commun., 28 , 8495.

  • Lloyd, S. P., 1982: Least squares quantization in PCM. IEEE Trans. Inf. Theory, 28 , 129136.

  • Max, J., 1960: Quantizing for minimum distortion. IRE Trans. Inf. Theory, 6 , 712.

  • Motta, G., , Rizzo F. , , and Storer J. A. , 2003: Compression of hyperspectral imagery. Proc. IEEE Data Compression Conf., Snowbird, UT, Institute of Electrical and Electronic Engineers, 333–342.

  • Penna, B., , Tillo T. , , Magli E. , , and Olmo G. , 2006: Progressive 3-D coding of hyperspectral images based on JPEG2000. IEEE Geosci. Remote Sens. Lett., 3 , 125129.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Phulpin, T., , Cayla F. , , Chalon G. , , Diebel D. , , and Schlüssel D. , 2002: IASI onboard Metop: Project status and scientific preparation. Proc. 12th Int. TOVS Study Conf., International Association of Meteorology and Atmospheric Sciences, 234–243.

  • Qian, S-E., , Hollinger A. B. , , Dutkiewicz M. , , Tsang H. , , Zwick H. , , and Freemantle J. R. , 2001: Effect of lossy vector quantization hyperspectral data compression on retrieval of red-edge indices. IEEE Trans. Geosci. Remote Sens., 39 , 14591470.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ryan, M. J., , and Arnold J. F. , 1998: A suitable distortion measure for the lossy compression of hyperspectral data. Proc. IEEE Int. Geoscience and Remote Sensing Symp., Vol. 4, Seattle, WA, Institute of Electrical and Electronic Engineers, 2056–2058.

  • Saghri, J. A., , Tescher A. G. , , and Reagan J. T. , 1995: Practical transform coding of multispectral imagery. Signal Processing, Sensor Fusion, and Target Recognition IV, I. Kadar and V. Libby, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 2484), 354–357.

    • Search Google Scholar
    • Export Citation
  • Said, A., 2004: Introduction to arithmetic coding theory and practice. H. P. Labs Tech. Rep. HPL-2004-76, 67 pp.

  • Schmit, T. J., , Walsh T. , , Heymann R. W. , , and Gurka J. , 2006: GOES-R data dissemination. Second Symp.: Toward a Global Earth Observation System of Systems—Future National Operational Environmental Satellite Systems, Atlanta, GA, Amer. Meteor. Soc, 4.4.

  • Serra-Sagrista, J., , Garcia F. , , Minguillon J. , , Megias D. , , Huang B. , , and Ahuja A. , 2005: Wavelet lossless compression of ultraspectral sounder data. Proc. IEEE Int. Geoscience and Remote Sensing Symp., Vol. 1, Seoul, South Korea, Institute of Electrical and Electronic Engineers, 148–151.

  • Shaw, G. A., , and Burke H-H. K. , 2003: Spectral imaging for remote sensing. Lincoln Lab. J., 14 , 328.

  • Shen, S. S., , Lindgren J. E. , , and Payton P. M. , 1993: Effects of multispectral compression on machine exploitation. Proc. IEEE Conf. Record of the Twenty-Seventh Asimolar Conf. on Signals, Systems, and Computers, Vol. 2, Pacific Grove, CA, Institute of Electrical and Electronic Engineers, 1352–1356.

  • Smith, W. L., , Harrison F. W. , , Hinton D. E. , , Revercomb H. E. , , Bingham G. E. , , Petersen R. , , and Dodge J. C. , 2002: GIFTS—The precursor geostationary satellite component of the future earth observing system. Proc. IEEE Int. Geoscience and Remote Sensing Symp., Toronto, ON, Canada, Institute of Electrical and Electronic Engineers, 357–361.

  • Witten, I. H., , Neal R. M. , , and Cleary J. C. , 1987: Arithmetic coding for data compression. Commun. ACM, 30 , 520541.

  • Wu, X., 1997: Lossless compression of continuous-tone images via context selection, quantization, and modeling. IEEE Trans. Image Process., 6 , 656664.

    • Crossref
    • Search Google Scholar
    • Export Citation
Fig. 1.
Fig. 1.

AIRS digital counts at wavenumber 800.01 cm−1 for the 10 selected granules on 2 Mar 2004.

Citation: Journal of Atmospheric and Oceanic Technology 25, 6; 10.1175/2007JTECHA917.1

Fig. 2.
Fig. 2.

Block diagram of the AVQLP method.

Citation: Journal of Atmospheric and Oceanic Technology 25, 6; 10.1175/2007JTECHA917.1

Table 1.

Ten selected AIRS granules for ultraspectral sounder data compression studies from 2 Mar 2004.

Table 1.
Table 2.

Comparison of compression ratios from CALIC, JPEG-LS, JPEG2000 Parts 1 and 2, 3D SPIHT, CCSDS IDC 5/3, and AVQLP for the 10 tested AIRS granules with 2107 channels. JPEG2000 Part 2 supports 3D data compression.

Table 2.
Table 3.

Comparison of compression ratios from the AVQLP method and its adaptive VQ counterpart (without linear prediction) for the 10 tested AIRS granules with 2107 channels.

Table 3.
Table 4.

Compression comparison of the AVQLP method to the adaptive clustering method applied to the 1502-channel lower noise subset of the 10 tested granules with increased bit depth.

Table 4.
Save