1. Introduction
Oceanographic and meteorological applications of empirical orthogonal decompositions have been commonly used for several decades and are well documented in textbooks (e.g., Preisendorfer 1988). Empirical orthogonal functions (EOFs) are however not a tool specific to the ocean sciences, but are widely used in a number of scientific domains under different names: proper orthogonal decomposition (POD) or Karhunen–Loève decompositions are mostly used in a functional framework [working on mathematical continuous functions; e.g., Lumley (1970); Holmes et al. (1996)], whereas the matrix versions (working on discrete datasets) are called proper orthogonal modes (POM), principal component analysis (PCA), factor analysis (e.g., Hair et al. 1998), or empirical orthogonal function analysis.
In oceanography, EOF analysis has been used for several kinds of applications, among them objective analysis (or optimal interpolation) of in situ data; for time-dependent 3D datasets, classic objective analysis methods are generally intractable and the use of vertical EOFs and horizontal objective analysis of their amplitudes drastically reduces the amount of data during the costly optimal interpolation, but takes into account the strong vertical correlations contained in the EOFs (Holbrook and Bindoff 2000). Other applications of EOFs include statistical comparison between data and model results (e.g., Beckers and MEDMEX 2002), data compression (e.g., Pedder and Gomis 1998), analysis of variability (e.g., Hendricks et al. 1996), and filtering (e.g., Vautard et al. 1992). Reducing the size of a problem by projecting (complex) reference model equations onto a small number of principal components (D'Andrea and Vautard 2001) is also a powerful application of EOF analysis.
Each of these applications basically exploit the fact that EOFs allow a decomposition of a signal into a series of data-based orthogonal functions, which are designed so that only a few of these functions are needed to optimally reconstruct the initial signal.
Contrary to most other domains, in oceanographic data and in particular satellite data analysis, both noise in the data and missing data points are a concern and must be dealt with: in the datasets, there is a combination of missing data due to misfunctions of instruments, inaccessible data (e.g., due to cloud coverage), instrumental error bars (generally small, except for altimetric data for which the precision of a few centimeters can be in the range of the dynamical signal), and noise related to small-scale features not resolved by the observational network (at least all features below the Nyquist frequency).
The aim of the present paper is to analyze the problem of missing or unreliable data in oceanographic datasets and in their EOF analysis. We will present the method on classic EOF (allowing for complex data), although generalizations of the present method are possible, as for example to extended EOFs, delayed EOFs, etc. (e.g., Emery and Thomson 1998); singular spectrum analysis (e.g., Vautard et al. 1992); or other transformations (e.g., Elmore and Richman 2001). For the sake of clarity and without loss of generality, we also focus on data given in a uniform spatiotemporal grid; for nonuniform grids, the approach of North et al. (1982) or Shen et al. (1998) (basically replacing the discrete problem by a continuous integral problem, allowing weighting of the data according to their associated spatial or temporal coverage) can be used to work numerically in a discrete uniform space.
To finish the general description of EOF decompositions, we can mention that when working on EOFs or covariances, some a priori background field or trend is subtracted from the data series (e.g., Kantha and Clayson 2000), notably to allow for an easy interpretation of the eigenvalues in terms of the energy of the variability [see Eq. (9)]. For the sake of symmetry in terms of space or time coordinates, we assume here that the a priori average we substract from the field (in order to work with anomalies) is the spatiotemporal average of all available data.
2. Missing data
The SVD method presented above assumes that the matrix 𝗫 is perfectly and completely known. Currently, when dealing with missing or unreliable data, one fills in the missing data by objective analysis methods or optimal interpolation (e.g., Houseago-Stokes 2000): of course the problem with such an approach is that one needs information about the correlation function and the signal/noise ratio of the data for the interpolation (e.g., Rixen et al. 2000; Gomis et al. 2001). Since the purpose of using EOFs is often for the data analysis itself, another preliminary data analysis is not very natural and introduces a priori information into the subsequent EOF analysis. This filling method (and its variants that use simpler interpolation methods) is the most commonly used option to estimate the missing data values, because it consists of applying a classic analysis tool (e.g., Wunsch 1996) as preprocessing, before using any SVD or EOF analysis package.
Another method for dealing with missing data consists of calculating the eigenvalues and eigenvectors of the correlation or covariance matrix 𝗖, where the covariance is calculated using only the available data (Boyd et al. 1994; Kaplan et al. 1997; von Storch and Zwiers 1999). This however does not necessarily lead to a semipositive defined covariance matrix (since it is no longer possible to write 𝗖 = 𝗫*𝗫, which would lead to an always positive defined form x*𝗖x, with x being a column vector). Therefore its eigenvalues can be negative. Since the energy of the system is related to the trace of the covariance matrix that is equal to the sum of its eigenvalues, having negative eigenvalues means that other eigenvectors will have higher eigenvalues than in reality, thus overestimating the corresponding amplitude of the EOF. In addition, once the eigenvalues and eigenvectors are calculated, one still faces the problem of deciding which EOFs are relevant. Due to the presence of negative singular values, the selection criteria can no longer rely on the relative importance of the eigenvalue, but should also take into account the effect of the negative eigenvalues on the distribution of the larger eigenvalues. Moreover, using a temporal instead of a spatial covariance matrix (based on averaging the available data only) will generally lead to different eigenvalues (and thus singular values) though the mathematical counterpart for the correlation matrix based on 𝗫𝗫* or 𝗫*𝗫 gives identical results. Another way of interpreting this can be based on the observation that the EOF coefficients (the product of the singular value and the temporal EOF) of a gappy data vector cannot simply be obtained by a dot product, but must be estimated by a least squares approach (von Storch and Zwiers 1999).
Since both of the approaches (filling or correlation calculated on the sole available data) most commonly used contain some subjective components (choice of interpolation methods and interpolation parameters, level of significant EOFs, …), any method that would not need such additional choices could be of interest. The development of such a method is the purpose of the present work and is presented in the following way. First we present the interpolation method and analyze how it modifies the data at the missing data points in section 3a. Then we will tackle the problem of deciding on the optimal number of EOFs to be retained (section 3b). We will validate the method on two datasets: a synthetic one for objective quantifications of the errors of the method (section 4a), and then a real Advanced Very High Resolution Radiometer (AVHRR) dataset in section 4b. Finally, conclusions are drawn in section 5.
3. EOF-based interpolation
We now present a method aimed at filling in the gaps in the datasets and determining the structure and number of significant EOFs.
Basically, since EOFs are supposed to summarize in a very efficient way the signal and are also supposed to detect coherent features, it is tempting to use the EOFs themselves for interpolations. If we knew the dominant EOFs of a given field and their amplitudes, their combination would also give us the value of the field at the missing data points.
a. Interpolation and adaptation of the EOFs
Large-scale EOFs should not be influenced by local changes in the values of a few points, and so one could try to calculate the EOFs based on a matrix in which a first rough estimate of the variable is used in the missing data points. Then, once the larger-scale EOFs and their amplitudes are estimated, they can serve to calculate the value of the field at the missing points. Then, of course, the EOFs themselves can be reevaluated and the process can be repeated until convergence. One of the problems, which will of course arise, is the decision on the number N of relevant EOFs to be retained to recompose the signal at the missing data points. We will come to this question later. Formally the method can then be described as follows:
let I be the ensemble of discrete points where data are missing; that is, when (i, j) ∈ I, we do not have data or they are unreliable; I is of course a subset of [1, m] × [1, n], and the number of missing data points is no;
𝗫t is the true field we would like to recover;
𝗥 is a noise field (unresolved features, instrumental noise, etc…);
𝗫p = 𝗫t + 𝗥 is the true field with superimposed noise;
𝗫o is the observed field with gappy data (typically cloud coverage for satellite images) and zero values at those locations; and
𝗫a is the observed field where missing data were filled in (also called the analyzed field in analogy with objective analysis methods).
We will use the same sub- and superscripts for other variables (e.g., singular values) when we need to distinguish between those that are relevant to the real signal data, the noisy signal, signals with missing information, etc.
Our proposed algorithm performs as follows. First for all missing data points [(i, j) ∈ I] we put a value of 0 in the matrix (unbiased guess by assuming that the average value we subtracted was unbiased) to get the matrix 𝗫o.
It is instructive to analyze how the changes from one iteration to the next are structured. To do so, we assume that the changes are very small so that we can apply the perturbation theory of appendixes A and B.
If we start from the initial guess of zero anomalies at the missing data point, it can be shown (appendix A) that the variance of the dominant modes will generally be increased, whereas the less dominant modes will see their amplitude decreased. This is coherent with the interpretation that the initial guess overestimates “high frequency” signals and underestimates “large scale” signals by disregarding any local information around the missing data point.
We can also look at how the EOF structures are modified. In fact, it appears that the EOFs are slightly rotated so as to diminish the contribution to the dominant EOFs from the “noisy” EOFs partly introduced by the initial filling by zero values. We indeed observe in the examples (and show the mechanism in appendix A) that not only do we have a tendency to decrease the energy of higher modes, but that we subtract part of the corresponding higher EOF structure from the dominant EOF. This will, as will be seen in the examples, lead to a better representation of the real EOF structure.
In appendixes A and B it is also shown that if two EOFs have similar singular values, the iterations will strongly affect their representation, which could lead to large variations and convergence problems during the iterations. In this case, however, the EOFs do not have a proper meaning anyway, since any combination of two eigenvectors associated with the same eigenvalue is still an eigenvector. In this case (which was not encountered in the examples used later) iterations could be limited to a prescribed value.
b. Determination of the optimal number of EOF
We now have an algorithm that iteratively modifies EOFs and their amplitudes so as to achieve a self-consistent set of EOFs and interpolated data. However, the number N of relevant EOFs is yet undetermined. Subjective or objective criteria based on Monte Carlo methods could be applied (Preisendorfer and Overland 1982), but this would (a) introduce some subjective parameters or (b) introduce a criteria for truncation purely based on the matrix dimensions and not at all on the power spectra of the singular values of the given data. Furthermore, we would neglect a new possibility introduced by the method, the possibility to interpolate. Indeed, once we can interpolate missing data, we can turn toward cross-validation techniques (e.g., Brankart and Brasseur 1996). We can discard some additional points for validation by using the algorithm to interpolate these values too. Then we can compare the interpolated values to the data we put aside and have a measure of the interpolation error! Then of course finding the optimal number of EOFs is straightforward if we accept that the optimal number of EOFs as the one that minimizes the interpolation error. The procedure of a cross validation in the determination of the number of relevant EOFs is then the following. We set aside a random set of valid data typically at least 30 data points in order to have a robust statistical estimate of the interpolation error; but even better that a small percentage of the data points with more than 30 points should be used. The interpolation error estimate is then simply the rms distance between the interpolated field at these points and the data there.
The algorithm to determine the optimal number of EOFs and the corresponding interpolated data and EOFs is thus the following. We first apply the interpolation method with a single EOF retained until convergence. We can then calculate the error estimate. A second EOF is now taken into account in the interpolation method (starting of course with the interpolated data based on the single converged EOF) and interpolated until convergence with two EOFs. Then we can again calculate the error estimate. If the error starts to increase, the procedure stops; otherwise, we continue with more and more EOFs. To make sure that the process does not to stop at a local minimum, we could also continue the procedure until reaching a prescribed number of EOFs, never more than q, and retain the optimal set. Once the number of EOFs that minimizes the error estimate is found, we can fine-tune the calculation of the EOFs and data interpolation by finally injecting into the interpolation procedure the data that were set aside for the cross validation. In this case, one should of course start the final interative procedure with the interpolated data obtained with the optimal number of EOFs during the cross validation. In the case where there are no missing data, the proposed cross-validation technique reduces to finding the optimal number of EOFs and then in a last step performing the normal EOF decomposition on the data, knowing the optimal truncation.
With the cross-validation technique, we can thus find the optimal number of EOFs and an error estimate of the interpolation procedure (the rms distance between interpolated data and the validation data).
We now will show how the algorithm works on two datasets: a synthetic one and an AVHRR SST dataset in the Adriatic Sea.
4. Test cases
a. Synthetic data
The objective is to retrieve from 𝗫o the best estimates of 𝗫t and its relevant empirical orthogonal functions. In this synthetic dataset, we thus know the true solution and the noise, so that we can make all necessary statistical comparisons and error analyses. Here, we decide to normalize the error and we can choose either ‖ 𝗫t ‖ (when known, as in our synthetic example), or ‖ 𝗫a ‖ if only observations are available. Thus, a normalized error of 0.01 means an error with less than 1% energy of the signal. In the present example we normalize by ‖ 𝗫t ‖ .
We first apply our method to a case with a 20% noise level and 10% cloud coverage as illustrated in Fig. 1.
From the analysis of the singular values in this case (Table 1) the following points are clear:
The noise added to the data increases all singular values ρp compared to the unperturbed field (ρt), most significantly those from the fifth singular value on (cf. lines 1 and 2 in Table 1). This means that the first four singular values are slightly modified by the noise, or equivalently, that some of the noise structure was interpreted as EOFs signals. In view of this, there will be no chance to exactly recover the first four singular values (ρt).
The singular values obtained from the first guess underestimate the energy of the first modes (because we impose zero values, not taking into account the correlation between neighbors) and overestimates the higher EOFs' energy.
The reconstructed data (ρa) have energy levels for the EOFs much in line with the energy of the EOFs including the noise interpreted as signal (ρp), but the cross validation was able to detect that the optimal number of EOFs was four. Then, cutting beyond EOF4 eliminates the noise associated with the higher EOFs but it is impossible to eliminate the noise interpreted as signal in the first EOFs. Therefore the reconstructed data have the energy level of the first EOFs comparable to the noisy field.
We can also ask ourselves if the EOF structures are better represented after the interpolation method was applied. For mode 1, we have the following measures of errors on the EOF structure: ‖ u0 − up ‖ 2 = 0.0027, ‖ ut − up ‖ 2 = 0.0035, ‖ ua − up ‖ 2 = 0.0005, ‖ ua − ut ‖ 2 = 0.0042, and ‖ u0 − ut ‖ 2 = 0.0059, which shows that the reconstructed EOF ua is a better representation of the filtered ut or unfiltered up EOF than the initial guess u0 and that it is closer to the EOF including the interpreted noise than to the filtered EOF. Therefore we can conclude that the interpolation clearly helped to improve the EOF representation.
Then we modify the importance of the cloud coverage and noise level, in order to assess if the method systematically improves the estimates of the EOFs. Different errors are shown in Table 2 for different combinations of (uniform but random) cloud coverage and noise/signal ratios.
From Table 2 we can see the following.
The truncated EOF reconstructions from the analyzed field
𝗫ea 𝗫et 𝗫e0 The untruncated analyzed field 𝗫a is always a better representation of the true untruncated field 𝗫t than the initial guess 𝗫0, (columns 4 and 5), except again in the case when there are no clouds, in which case they are identical.
The truncated EOF series are always a better representation of the true field than the untruncated series, be it the initial guess solution (cf. columns 5 and 6) or the analyzed field (cf. columns 3 and 4).
The effect of repeating the iterations can be analyzed by comparing the calculations with a repeated iteration compared to a computation where iterations were limited to one application of the iterations (for the same number of EOFs). Errors ‖
Looking now qualitatively at the way the method re-creates the fields from the incomplete information, we observe (Fig. 1) that for a moderate noise/signal level and a cloud coverage of 10%, the reconstruction performs quite well, except for some higher-frequency signals imposed by the noise (in this case, truncating the EOF series earlier gives worse results because too much information is then eliminated, which shows that the cross-validation technique was able to pick up the best truncation). Increasing the noise (Fig. 2) shows that indeed the noise is responsible for the higher-frequency signals in the reconstruction. For a noise/signal ratio of O(1) (Fig. 3) even without cloud cover, the optimal EOF truncation filters only part of the noise. For lower noise, but a 50% cloud cover (Fig. 4), the reconstruction performs surprisingly well in view of the very fuzzy and incomplete original image. When the noise/signal ratio is decreased (Fig. 5), the reconstruction is still enhanced. For very low noise levels, cloud coverages of more than 75% (Fig. 6) still include the most relevant information that the EOF-based analysis method is able to extract. For even higher cloud coverage (Fig. 7) we see the problem that at some locations or moments there are no data at all. This clearly shows up in the EOF reconstruction and is due to the fact that no correlation information at all is available at these moments or points. Since this is the only information the EOF method uses (and no other a priori information in the correlation functions), it is impossible to infer the missing data when the whole time series is missing at some point (or one spatial pattern at one moment).
Concerning the detection of the optimal number of EOFs to retain, Fig. 8 shows the real error associated with the EOF truncation and error estimation from cross validation. In order to verify the stability of the truncation with respect to the random choice of the data put aside for cross validation, we repeated the experiment five times, with different random points and even a different number of data points set aside. In all cases the optimal number of EOFs detected was four, which corresponds to the minimum error. The difference in the absolute error estimation simply stems from the fact that, as already mentioned several times, the significant EOFs contain some interpreted noise as confirmed by the analysis of the singular values in the presence and absence of noise. Fig. 9 shows that the signal with noise has added some energy to the significant EOFs. We can also observe that on a subjective basis, the truncation with four EOFs would be the most natural, since only from EOF5 on are differences in the singular values between the noisy data and real data important. We also note that changing the random dataset put aside for the cross validation slightly modified the error estimate as well as the real error. For practical purposes, changing the random set of data put aside is thus an easy way to verify the robustness of the truncation. If for some reason there is then a doubt on the robustness, the cross validation presented here (simultaneously putting aside a number of data points) could be replaced by classical cross validation. The latter sets aside only a single data point (which makes the analysis less sensitive to the data elimination) and then repeats the experiment a sufficient number of times until a statistically significant measure of the error is obtained. This approach is then even more robust at the price of a higher number of analyses.
Concerning the present synthetic dataset, we have to mention that in reality cloud coverage is of course not random and that when larger patches of points are missing some degradations could appear. On the other hand, we did not implement any algorithm here to discard spatial points with too few data (or moments with full cloud coverage), so that some lines or columns in the reconstruction clearly show up as being underdetermined.
For comparison with other methods that deal with missing data, we use a simple method that is classically used and that linearly interpolates the missing data in space. No interpolation in time is done, since most of the time, satellite images are treated individually during the filling procedure. We can then compare our EOF-based interpolation and EOF truncation (Fig. 10) with the linearly interpolated field and its associated truncated EOF representation (Fig. 11, where the same number of EOFs is used). In the latter case, the systematic effects of linear interpolation are still visible in the EOF truncated (and hopefully filtered) reconstruction. One could argue that the improvement made by using the new method is not obvious, but besides the fact that the methods allow to detection of the optimal number of EOFs, a quantitative analysis of the error reveals the advantage of the new method. In the case presented here the error ‖
b. Real satellite data
As an illustrative example to demonstrate how the method behaves on real data and nonuniformly distributed cloud coverage, we will use an AVHRR dataset available from Ifremer (and available online at http://satftp.soest.hawaii.edu/adriatic/Adriatic/html/tecdoc.htm) from over the Adriatic Sea.
Seventy-one images of 400 pixels each centered in the middle of the Adriatic Sea (Fig. 12) from Julian day 130 to 294 of were retained for the present testing purposes2 with images at night (around midnight or 0200 LT) in order to avoid problems related to skin temperature.
As seen from the cloudy sequence (Fig. 13), the cloud coverage is not at all uniform, some pictures being more than half covered by clouds, and noisy patterns in the data are obvious. We therefore expect the present sequence to be a difficult test for the method. After application, the method selects an optimal truncation with nine EOFs. When reconstructing the data with these nine EOFs (Fig. 14), a clearly filtered sequence is visible, while the regions where clouds were filled in are almost impossible to detect (except maybe for one or two pictures in the middle of the sequence, where a persistent large cloud coverage can be seen). The benefit of updating the EOFs is clearly seen by looking at Fig. 15. Indeed, the original EOFs are neither able to eliminate the cloud signature nor the noise in the data. As in the synthetic data example, the choice of the optimal truncation by the cross-validation error estimator is quite robust (Fig. 16); however there is an indication that the error sequence contains some local minima. Our interpretation of the EOF modifications is also confirmed: Fig. 17 shows that the most significant EOF has increased energy compared to the initial guess, while the higher modes have a decreased energy after readjustments. Looking at the evolution of the temperature at a given location for the initial data and the reconstructed version in Fig. 18, the effects of both filtering and interpolation by the EOFs are obvious.
5. Discussion
Two improvements to classic EOF analysis were presented herein:
An automatic, parameter-free, and self-consistent filling mechanism was proposed. Compared to a classic interpolation method (as optimal interpolation), the use of the EOFs generated by the data themselves has the advantage of not needing any a priori knowledge of a correlation function or correlation length. Moreover, since it is based solely on the data, inhomogenities or nonisotropic behaviors are automatically taken into account (as long as they are in the available data) during the interpolation.
A cross validation that allows for establishing the optimal cutoff in EOF development (number of relevant EOFs to be retained) and the prediction error in the filled data gaps (from the cross validation estimator) were introduced.
Here computational efficiency considerations were not dealt with, since if only a few eigenvalues are needed, efficient eigenvalue solvers exist {compared to objective analysis used to fill the missing data [cost proportional to nm3 if done on space], the present interpolation/EOF decompositions are less time expensive consuming [roughly, (mn)][min(m, n)]2.5 for the analysis of the full range of possible N without optimization}. Moreover, the number of iterations could probably be reduced by using a better first guess. One could, for example, use any other classical method (be it filling methods or estimations of EOFs from the data correlation matrix) before actually starting the iterations.
We can also recall that with an enhanced and self-consistent filling mechanism, we are also improving the representation of the EOFs, meaning that any method relying on EOFs (e.g., some Kalman filters or predictions tools based on satellite images) will likely benefit from this better representation. For the method to have a chance to work, one needs, for each moment, at least a sufficiently large number of data points (otherwise one should drop the whole picture) and for each spatial point a sufficient amount of data in time (otherwise one should discard the point from the analysis). In both cases, disregarding a complete picture or a pixel can easily be dealt with. In the case where an image at a given moment is not available amounts to using nonuniform time steps between the data, so the methods to deal with nonuniform data can then be used.
Acknowledgments
European projects MEDAR (MAS3-CT98-0174) and SOFT (EVK3-CT-2000-00028) supported this work. M. Rixen presently benefits from a Marie-Curie fellowship at the Southampton Oceanographic Center (SOC). The National Fund for Scientific Research, Belgium, is acknowledged for the financing of supercomputer and the position of the first author. Two anonymous reviewers made constructive comments and are acknowledged. This is MARE publication MARE003. (Color images can be downloaded from http://modb.oce.be/personal/beckers/publications/cv.html.)
REFERENCES
Beckers, J-M., and Coauthors, 2002: Model intercomparison in the Mediterranean: MEDMEX simulations of the seasonal cycle. J. Mar. Syst., 33 , –34. 215–251.
Boyd, J., Kennelly E. , and Pistek P. , 1994: Estimation of EOF expansion coefficients from incomplete data. Deep-Sea Res., 41 , 1479–1488.
Brankart, J-M., and Brasseur P. , 1996: Optimal analysis of in situ data in the western Mediterranean using statistics and cross validation. J. Atmos. Oceanic Technol., 13 , 477–491.
D'Andrea, F., and Vautard R. , 2001: Extratropical low-frequency variability as a low dimensional problem. Part 1: A simplified model. Quart. J. Roy. Meteor. Soc., 127 , 1357–1375.
Elmore, K., and Richman M. , 2001: Euclidean distance as a similarity metric for principal component analysis. Mon. Wea. Rev., 129 , 540–549.
Emery, W., and Thomson R. , 1998: Data Analysis Methods in Physical Oceanography. Pergamon, 634 pp.
Gomis, D., Ruiz S. , and Pedder M. , 2001: Diagnostic analysis of the 3D ageostrophic circulation from a multivariate spatial interpolation of CTD and ADCP data. Deep-Sea Res., 48 , 269–295.
Hair, J., Anderson R. , Tatham R. , and Black W. , 1998: Multivariate Data Analysis. Prentice Hall, 730 pp.
Hendricks, J., Leben R. , Born G. , and Koblinsky C. , 1996: Empirical orthogonal function analysis of global TOPEX/POSEIDON altimeter data and implications for detection of global sea rise. J. Geophys. Res., 101 , 14131–14145.
Holbrook, N., and Bindoff N. , 2000: A statistically efficient mapping technique for four-dimensionsal ocean temperature data. J. Atmos. Oceanic Technol., 17 , 831–846.
Holmes, P., Lumley J. , and Berkooz W. , 1996: Turbulence, Coherent Structures, Dynamical Systems and Symmetry. Cambridge University Press, 420 pp.
Houseago-Stokes, R., 2000: Using optimal interpolation and EOF analysis on North Atlantic satellite data. International WOCE Newsletter, No. 38, WOCE International Project Office, Southampton, United Kingdom, 26–28.
Kantha, L., and Clayson C. , 2000: Numerical Models of Oceans and Oceanic Processes. International Geophysics Series, Vol. 66, Academic Press, 940 pp.
Kaplan, A., Kushnir Y. , Cane M. , and Blumenthal B. , 1997: Reduced space optimal analysis for historical data sets: 136 years of Atlantic sea surface temperature. J. Geophys. Res., 102 , 27853–27860.
Lawson, C., and Hanson R. , 1974: Solving Least Square Problems. Prentice Hall, 340 pp.
Lumley, J., 1970: Stochastic Tools in Turbulence. Academic Press, 194 pp.
North, G., Bell T. , Cahalan F. , and Moeng F. , 1982: Sampling errors in the estimation of empirical orthogonal functions. Mon. Wea. Rev., 110 , 699–706.
Pedder, M., and Gomis D. , 1998: Applications of EOF analysis to the spatial estimation of circulation features in the ocean sampled by high-resolution CTD soundings. J. Atmos. Oceanic Technol., 15 , 959–978.
Preisendorfer, R., 1988: Principal Component Analysis in Meteorology and Oceanography. Elsevier, 425 pp.
Preisendorfer, R., and Overland J. , 1982: A significance test for principal components applied to a cyclone climatology. Mon. Wea. Rev., 110 , 1–4.
Rixen, M., Beckers J-M. , Brankart J-M. , and Brasseur P. , 2000: A numerically efficient data analysis method with error map generation. Ocean Modell., 2 , 45–60.
Shen, S., Smith T. , Ropelewski C. , and Livezey R. , 1998: An optimal regional averaging method with error estimates and a test using tropical Pacific SST data. J. Climate, 11 , 2340–2350.
Vautard, R., Yiou P. , and Ghil M. , 1992: Singular spectrum analysis: A toolkit for short, noisy chaotic signals. Physica D, 58 , 95–126.
von Storch, H., and Zwiers F. , 1999: Statistical Analysis in Climate Research. Cambridge University Press, 484 pp.
Wunsch, C., 1996: The Ocean Circulation Inverse Problem. Cambridge University Press, 442 pp.
APPENDIX A
Transfer of Variance between Different EOFs and Changes of Structure during Iterations
This shows that the first EOF number will increase the energy of mode 1 after the interpolation is done, except in the case when the corresponding EOF is zero at the missing data point. Otherwise the term is always positive and simply the product of the temporal and spatial EOF amplitude at the missing data point. This interpretation shows that by replacing the missing data value by the first EOF guess, we add energy to the system, which is simply the energy of the first EOF at that location.
APPENDIX B
Perturbation Theory on Singular Value Decomposition
Synthetic dataset of a spatial 1D problem with time evolution. N/S = 0.197, n = 0.1 (10%) cloud cover. (upper left) True field, (upper right) perturbed field, (lower left) clouded field, and (lower right) reconstructed field. Shading scale varies linearly from −4 at left to 4 at right
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Effect of increasing noise: N/S = 0.258, n = 0.1 cloud cover, (left) clouded field, and (right) reconstructed field
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Effect of increasing noise: N/S = 0.744, (left) no cloud cover, and (right) reconstructed field
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Effect of increasing noise: N/S = 0.118, n = 0.5 cloud cover, (left) clouded field, and (right) reconstructed field
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Effect of increasing noise: N/S = 0.023, n = 0.5 cloud cover, (left) clouded field, and (right) reconstructed field
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Effect of increasing noise: N/S = 0.0015, n = 0.77 cloud cover, (left) clouded field, and (right) reconstructed field
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Effect of increasing noise: N/S = 0.00037, n = 0.83 cloud cover, (left) clouded field, and (right) reconstructed field
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
(left) Real error (compared to the filtered field) in the function of the number of EOFs N retained. (right) estimated error based on cross validation. The right one includes error due to noise. Five different sets of random data points set aside for the cross validation were used to verify robustness. For the real error, using another random set of data put aside does not change the optimal truncation and only slightly changes the error itself. For the error estimate through cross validation, the optimal truncation was not sensitive to a change in the random set, but the error estimation itself is slightly changed
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Noise 4.5 (dots) vs no noise (solid line) for the first 10 eigenvalues: no cloud, logarithmic scale
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
EOF-based interpolation and EOF truncation: N/S = 0.15, n = 0.50 cloud cover, (left) clouded field, and (right) reconstructed field
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Linear interpolation and EOF truncation: N/S = 0.15, n = 0.50 cloud cover, (left) clouded field interpolated linearly in space (but not in time), and (right) EOF truncated field
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Location of the AVHRR images rectangle in the Adriatic Sea used in this study
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Sequence of AVHRR images in the Adriatic Sea with clouds. The sequence is presented in left to right, top to bottom order
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Reconstructed sequence of images in Fig. 13 with optimal number of EOFs and the proposed algorithm including the changed EOFs
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Reconstructed sequence of images in Fig. 13 with the optimal number of EOFs, but using unmodified EOFs from the first guess x = 0
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Normalized error estimation from cross validation as a function of the number of EOFs N retained. Normalization is done by using the data standard deviation of 2.6° with the rms error in the prediction being 0.36°
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Energy vs EOF with singular values from initial dataset filled by zeros (dots) and the reconstructed dataset (solid line)
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
Evolution of temperature at a randomly chosen position. Initial values (dots) filled by zero anomalies (ticks instead of dots) corresponding to the average value 21.5°C. The line corresponds to the reconstructed time series
Citation: Journal of Atmospheric and Oceanic Technology 20, 12; 10.1175/1520-0426(2003)020<1839:ECADFF>2.0.CO;2
First seven singular values for exact field (ρt ), noisy field (ρp , clouded field filled with zeros (ρ0 ), and analyzed field (ρa )
Errors for different combinations of noise/signal ratios N/S and cloud coverage n
MARE Publication Number MARE003.
Strictly speaking the covariance matrix should be 1/n𝗫𝗫*, since it is obtained by time averaging.
Optimized versions with larger datasets are surely possible in view of the computational cost being roughly {[min(n, m)2.5](m + n)}.
This requires that 𝗔𝗕* is essentially different from 0 meaning that the perturbation matrix must not be orthogonal to the initial matrix.
