1. Introduction

Simple stochastic models fit to time series of daily precipitation amount commonly do not include a component that explicitly accounts for interannual (i.e., low frequency) variation. Of course, such models are still capable of producing a substantial interannual variance in monthly (or seasonal) total precipitation, in this case being solely attributable to their representation of high-frequency, day-to-day variations. Nevertheless, these models typically underestimate the observed interannual variance of monthly total precipitation by a nonnegligible fraction (Buishand 1978; Wilks 1989). This phenomenon is, more generally, termed “overdispersion” in the statistics literature (e.g., Cox 1983).

Two conflicting explanations for the overdispersion phenomenon have been proposed. The first one involves viewing the discrepancy in variance as evidence of an inadequate model for the high-frequency variations of daily precipitation (Gregory et al. 1993). In this regard, Katz and Parlange (1998) obtained results suggesting that the extent of the overdispersion could be reduced, but not necessarily eliminated, by fitting more complex stochastic models that better reflect the nature of the temporal dependence of daily precipitation.

The second explanation involves attributing the overdispersion phenomenon to low-frequency variations ignored by these stochastic models. In fact, some researchers have treated this difference in variance as a measure of the “potential predictability” of precipitation on an interannual timescale (Madden et al. 1999; Singh and Kripalani 1986). In the present paper, the extent to which the overdispersion of precipitation can be reduced by explicitly accounting for low-frequency variations is studied in greater detail. Recent research in statistics indicates that these two explanations can be difficult to distinguish in practice without systematically considering each possibility (Fitzmaurice 1997).

The approach to be taken is motivated by the work of Katz and Parlange (1993), in which stochastic models for daily precipitation were fitted conditionally on an index of large-scale atmospheric circulation. Despite the index being only imperfectly related to local precipitation, this approach did greatly reduce, if not eliminate, the overdispersion. Essentially the same approach to the modeling of daily precipitation is taken, except that now the index is treated as “hidden” (i.e., unobserved). The premise of the present approach is that, a priori, the relationship between large-scale atmospheric circulation patterns and precipitation at a particular location may not necessarily be well understood. As such, it could play a diagnostic role in climate research, detecting hidden sources of low-frequency variation whose origin might be the focus of subsequent study.

Specifically, a mixture model is proposed with a hidden index that takes on one of two possible states in a given year. Conditional on this index, the parameters of a stochastic model for daily time series of precipitation amount, known as a chain-dependent process (Katz and Parlange 1993), are permitted to vary. It should be noted that both Jones et al. (1995) and Zheng (1996) have formulated statistical models for times series of daily temperature in which the mean is permitted to vary interannually, in effect, taking on infinitely many states (termed a “random effect” in the statistics literature). Because of the complex nature of the precipitation process (especially its intermittency), it is not feasible to directly apply this analysis of variance approach. The assumption of two (or a small number of) hidden states, while highly restrictive, is consistent with the belief that only a few dominant modes of large-scale atmospheric circulation exist (e.g., Hansen and Sutera 1995).

In section 2, a stochastic model for time series of daily precipitation amount, consisting of a mixture of two conditional chain-dependent processes, is defined and some of its properties are outlined. Because the index is hidden, section 3 describes a specialized statistical technique, known as the expectation-maximization (EM) algorithm (Dempster et al. 1977; McLachlan and Krishnan 1997), needed to estimate the model parameters by maximum likelihood. This method has only rarely been applied in the climate literature (Sansom 1995; Sansom 1998; Sansom and Thomson 1992). Some technical details concerning the implementation of the EM algorithm are relegated to an appendix. In section 4, results are presented for a location in California previously analyzed by Katz and Parlange (1993, 1996, 1998), as well as for another site in New Zealand. Finally, section 5 consists of a discussion of the interpretation of the results and of other potential applications of the methodology, as well as of some possible extensions.

2. Stochastic model

a. Chain-dependent process

First the definition and properties of a chain-dependent process, a relatively simple stochastic model for daily precipitation, are briefly reviewed. This model represents the most important features of precipitation, including its intermittency and the tendency of wet or dry spells to persist (for further details, see Katz and Parlange 1993).

1) Definition

Let {J_t : t = 1, 2, . . . , T} denote the sequence of daily precipitation occurrence (i.e., J_t = 1 indicates a“wet day” and J_t = 0 a “dry day”). It is assumed that this process is a first-order Markov chain, a model completely characterized by the transition probabilities

P_jkJ_t+1kJ_tjj, k(1)

Note that P_j0 = 1 − P_j1, j = 0, 1. For some purposes, it is convenient to reparameterize the Markov chain in terms of the unconditional probability of a wet day, π = Pr{J_t = 1}, and the first-order autocorrelation coefficient (or “persistence parameter”), d = Corr(J_t, J_t+1). Now it is well known that

πP₀₁P₁₁P₀₁dP₁₁P₀₁(2)

Let {X_t : t = 1, 2, . . . , T} denote the time series of daily precipitation amount. The “intensities” X_t > 0 (i.e., days for which J_t = 1) are taken to be conditionally independent and identically distributed with mean μ = E(X_t|J_t = 1) and σ² = Var(X_t|J_t = 1). It is further assumed that daily precipitation intensity has a power transform distribution. That is, a transformation,

X^*_tX^p_tp(3)

exists such that the transformed variable X^∗_t has a normal distribution, say, with mean μ∗ and variance (σ∗)² [note that μ and σ² each are functions of both μ∗ and (σ∗)²; e.g., Katz (1999).] For example, a value of p = ½, ⅓, or ¼ is commonly employed to account for the high degree of positive skewness in the distribution of daily intensity. Alternatively, a positively skewed distribution, such as the gamma, may be fitted directly to the untransformed intensities.

2) Properties

The overdispersion phenomenon concerns the variance of precipitation totaled over a period of length T days (e.g., a month or season), S(T) = X₁ + X₂ + · · · + X_T. The mean and variance of total precipitation are related to the parameters of a chain-dependent process by

View Expanded

(e.g., Katz and Parlange 1993). The expression for the variance is an approximation valid for a large number of days T. Estimates based on (4) can be substantially lower than the observed interannual variance of monthly or seasonal total precipitation (Katz and Parlange 1998).

3. Parameter estimation method

Parameter estimation for a mixture of two conditional chain-dependent processes would be straightforward if the index were actually observed. In this case, the estimation problem can be separated in two subsets (i.e., by classifying the daily precipitation time series according to which index state occurs in a given year). Then conventional maximum likelihood techniques can be applied to fit a chain-dependent process to each subset individually (e.g., as in Katz and Parlange 1993). However, when the index is hidden, the likelihood function is sufficiently complex that direct maximization is infeasible. In particular, even for just a mixture of two conditional normal distributions (effectively, the mixture model for the intensity component of the precipitation process), iterative numerical techniques are required to obtain maximum likelihood estimates (e.g., McLachlan and Krishnan 1997).

The EM algorithm (Dempster et al. 1977) is an iterative numerical technique to obtain maximum likelihood estimates, with the basic idea being to exploit the relative simplicity of likelihood maximization in the “complete-data” situation (i.e., if the index were observed). The situation actually faced is termed “incomplete data,” because the index state is regarded as “missing.” The “E” (for expectation) step of the EM algorithm involves replacing the unobservable complete-data likelihood function with its conditional expectation. Then updated parameter estimates can be obtained (the “M” or maximization step of the EM algorithm) in essentially as simple a manner as for the complete-data situation. Navidi (1997) provides a heuristic explanation of how the EM algorithm works, whereas McLachlan and Krishnan (1997) give an in-depth treatment. Making use of their conditional independence, it might have been anticipated that the occurrence and intensity components could be treated separately. But this simplification is not possible, because both components provide evidence about the likelihood of a particular hidden state having occurred during a given year. In other words, although the two components can be treated separately in the M step of the algorithm, they must be treated simultaneously in the E step.

a. Parameter estimation for chain-dependent process

First parameter estimation is reviewed for a single chain-dependent process, as these results will be utilized in formulating the EM algorithm. Suppose that an observed time series of daily precipitation for a single year {x_t : t = 1, 2, . . . , T} is generated from a chain-dependent process with parameters P₀₁, P₁₁, μ∗, and (σ∗)². Treating the first observation x₁ as if it were fixed, the exact maximum likelihood estimates of these parameters are given by

P̂_j1n_j1n_j·jμ̂∗s₁n_·1(σ̂∗)²s₂n_·1(μ̂∗)²(7)

where

View Expanded

(e.g., Katz and Parlange 1993). Here the transition count n_jk denotes the number of times that the Markov chain for precipitation occurrence makes a transition from state j to state k in the sample (e.g., n₁₁ denotes the number of times that a wet day is followed by a wet day). Note that the total number of transitions is n₀₀ + n₀₁ + n₁₀ + n₁₁ = T − 1, and that the two summations appearing in (8) are only taken over those days t = 2, 3, . . . , T that are wet (i.e., n_·1 = n₀₁ + n₁₁ terms). An expression for the corresponding likelihood function of a single chain-dependent process is included in appendix A [see (A1)].

b. Complete-data likelihood function

Now the complete-data likelihood function for a mixture of two conditional chain-dependent processes is considered. The daily precipitation data consist of M time series, each of length T days. Let x_t(m), t = 1, 2, . . . , T; m = 1, 2, . . . , M, denote the observed precipitation amount on the tth day of the mth year, with i(m) denoting the known index state for the mth year. Also let the vector of parameters be denoted by θ = (w, θ₀, θ₁), where

θ_iP₀₁iP₁₁iμ^∗_iσ^∗_i²i(9)

are the parameter vectors for the individual conditional chain-dependent processes. It is convenient to work with log likelihood, and the logarithm of the complete-data likelihood function is related to those for the two individual processes by

View Expanded

Here L_m(θ_i) denotes the likelihood function (A1), with parameter vector corresponding to index state i, evaluated for the daily precipitation time series in the mth year. Note that the first term (second term) on the right-hand side of (10) only appears if i(m) = 0 [i(m) = 1]. The simple form of the likelihood function is evident, as (10) reduces to a separate sum for each chain-dependent process.

The maximum likelihood estimates for the model parameters are given by

View Expanded

Here the “(m)” appended to a statistic, defined in section 3a for a single chain-dependent process, indicates that it is now calculated for the time series in the mth year, and the summations in (11) all range over m = 1, 2, . . . , M. Expressions for the corresponding estimators of P_j1(0), μ^∗₀, and (σ^∗₀)² can be obtained by replacing i(m) with 1 − i(m) in (11). These estimates in (11) correspond to those for a single chain-dependent process fit to a subset of years, as categorized by the index state (see Katz and Parlange 1993). With a slight modification, these same expressions will be utilized in the EM algorithm.

c. EM algorithm

When the state of the index is actually hidden, the incomplete-data log likelihood function can be expressed as

View Expanded

Here the likelihood function L_m(θ_i) is the same as the one that appears on the right-hand side of (10). Contrasting (12) with (10), the complex structure of the actual likelihood function is evident, as each term in the sum involves both conditional chain-dependent processes. Recalling that the terms L_m(θ_i), i = 0, 1, in (12) are given by (A1), it is clear that direct maximization is not practicable.

1) E step

At any stage of the EM algorithm, the E step involves the determination of the conditional expectation of the complete-data likelihood function (10) given the observed data. Because (10) is linear in the unobservable index states, this conditional expectation simply requires the calculation of the posterior probability of the index state for each year given the present parameter estimates. By Bayes’s theorem,

ImwL_mθ₁wL_mθ₀wL_mθ₁(13)

m = 1, 2, . . . , M. In practice, the likelihood functions that appear in (13) are evaluated at the parameter estimates produced by the most recent M step of the EM algorithm (see description of M step below). The posterior probabilities, corresponding to the parameter estimates obtained at the final stage of the algorithm, can also be used to relate the hidden index to known measures of atmospheric circulation (see section 4). As remarked earlier, these posterior probabilities of the hidden states reflect contributions from both the occurrence and intensity components of the chain-dependent processes.

2) M step

The M step of the EM algorithm involves replacing the hidden index state i(m) with the posterior probability Pr{I(m) = 1} [determined in the E step by (13)] in the expressions (11) for the estimates if the index were observed. To start the algorithm, initial values are required for the model parameters. Then the E and M steps are repeated alternately until convergence [i.e., maximizing the incomplete-data log likelihood function (12)]. In other words, the parameter estimates are just weighted analogs of the observed index case, with these weights being revised at each stage of the algorithm. Details about the implementation of the EM algorithm are provided in appendix B.

4. Results

Two time series of daily precipitation amounts are considered, one for January at Chico, California, and another for July at Napier, New Zealand. Both of these sites are roughly 40° away from the equator, and in each case the time period is midwinter. Because the large-scale atmospheric circulation is known to exert a major influence on local precipitation patterns in winter in California (e.g., Cayan and Peterson 1989), the Chico example can be viewed as somewhat confirmatory in nature. So this example is somewhat unrealistic, in the sense that it fails to take into account known information about circulation influences. In contrast, New Zealand precipitation appears to be only weakly related to those indexes of atmospheric circulation that have been constructed so far (Sallinger 1980; Tait and Fitzharris 1998;Trenberth 1976). So the Napier example can be regarded as somewhat exploratory in nature, a more realistic application.

The signal of the hidden index may be relatively weak, suggesting that various constraints on the model parameters should be considered to make the modeling approach more parsimonious. In this context, the two competing stochastic models defined in section 2 could be termed the “completely unconstrained” model (i.e., a mixture of two conditional chain-dependent processes with all parameters differing) and the “completely constrained” model (i.e., a single unconditional chain-dependent process). As additional candidate models, constraints will be imposed on either of the two transition probabilities [i.e., either P₀₁(0) = P₀₁(1) or P₁₁(0) = P₁₁(1)] and on the transformed intensity variance (i.e., σ^∗₀ = σ^∗₁), with how these constraints are implemented within the EM algorithm being described in appendix B.

To compare the fit of these candidate models, Akaike’s information criterion (AIC) (Akaike 1974) and the Bayesian information criterion (BIC) (Schwarz 1978) are employed. Both of these criteria involve penalizing the maximized likelihood function for the number of parameters estimated:

View Expanded

Here L̂_l denotes the maximized likelihood function for the lth model, with this model requiring the estimation of K_l parameters. The model with minimum AIC or BIC is ordinarily selected, although we use these criteria as guidelines rather than rigid rules. For the particular form of stochastic model being fit in our application, the appropriate sample size in the penalty term for the BIC is not completely clear (Kass and Wasserman 1995). In (14), the relevant sample size has been treated as M years, as opposed to MT days.

5. Discussion

The method introduced for fitting a hidden mixture of two conditional chain-dependent processes to time series of daily precipitation amount is a direct means of providing evidence of the presence of low-frequency modes of variation. The question remains of how to reconcile these results with those obtained from the more traditional approach of fitting increasingly complex stochastic models for high-frequency variations of daily precipitation (e.g., Katz and Parlange 1998). If a source of low-frequency variation were actually present, then the resultant unconditional stochastic model for daily precipitation would actually resemble a more complex form of chain-dependent process [e.g., higher than first-order Markov chain model for precipitation occurrence; Katz and Parlange (1996)]. In this situation, the traditional approach might well result in the erroneous conclusion that a more complex stochastic model is appropriate. Even model selection criteria, such as AIC and BIC, specifically designed to deal with model complexity would not necessarily be robust against the possible presence of such a source of overdispersion (Fitzmaurice 1997).

The proposed method could provide corroboration of the hypothesized existence of potential predictability for precipitation on an interannual timescale. Estimates of such predictability have been based on a generalized analysis of variance approach (Madden et al. 1999). The present approach can be viewed as complementary in the sense that, through a hidden index, it specifies the most likely state of a source of low-frequency variation. Nevertheless, the results of Zheng (1996) suggest that it should not necessarily be relied on to give an alternative estimate of potential predictability. Through the elimination or reduction of overdispersion, this research also has potential application as a technique for improving the performance of stochastic weather generators used to produce scenarios of climate variability and change (Katz 1996).

Several extensions of the methodology developed in the present paper could be considered in future work. One of the limitations of the EM algorithm is that it does not automatically provide standard errors of the parameter estimates as a byproduct (McLachlan and Krishnan 1997). Bayesian methods would be a natural way to generate such information, as well as to quantify the uncertainty in derived statistics, such as estimates of overdispersion. Conditioning simultaneously on both observed and hidden variables would constitute a more realistic treatment of the present situation in climate research (Hughes et al. 1999). Although limited by the length of climate records, it would be natural to allow the hidden index to assume more than two possible states.

Some recent research has dealt with the somewhat analogous problem of detecting hidden sources of high-frequency variation in daily time series of climate variables such as precipitation. One technique relies on“hidden Markov models,” essentially the same approach as treated here, except that the hidden index is permitted to change its state on a daily basis and that the sequence of index states is modeled as a Markov chain (Guttorp 1995). A systematic comparison of these two approaches would be worthwhile.