a. Models

We applied the following three large-scale hydrologic models: VIC (version 4.0.6), SAC, and Noah (version 2.7.1). VIC (Liang et al. 1994; Cherkauer and Lettenmaier 1999) is a physically based, semidistributed model with a three-layer soil column overlain by multiple vegetation tiles and a two-layer snowpack. It solves both water and energy balances at the soil and snow surfaces on daily and subdaily time steps. Subgrid heterogeneity of soil moisture infiltration is represented using a statistical parameterization. Noah (Chen et al. 1996; Koren et al. 1999) is also a physically based model, with a four-layer soil column, a single vegetation layer, and a one-layer snowpack. Noah solves the water balance and a linearized energy balance at the soil and snow surfaces at a subdaily time step. SAC uses the conceptually based Sacramento Moisture Accounting Model (Burnash 1995; Burnash et al. 1973), originally designed as a lumped model but adapted to gridded parameters for the North American Land Data Assimilation System (NLDAS) project (Mitchell et al. 2004). It does not perform an energy balance, and does not explicitly represent vegetation, but rather bases its evapotranspiration calculations on a prescribed potential evapotranspiration (PET) modulated by modeled soil moisture relative to the contents of multiple soil zones. Following Mitchell et al. (2004) we used the PET computed by Noah as an input for SAC. SAC’s snow model, Snow-17 (Anderson 1973), uses a degree-day formulation to simulate snow accumulation and melt. The surface and subsurface runoff simulated by each model was routed by a common external routing program, described by Lohmann et al. (1996, 1998).

b. Study basins and observations

We applied the individual models and multimodel methods to three river basins shown in Fig. 1: the Feather River at the Oroville Dam, California; the Salmon River at White Bird, Idaho; and the Colorado River above Grand Junction, Colorado. The Feather River basin has a drainage area of 9390 km². Naturalized monthly streamflow (upstream storage and diversion effects removed) at Oroville Dam (OROVI) was obtained for the period 1951–2005 from the California Data Exchange Center (CDEC) of the California Department of Water Resources. The Salmon River basin at White Bird has a drainage area of 35 100 km². Streamflow observations were taken from U.S. Geological Survey records for station 13317000 for the period 1951–2005. The Colorado River basin at Cameo (CAMEO) has a drainage area of 20 800 km². Discharge for the period 1951–2003, adjusted for the effects of upstream storage and diversion, was obtained from the U.S. Bureau of Reclamation (USBR).

Monthly average water budget terms are plotted in Fig. 2 for all basins. Row 1 shows basin-average observed precipitation; rows 2–4 show basin-average snow water equivalent (SWE), snowmelt, and evaporation, respectively, as simulated by each of the three models; and row 5 shows observed and simulated discharge at the gauge. The three models exhibit substantial variation in their simulated SWE, snowmelt, and evaporation (Fig. 2, panels 2a–4c) but because larger snowpacks tend to be accompanied by larger evaporative fluxes, the resulting calibrated flows (Fig. 2, panels 5a–c) are much more similar among the models, with the notable exception of Noah in the Feather basin (Fig. 2, panel 5b). For all the basins, the snowmelt peak occurs in April–May and plays a large role in the seasonal cycle of discharge (Fig. 2, panels 3a–c). However, the Colorado and Salmon River basins are more snowmelt-dominated than the Feather basin due primarily to their higher average elevations, combined with the fact that precipitation in the Feather basin has a much more distinct seasonal cycle than in the other two basins, with wet winters and dry summers (Fig. 2, panels 1a–c). Thus, in the Colorado and Salmon basins, the months of highest flow are May–July, corresponding to the snowmelt peak (Fig. 2, panels 5a and 5c); while for the Feather basin high flows are distributed over December – April and consist of a mix of rainfall and snowmelt (Fig. 2, panel 5b).

c. Meteorological inputs and model parameters

All models were applied at ⅛° latitude–longitude spatial resolution. The models were driven by a common set of meteorological forcings consisting of 56 yr (1949–2005) of daily maximum temperature, minimum temperature, precipitation, and wind speed for the Feather and Salmon basins and 54 yr (1949–2003) of the same inputs for the Colorado basin. The daily meteorological inputs were taken from Maurer et al. (2002) for the period 1949–2000 and extended to 2003 for the Colorado and to 2005 for the Salmon and Feather basins. To account for topographic effects, each grid cell was subdivided into up to five separate elevation bands, depending on the elevation range within the grid cell. Within each band, the air temperature was lapsed to the band’s average elevation using a lapse rate of 6.5°C km⁻¹. The same elevation bands were used in all models.

Initial model parameters were taken from the NLDAS experiment (Mitchell et al. 2004). All models were calibrated using the multiobjective complex evolution (MOCOM) algorithm (Yapo et al. 1998) to maximize simultaneously the Nash–Sutcliffe efficiency (NSE) and the correlation between model simulations and observations. While these calibrations yielded nontrivial reductions in RMSE relative to the NLDAS parameters, the objective function arguably is not ideally formulated (NSE contains a component that is sensitive to correlation, thereby rendering the dual objective functions somewhat redundant; see e.g., Murphy 1988; Gupta et al. 2009). For this reason, we performed an exploratory analysis, in which the multiple criteria in MOCOM were the three components of MSE (see section 4a and Gupta et al. 2009), over one of our test basins (the Feather River basin). We found that the results were qualitatively similar to those obtained using NSE and the model-observed correlations. We address some implications of calibration methods for our results in section 4c.

Calibrations were performed for the years 1976–90. This period was chosen to be representative of the range of temperature and precipitation found over the last half-century in the three basins. The VIC parameters calibrated were the baseflow parameters Ds, Ds_max, and Ws; the infiltration parameter b_inf; and the bottom-layer soil depth D3. For SAC, LZTWM, UZTWM, LZFPM, PFREE, LZSK, and ADIMP were calibrated based on the parameter sensitivity study by Tang et al. (2007). Additionally, the snow parameters MFMIN, MFMAX, and SCF were calibrated. For Noah, calibration was performed on the four soil layer depths and maximum snow albedo.

d. Bias correction

Our bias correction approach was to compute the anomalies of the model results with respect to their mean, rescale them so that their variance matched the variance of the observations, and add them to the mean of the observations. This is equivalent to a linear regression of the model results against the observations:

View Expanded

where X′_k is the kth model flow after bias correction, X_k is the kth model flow, Y is the observed flow, s_k is the standard deviation of flows of the kth model, s_Y is the standard deviation of observed flows, and a_k, b_k are equivalent linear regression parameters. We applied the bias corrections on a monthly basis.

e. Multimodel ensemble techniques

The multimodel ensemble average E can be defined as

View Expanded

where E is the ensemble average flow, X_k is the kth model flow, w_k is the weight of the kth model, and M is the number of models in the ensemble. Several strategies can be used for computing the model weights w_k. We examined the performance of two strategies in particular:

1. Simple model average (SMA): all w_k = 1/M.

2. Unconstrained multiple linear regression (MLR): w_k determined by multiple linear regression of observations against all models over a historical “training” period; w_k can be negative or >1.

Because among multimodel techniques SMA and MLR have the fewest and most degrees of freedom, respectively, they should bracket the range of multimodel performance. Similar to the bias correction, model weights vary on a monthly basis, resulting in 12 sets of model weights.

f. Training and validation periods

We evaluated the performance of the bias correction and multimodel ensemble-based estimates during a validation period separate from the training period used to estimate bias correction parameters (a_k, b_k) and ensemble model weights w_k. To minimize the differences in flow statistics between training and validation periods, we chose a training period consisting of the even years 1952–2004 and a validation period consisting of the odd years 1951–2005 (or 1952–2002 and 1951–2003, respectively, in the case of the Colorado River).

g. Probabilistic forecasts: ESP technique

To investigate relative performance of the methods for seasonal streamflow forecasting, we used the ESP method (Day 1985). This method generates an ensemble of hydrological simulations by resampling from sequences of historical meteorological inputs to the hydrological model, under the assumption that these meteorological inputs represent the distribution of possible conditions during the forecast period. In this study, forecasts were made for each month in the period 1951–2004 (1951–2002 in the Colorado basin), with lead times of 1–12 months. For each forecast/lead time combination, simulations started from the model state on the first day of the forecast start month, as estimated by the retrospective run using observed meteorology. An ensemble of simulations spanning from the start month to the forecast month was then generated from an ensemble of historical meteorological inputs composed of the same set of years, 1951–2004 (1951–2002 in the Colorado basin), excluding the year being forecast.

a. Retrospective simulations

We first examine the relative performance of the two methods in retrospective simulations. Monthly fractional RMSE (RMSE divided by observed mean flow) is plotted in Fig. 3 for individual models and multimodel averages, for each of the three study basins. The fractional RMSE values of “raw” results are shown in the first column, the fractional RMSE values of bias-corrected results are shown in the second column, and the differences between fractional RMSE values of bias-corrected and raw results are shown in the third column. The fractional RMSE values of raw results tend to be smaller in high-flow months (April–September in the Colorado and Salmon River basins; January–June in the Feather River basin).

Considering first the RMSE of the raw results (column a), we see considerable spread in the individual model results, with no single model consistently outperforming the others (although VIC and SAC tend to perform better than Noah). It is clear that in general, the best-case multimodel average (MLR) performs at least as well as, and sometimes substantially better than (particularly in the Salmon basin), the best raw model output, with the exception of month 11 in the Feather basin. The SMA multimodel average, on the other hand, never substantially outperforms the best raw model, and often performs much worse. In contrast, comparison of the fractional RMSE values after applying the monthly bias correction (columns b and c) shows that in general, the bias correction reduces RMSE values of not only individual models but also multimodel averages, with some exceptions in the Feather basin in months 1, 2, 4, and 11. These reductions in RMSE are mostly at least as large, if not larger, than those yielded by the MLR multimodel average of raw model results (column a). In the Colorado and Salmon basins, these improvements range from 0% to approximately 50% of the original RMSE in high-flow months (April–September), and approximately 50%–100% in low-flow months (October–March). In addition, after bias correction, the spread of RMSE values (column b) is much narrower than for raw values (with the exception of month 11 in the Feather basin, which may be the result of nonstationarity, discussed further in section 4b). Not only is the bias-corrected MLR multimodel average rarely better than the best bias-corrected model (and only marginally better in those cases), but all bias-corrected models and multimodel averages (with the exception of Noah in the Feather River basin), become very similar after bias correction.

b. ESP forecasts

We also evaluated the performance of bias correction and multimodel averaging with respect to the behavior of the ESP ensemble mean [the ensemble spread is also a useful metric worth investigating (e.g., Weigel et al. 2008), but we thought it important first to understand the behavior of the ensemble mean, which influences the bias of the forecasts]. We evaluated forecast RMSEs for each start month/forecast month pair over all 27 (Salmon and Feather basins) or 26 (Colorado basin) validation year forecasts. The parameters used in the bias correction and multimodel averages were the same as those used in section 3a for the simulation comparisons. Figure 4 shows the fractional RMSE of ESP ensemble means as a function of simulation start month and forecast month, for the three study basins, for VIC (chosen to represent the behavior of a typical individual model), with and without bias correction and the two monthly multimodel averages without bias correction. For each basin, the fractional RMSE of each model or multimodel average is displayed in the first row and differences between these fractional RMSEs and those of the raw VIC results are displayed in the second row. In each plot, the y axis corresponds to forecast month and the x axis corresponds to start month. For reference, the 1:1 line, along which the start month is equal to the forecast month and the lead time is equal to 0, is drawn in each plot. Forecasts along this diagonal incorporate the smallest uncertainty in model forcings, and therefore forecast errors along this diagonal approach those of the retrospective simulations (section 3a). Just to the right of this diagonal, the lead time is 12 months, for which model forcing uncertainty is greatest.

Two main features are evident in the raw model results, represented by “VIC_RAW” in the first column of Fig. 4 (all three raw models exhibit the same general patterns). For forecast months that are not dominated by snowmelt (October–March in all basins), fractional RMSE values are relatively high and vary little with lead time, except for small variations (decreasing or increasing) at lead times of 1–2 months. For snowmelt-dominated forecast months (April–September in all basins), fractional RMSE is relatively high and constant with lead time before the start month of December or January (when snowfall begins to accumulate), at which point it begins a steady decline until the start month equals the forecast month (lead time of 0). This is an inherent quality of ESP forecasts in snowmelt-dominated basins, as noted by Pagano et al. (2004) and others. At long lead times, the influence of the forecast’s initial conditions is small, and climatology dominates the forecast. In nonsnowmelt-dominated forecast months, forecast skill changes appreciably only at short (1–2 months) lead times (this can be an increase or decrease, depending on whether the short-lead-time forecast is better than climatology). In contrast, in those forecast months dominated by snowmelt, forecast skill begins to increase for those start months when the snowpack has begun to form, and increases steadily with decreasing lead time as the snowpack matures. This can lead to appreciable forecast skill with lead times up to 6 months in some cases.

Columns 2–4 of Fig. 4 illustrate the differences in performance between bias correction of the VIC model (denoted by “VIC_BC”) and multimodel averaging, in terms of fractional RMSEs of ESP means. In the Colorado and Salmon basins, both bias-corrected VIC and raw MLR tend to reduce RMSE substantially (50%–75%) in nonsnowmelt-dominated forecast months at all lead times. In snowmelt-dominated forecast months (which correspond to the highest-flow months in these basins), the reduction in RMSE is smaller and depends on lead time, with short-lead-time reductions of 0%–25% generally decreasing to near zero for start months before December. In the Feather basin, a similar small lead-time-dependent reduction is evident in snowmelt-dominated forecast months (which form a subset of the highest-flow months in this basin), but in nonsnowmelt-dominated forecast months, fractional RMSE is relatively high to begin with and bias-corrected VIC and raw MLR have little effect on RMSE (this may also be a result of nonstationarity in the Feather basin, discussed further in section 4b). The behaviors displayed by bias correction and multimodel averaging in snowmelt-dominated forecast months come about because errors in the raw model results are the combination of two factors: lead-time-independent climatological bias and lead-time-dependent errors from snowpack state. While bias correction and MLR show similar dependence on forecast month and lead time, bias-corrected VIC tends to perform as well as or better than MLR in most forecast month–start month combinations; for snowmelt-dominated forecast months, the difference between bias-corrected VIC and MLR tends to increase as lead time decreases. In all basins, raw SMA rarely outperforms raw VIC, and sometimes performs worse. As with the retrospective case, after bias correction, both MLR and SMA, as well as the other two individual models, all look very similar to bias-corrected VIC (and therefore have been omitted from the plot). Thus, applying bias correction to a single model tends to yield benefits that are similar to or greater than those produced by multimodel averaging.

a. Components of MSE

The primary reason why bias correction yields improvements as good as or better than multimodel averaging is because bias correction addresses, more directly than multimodel averaging, the predominant types of errors found in monthly data—at least for the three river basins examined in this study. Mean-square error (MSE) and its square root (RMSE) have been used as metrics in many studies (including ours). It can be shown (e.g., Murphy 1988; Gupta et al. 2009) that, if MSE is normalized by the variance of the observed discharge, this normalized MSE can be decomposed into separate terms corresponding to the contributions of bias, correlation, and amplitude:

View Expanded

where MSE_k is the mean-square error of model k with respect to observations Y, B_kY is the bias of model k with respect to observations (=X_k − Y), r_kY is the correlation of model k and observations Y, s_Y is the standard deviation of observations Y, and s_k is the standard deviation of model k. In other words, a model’s normalized MSE consists of a bias term, , a correlation term, , and an amplitude term, (s_k/s_Y − r_kY)². All of these terms have minimum values of 0; the correlation term has a maximum of 1 while the other terms have no upper limit. Thus, a model’s MSE can be minimized by 1) reducing its bias, 2) increasing its correlation with observations, and/or 3) adjusting its amplitude s_k to match s_Yr_kY as closely as possible. As noted by Gupta et al. (2009), minimizing a model’s MSE is therefore equivalent to minimizing the sum of these three terms, two of which (the correlation and amplitude terms) are linked. The relative impacts of bias correction and multimodel averaging will therefore depend on how each method handles the different components of MSE.

Figure 5 displays the fractions of total monthly normalized MSE contributed by the bias, correlation, and amplitude components, for the retrospective simulations in the three basins used in this study, for VIC (a “representative” individual model; row 1) and the two multimodel averages (rows 2 and 3), with and without monthly bias correction. To facilitate comparisons among the different processing techniques, fractions are relative to the total monthly normalized MSE of raw VIC. For ease of comparison, a horizontal dashed line is drawn at 1.0 across all panels to indicate this value. Here again the Feather basin exhibits different behavior from the other two basins, particularly in months 9 and 11, possibly due to nonstationarity (discussed further in section 4b).

For the individual model (VIC), before bias correction (row 1, columns a, c, and e), the bias and amplitude terms often form the majority of normalized MSE. In the Colorado and Salmon basins (panels 1a and 1e), bias and amplitude components dominate in the majority of months, although in the Feather they dominate in only three of the months (panel 1c). The partitioning of normalized MSE into its components exhibits some dependence on flow volume; as a result of inverse dependence on observed variance [Eq. (3)], which can be quite small in lower-flow months, the bias and amplitude terms are more likely to dominate total normalized MSE in those months. After bias correction (row 1, columns b, d, and f), the bias term is essentially eliminated and the amplitude term is substantially reduced (although the bias correction actually increased the amplitude term in some months in the Feather basin), with no change in the correlation component. One notable exception is month 11 in the Feather River basin, in which the bias correction worsened performance in all three components of normalized MSE (possible nonstationarity in the Feather basin is discussed in section 4b). Other individual models exhibit the same general behavior as VIC and are omitted here for brevity.

In contrast, the multimodel averages of raw model results (rows 2–3, columns a, c, and e) generally do not reduce the bias term as much as the bias correction does, and give mixed results for the amplitude term. Indeed, the simple model average (SMA) often increases the bias term and both SMA and MLR often increase the amplitude term. Unlike bias correction, multimodel averages can effect some change in the correlation term, but the change is usually small and not always a reduction. Applying multimodel averages to bias-corrected models (rows 2–3, columns b, d, and f) yields results similar to applying bias correction to the best individual model (row 1, columns b, d, and f), that is, little or no bias and a greatly reduced amplitude term, but with a slightly changed correlation term.

Thus, it appears that bias correction owes its success, relative to multimodel averaging, to its superior performance in reducing the bias and amplitude components of error. This should not come as a surprise; bias correction methods are specifically designed to remove these types of error from an individual model, independently of any other models. In each month, our bias correction adjusts the mean and rescales the standard deviation [Eq. (1)] and therefore directly adjusts both the bias and amplitude terms of normalized MSE [Eq. (3)], but has no effect on the correlation term [as an aside, we also tested a quantile-mapping bias correction as used by Snover et al. (2003) rather than the simple scaling approach we used, and found that it too had minimal impact on model–observation correlations]. It should be noted that our bias correction, in adjusting the model standard deviation s_i to match the observed standard deviation s_Y, will not in general minimize the amplitude term of Eq. (3); ideally the bias correction should adjust s_i to match s_Yr_iY. Still, the bias correction we have employed tends to reduce the amplitude component of error substantially.

Multimodel averaging, on the other hand, has varying success because 1) it only has one parameter per model (the model’s weight) and 2) its ability to reduce the bias and amplitude terms depends on the relationships among the individual models, which may or may not favor simultaneous elimination of both the bias and amplitude terms. Expressing the components of normalized MSE for the multimodel average in terms of the individual models gives

View Expanded

where ‖𝗥‖ is the norm of the weighted model covariance matrix:

View Expanded

In the case of SMA, the model weights are fixed and thus the ability of SMA to reduce MSE depends heavily on the particular set of models in the average. In the case of MLR, the model weights can be chosen with no constraints; therefore we should expect MLR to outperform SMA in general. However, the model weights that minimize bias will not in general minimize the other two error terms. Consequently the model weights that minimize total MSE may not reduce bias as dramatically as will a bias-correction scheme.

We see the same behavior in the means of ESP forecasts. At long lead times, the model results are dominated by model climatology. Because model climatology is the same for all forecasts of a given month, long-lead-time forecasts have small variances and essentially no correlation with observations. Because the amplitude and correlation components depend on these two quantities, they are relatively insensitive to postprocessing when dominated by climatology. Thus, for long-lead-time forecasts, the only error component that can be appreciably reduced by postprocessing techniques is the bias arising from the model climatology. The method that reduces this bias most effectively will yield the most accurate forecasts at long lead times. Our results indicate that bias correction is more effective than multimodel averaging in this respect. Similarly, as lead time diminishes, the forecast errors begin to resemble those of the retrospective simulations, which, as we have already seen, are dominated by the bias and amplitude terms, which are reduced most effectively by bias correction.

The benefits of multimodel averaging shown in the retrospective simulations are similar to those suggested by other hydrologic studies. For example, Ajami et al. (2006) found that multimodel averages of uncalibrated models, based on multiple linear regression, exhibited substantial improvements in RMSE (8%–16%), but with little or no improvements in correlation relative to the best individual model. Incorporating some form of bias correction into the multimodel averages further improved RMSE to some degree (0%–10%), but again had little effect on correlation. These results imply that the types of errors being addressed by the multimodel averages and bias corrections were bias or amplitude components, rather than the correlation component. While Ajami et al. (2006) did not examine the performance of individual bias-corrected models, it seems plausible that applying bias corrections to the individual models would have produced results that were competitive with the bias-corrected multimodel averages, given that the multimodel averages did not have much impact on correlations.

b. Monthly data and model cross correlations

It should not be a surprise that bias correction outperforms multimodel averaging in the reduction of bias. But why did the multimodel averages perform so poorly in reducing the correlation and amplitude components [i.e., the second and third terms on the right-hand side of Eq. (4)]? While the number and choice of models in the ensemble likely played a role, a primary reason was our analysis of monthly data.

The correlation and amplitude terms depend on the models’ correlations with each other and with the observations. Winter and Nychka (2009) provide a useful geometric analysis of the dependence of multimodel performance on the relationships among the member models. If we represent model errors as vectors in T dimensions (where T is the number of points in the time series), the multimodel average will outperform the best individual model to the degree that the set of member models samples the error space evenly (i.e., the models’ error magnitudes must be somewhat similar, but their directions in error space should be as diverse as possible, so that their average is as close to the origin as possible). The larger the region of error space these error vectors span (i.e., the less collinear the models are), the greater the possible gains. This is independent of the number of models in the ensemble, although sampling of a larger span of error space is more likely with a larger number of models.

Techniques commonly applied in seasonal hydrologic forecasting, and which we have applied here, can increase the collinearity of the models and reduce the effectiveness of a multimodel average, relative to other hydrologic applications. Most prominent among these techniques is our aggregation of the model results to a monthly time scale. Other important factors are our use of methods with monthly varying parameters and our consideration of relatively large snowmelt-dominated river basins. For example, Ajami et al. (2006) used hourly discharge from river basins that were somewhat smaller than those that we studied (all <10 000 km²); the multimodel averages and bias corrections they applied involved time-invariant parameters. Under such conditions, intermodel differences can be expected to include significant shifts in the timing of runoff peaks and the overall shapes of the hydrographs on a daily and subdaily scale; time-invariant bias correction would preserve these differences, and time-invariant multimodel averaging would presumably have the benefit of relatively small correlations among the model errors. In contrast, we considered monthly discharge from larger (mostly >10 000 km²) basins, and used time-varying (monthly) bias corrections. Aggregating observed and simulated discharge to a monthly interval reduces intermodel differences mostly to differences in the monthly bias and the amplitude and correlation of interannual variability. To the extent that the interannual variability of simulated streamflow is sensitive to interannual variability in the meteorological forcings common to all models (this sensitivity should be strong in snowmelt-dominated basins, where the previous few months’ accumulation of snow dominates the peak seasonal flows), any technique that emphasizes the models’ interannual variability will increase model collinearity. Additionally, because monthly bias is the most important component of simulation and forecast errors, any technique (e.g., bias correction with monthly varying parameters) that reduces the bias of each month separately should yield substantial improvements, which appears to be a major factor in our results.

We can summarize the degree of model collinearity using the norm of the monthly model cross-correlation matrix ‖𝗿‖ (not to be confused with weighted model covariance matrix ‖𝗥‖):

View Expanded

The closer ‖𝗿‖ is to unity, the more highly cross correlated the models are.

To illustrate the impact of aggregation from daily to monthly time scales, we plot in Fig. 6 monthly ‖𝗿‖ computed from both the original daily discharge and the monthly aggregate discharge, for the three study basins. It is readily apparent that aggregating discharge from daily to monthly time scales almost always increased model collinearity, with especially dramatic results in the Feather basin. The only cases in which the aggregation decreased model collinearity were month 4 in the Colorado basin, months 8 and 9 in the Feather basin, and months 3, 4, and 12 in the Salmon basin. All of these months are low-flow months, in which the bias and amplitude components composed the dominant sources of error anyway (and thus the bias correction could still remove the bulk of the errors).

The use of monthly varying parameters had an impact on performance as well. This is explored further in Fig. 7, which plots grand (as opposed to monthly) fractional RMSE of individual models and multimodel averages, with and without bias correction. Both the multimodel averages and the bias correction were applied in two ways: constant parameters (fitted to minimize the MSE of the whole time series) and monthly varying parameters (fitted to minimize the MSE of each month). Figure 7 shows that for the Colorado and Salmon River basins, both the MLR multimodel average with monthly varying weights (column c) and the monthly varying bias correction (column d) yielded substantially larger reductions in errors than their time-invariant counterparts (columns a and b). Surprisingly, in the Feather basin, monthly varying parameters performed worse than constant parameters; in fact, the constant MLR average yielded the best results of any method. This was not the case in the training period. One explanation for this may be a lower degree of stationarity in the Feather basin, resulting from either nonstationary climate or an artifact of naturalizing flows to account for the large reservoir at the basin outlet. Under such conditions, one must be careful to avoid overfitting the parameters to the training data. Certainly, the monthly varying bias correction and multimodel averages have 12 times the number of parameters of their time-invariant counterparts. In this respect, the constant-parameter bias correction and multimodel averages may be more robust, statistically, than monthly varying methods. This would be consistent with the findings of Ajami et al. (2006), Gao and Dirmeyer (2006), and Duan et al. (2007). If nonstationarity is a significant concern and we restrict ourselves to time-invariant parameters, a multimodel average may be superior to a bias correction alone because it still can address the relative values of adjacent monthly data via the month-to-month differences between individual models.

c. Model calibration

Calibration can also impact model collinearity. Ajami et al. (2006) used uncalibrated models to form their multimodel averages. Presumably uncalibrated models would tend to be less similar than calibrated models (which we have used in this study) and therefore multimodel averages formed from them would yield more benefit than multimodel averages of calibrated models. Indeed, had we specifically designed our calibration scheme to maximize the diversity of monthly model errors, or to minimize monthly bias (note that this is not the same as the “grand” bias of the entire time series), the relative performances of the bias correction and multimodel averaging might have been affected.

Nevertheless, regardless of the calibration scheme, our use of monthly data and monthly varying parameters would have had the same relative effect that we report above: the models become more collinear and the effectiveness of the multimodel average, relative to a bias correction, is diminished by temporal aggregation. Indeed, when we used the uncalibrated NLDAS parameters for the models in this study (not shown), the relative performance of bias correction and multimodel averaging was the same (indicating that the main component of error in the uncalibrated models was also monthly bias). Similarly, Shi et al. (2008) examined monthly streamflow forecasts based on the VIC model in several western U.S. river basins and found that bias correction was nearly as effective as model calibration in improving seasonal forecast skill. Furthermore, the multimodel techniques employed by Ajami et al. (2006) on hourly data did not have a large effect on correlations, even for uncalibrated models. In practice it may be difficult to find a sufficiently diverse set of models for which the multimodel average substantially outperforms a bias correction, at least in the context of seasonal streamflow forecasts.

d. Differences between hydrologic and atmospheric multimodel ensembles

There are important differences between hydrologic multimodel ensembles as evaluated in our study and atmospheric multimodel ensembles such as those studied by Krishnamurti et al. (1999, 2000) that we think explain the failure of the multimodel ensemble approach to yield simulation and forecast accuracy improvements in the hydrologic case. The most important of these is that, while atmospheric models simulate chaotic systems, and therefore are susceptible to large differences in model state even when initial states are similar, hydrologic models simulate damped systems, in which large perturbations in initial state exert diminishing influence over time. In particular, the interannual variability of simulated streamflows in snowmelt-dominated months should have a high correlation with that of observed flows because the snowpack state is essentially reset every summer. Each model’s snowpack therefore has little memory of the previous year’s state, but rather depends strongly on the current winter’s precipitation, which is a common input to all models. Thus, simulated discharge from multiple hydrologic models tends to be more collinear, at least in snowmelt-dominated months (as seen in Fig. 6), than for atmospheric simulations, reducing potential forecast skill gains of multimodel averages of hydrologic models.