Abstract
The ensemble spread is often used as a measure of forecast quality or uncertainty. However, it is not clear whether the spread is a good measure of uncertainty and how the spread–error relationship can be properly assessed. Even for perfectly reliable forecasts the error for a given spread varies considerably in amplitude and the spread–error relationship is therefore strongly heteroscedastic. This implies that the forecast of the uncertainty based only on the knowledge of spread should itself be probabilistic.
Simple probabilistic models for the prediction of the error as a function of the spread are introduced and evaluated for different spread–error metrics. These forecasts can be verified using probabilistic scores and a methodology is proposed to determine what the impact is of estimating uncertainty based on the spread only. A new method is also proposed to verify whether the flow-dependent spread is a realistic indicator of uncertainty. This method cancels the heteroscedasticity by a logarithmic transformation of both spread and error, after which a linear regression can be applied. An ensemble system can be identified as perfectly reliable with respect to its spread.
The approach is tested on the ECMWF Ensemble Prediction System over Europe. The use of spread only does not lead to skill degradation, and replacing the raw ensemble by a Gaussian distribution consistently improves scores. The influences of non-Gaussian ensemble statistics, small ensemble sizes, limited predictability, and different spread–error metrics are investigated and the relevance of binning is discussed. The upper-level spread–error relationship is consistent with a perfectly reliable system for intermediate lead times.
