a. Numerical setup

We study the behavior of $\overline{\text{BDnSS}}$ in a series of realistic idealized experiments controlled by explicit key parameters. This more theoretical framework will enable us to verify that the results are consistent and provide relevant information on the quality of ensemble predictions.

The spatial domain contains 400 × 800 points. Data are simulated using R (R Core Team 2019) and geoR (Ribeiro et al. 2020) libraries. The observations are obtained by adding to the common background a complement that is more difficult to predict and which is drawn randomly from a centered normal distribution $N(0,{\sigma }_o^2)$, where σ_o is its standard deviation. The draws of the observation complement are performed independently at each point and time from the single distribution $N(0,{\sigma }_o^2)$. Figure 1 shows two examples of observation fields constructed with σ_c = 1 and σ_o = 0.2 and for values of L_c corresponding to 80 and 2 grid points. We note that the visible structures have sizes on the order of a few L_c. Thus, the field for L_c = 2 looks like white noise and appears much noisier than the one obtained for L_c = 80. The field for L_c = 80 looks plausibly like a meteorological field of rain or cloud.

View Full Size

Observation spatial field at one time constructed with a statistical distribution corresponding to σ_c = 1, σ_o = 0.2, and (a) L_c = 80 and (b) L_c = 2. A black circle with a radius equal to L_c is drawn at the center of the domain. Forecast spatial field of one member of the ensemble forecast built at the same time and with the statistical distribution corresponding to σ_c = 1, μ_f = 0.0, σ_f = 1.0, and (c) L_c = 80 and (d) L_c = 2.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

The probabilistic forecasts are constituted by an ensemble forecast of 35 members. Each of the 35 forecasts is obtained in the same way as the observations: we add to the common background a complement drawn randomly from a normal distribution $N({\mu }_f,{\sigma }_f^2)$, where μ_f is the value of its mean and σ_f its standard deviation. The draws of the forecast complement are performed independently at each point and time from the single distribution $N({\mu }_f,{\sigma }_f^2)$. For the sake of simplicity, the forecasting error is modeled using a homoscedastic approach to limit the number of parameters needed to control it (μ_f and σ_f are the same for all grid points at all times). The forecast of one member of the ensemble forecast is shown in Figs. 1c and 1d for the same day as the observation discussed just before, using the same common part and a complement drawn from the $N({\mu }_f=0,{\sigma }_f^2=1.0)$ distribution. We observe that the forecast complement is more visible on the total field than the observation complement, since it is on average 5 times stronger. For L_c = 80, we can visually find the common part in Figs. 1a and 1c, whereas the fields are too noisy to do so for L_c = 2 in Figs. 1b and 1d. To have statistically stable results, we iterate the process of generating observations and forecasts to constitute a set of 100 realizations.

b. Reference experiment EXP_ref

The reference experiment EXP_ref corresponds to the choices: L_c = 80, σ_c = 1, and σ_o = 0.2 for the observations and μ_f = 0 and σ_f = 0.2 for the forecasts. It corresponds to correlated observations over large distances to be compared with an unbiased probabilistic forecasting system giving forecast errors on average nearly 5 times smaller than the variability of the observations. Note that the forecasts of this reference experiment are drawn in the same distribution as the observations since μ_f = 0 and σ_f = σ_o. The chosen event is “the parameter being greater than 1.72”; this threshold has been chosen so that the observation mean frequency of occurrence is equal to 4.5%.

We compute the $\overline{\text{BDn}}$ on the whole domain of 400 × 800 points for 100 realizations for different values l_n of the neighborhood length scale (related to the number of points in a square neighborhood by $N=l_n^2$). We use the forecast frequencies ${\overline{fn}}_k$ in order to build a sharpness diagram (Jolliffe and Stephenson 2011) for the 36 bins (Fig. 2a).

View Full Size

Experiment EXP_ref: (a) sharpness diagram formed by plotting the number n_k of forecast cases falling in bin k against the central predicted frequency of bin k for different neighborhood length scales l_n: 1, 3, 5, 9, 15, 21, 35, and 49 grid points. The number of bins is 36 since the ensemble forecast contains 35 members. The vertical scale of the plot is logarithmic. (b) Variations of the terms of the decomposition of $\overline{\text{BDn}}$ are plotted as a function of the neighborhood length scale l_n: the $\overline{\text{BDn}}$ in a black solid line, RES in a green dotted line, WBV in a brown dotted line, WBC in an orange dotted line, REL in blue dashed line, and UNC in a red solid line. (c) Variations as a function of the neighborhood length scale l_n of $\overline{\text{BDnSS}}$ (full black line), GRES/UNC (dotted red line), and REL/UNC (dashed red lines) are plotted. Each symbol (disk or “x”) corresponds to a numerical experiment.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

The sharpness diagram shows that all bins of the predicted probability of the event are used. When considering larger and larger neighborhood sizes, the number of cases in each bin decreases because it is nearly proportional to the inverse of the number of points of the neighborhood $l_n^2$. This is due to the use of disjoint neighborhoods. We also note that the variation of n_k with k becomes noisier for large values of l_n because the bins are less populated.

The term $\overline{\text{BDn}}$ and the different terms of its decomposition are plotted in Fig. 2b. UNC, which can also be written as $\text{UNC}=\overline{{(on-\overline{on})}^2}$, represents the variance of on. UNC decreases with l_n because the pooling operation reduces the variability of the frequencies averaged over the neighborhood. The resolution term RES exactly follows the same decrease and quantifies the quality of the forecast that is able to predict the variation of the observations by construction. We can also note that the variability terms within the bin WBV and WBC become relatively more and more important with respect to RES as l_n increases but remain negligible against RES. REL is also negligible compared to the main terms in the decomposition for all neighborhoods used because drawing forecasts and observations from the same statistical distribution prevents reliability errors. The term $\overline{\text{BDn}}$ decreases with l_n because GRES and UNC are closer together the larger l_n is.

The observed data are spatially correlated over distances of the order of L_c and this correlation is fully reflected in the ensemble forecast by construction. Thus, the forecasts of the neighboring points are informative for the forecast at the central point. This is the basic postulate of the neighborhood technique which can be applied: the frequencies averaged over the neighborhood fn and on of size less than L_c will then be closer than their local values f and g at the center of the neighborhood. This leads to a reduction of $\overline{\text{BDn}}$ that is the sum of the squares of these differences and therefore an improvement of the forecast accuracy at the neighborhood scale.

We turn to $\overline{\text{BDnSS}}$ and its decomposition in terms of resolution and reliability. For EXP_ref, $\overline{\text{BDnSS}}$ increases with l_n (Fig. 2c) and is always nearly equal to GRES/UNC while REL/UNC is negligible. This presentation in terms of relative Brier distances highlights the relative importance of REL/UNC and GRES/UNC in the $\overline{\text{BDnSS}}$ variation as a function of l_n for EXP_ref.

We tested the impact of varying all disjoint neighborhoods by a few points in both directions on the computation of $\overline{\text{BDn}}$ for nine couples of values and found a standard deviation for $\overline{\text{BDn}}$ and decomposition terms on the order of 1% for the nine realizations when choosing l_n = 21. The use of a sliding window (with overlapping neighborhoods) gives estimates of $\overline{\text{BDn}}$ and all the decomposition terms which are close to the averages of the nine realizations using disjoint neighborhoods and are separated from them by less than half of the standard deviation of the nine realizations.

c. Variation according to the standard deviation σ_f

In a series of experiments, we vary the standard deviation of the forecast from σ_f = 0 to σ_f = 1. The other parameters are similar to those of EXP_ref: L_c = 80, σ_c = 1.0, σ_o = 0.2, and μ_f = 0. The σ_f = 0 corresponds to a deterministic forecast, since the dispersion of the ensemble is zero and the forecast is always equal to the background common to observations and forecasts. The σ_f = 0.2 corresponds to EXP_ref. The σ_f = 1 corresponds to a greater dispersion of forecasts than for EXP_ref and is on the same order of magnitude as the variability of the common background.

We define the frequency bias Bn at the neighborhood scale by $Bn=\overline{fn}/\overline{on}$. The term Bn is independent of the neighborhood size l_n. The term Bn increases monotonically with σ_f, as shown in Fig. 3b, from Bn = 0.93 for σ_f = 0 to Bn = 2.39 for σ_f = 1. To understand this underprediction (respectively, overprediction) of the event for σ_f < 0.2 (respectively, σ_f > 0.2) compared with the unbiased EXP_ref forecast, we must return to its construction: The forecasts are obtained at each point by adding two random variables, one being the common background drawn from the distribution using an exponential covariance model with the spatial distance and the other being the forecast complement drawn from the distribution $N(0,{\sigma }_f^2)$. The forecast then follows a distribution narrower (respectively, wider) with respect to the distribution of observed values. This leads to a systematic underprediction (respectively, overprediction) of events since the predicted distribution tends to 0 more (respectively, less) quickly for high values of the parameter than the observed distribution.

View Full Size

(a) Variation of the $\overline{\text{BDnSS}}$ as a function of the neighborhood length scale l_n for experiments using the same parameters as EXP_ref except σ_f = 0 (orange line), σ_f = 0.1 (yellow line), σ_f = 0.2 (black line for EXP_ref), σ_f = 0.3 (green line), σ_f = 0.6 (red line), and σ_f = 1.0 (blue line). Variations as a function of σ_f for $\overline{\text{BDn}}/\text{UNC}$ (black solid line), 1 − GRES/UNC (red dotted line), REL/UNC (blue dashed line), and Bn (green dotted-dashed line) are plotted for (b) l_n = 1 and for (c) l_n = 49. The vertical scale of (b) and (c) is logarithmic.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

The evolutions of $\overline{\text{BDnSS}}$ with l_n are superimposed for all σ_f values in Fig. 3a. As with EXP_ref, $\overline{\text{BDnSS}}$ increases with l_n but taking into account the neighborhood does not allow the improvement of $\overline{\text{BDnSS}}$ (Fig. 3a) beyond a maximum value which corresponds to σ_f = 0.2 or to EXP_ref. The EXP_ref corresponds to using the same statistical distribution for the forecasts as for the observations, since σ_o = 0.2. The $\overline{\text{BDn}}$ is therefore minimal in this configuration because BDn is a proper score and $\overline{\text{BDnSS}}$ is maximal for the same reason.

d. Variation according to the range L_c

The range parameter L_c was equal to 80 grid points for EXP_ref; we now vary this parameter by taking smaller values: 2, 8, 16, and 32 grid points while keeping the other parameters equal to those of EXP_ref: σ_c = 1, σ_o = 0.1, μ_f = 0, and σ_f = 0.2. Decreasing L_c corresponds to taking observations correlated over smaller distances. The $\overline{\text{BDnSS}}$ is plotted as a function of l_n for these five experiments in Fig. 4a. The five $\overline{\text{BDnSS}}$ variations are similar with a saturation of $\overline{\text{BDnSS}}$ toward a value close to 1 except for L_c = 2 where the asymptotic value is lower.

View Full Size

(a) Variation of the $\overline{\text{BDnSS}}$ on as a function of the neighborhood length scale l_n for experiments where the range parameter L_c is varied: L_c = 80 for EXP_ref (black line), L_c = 32 (green line), L_c = 16 (purple line), L_c = 8 (orange line), and L_c = 2 (blue line). Variations as a function of L_c for $\overline{\text{BDn}}/\text{UNC}$ (black solid line), 1 − GRES/UNC (red dotted line), REL/UNC (blue dashed line), and Bn (green dotted-dashed line) are plotted for (b) l_n = 1 and (c) l_n = 49. The vertical scale of (b) and (c) is logarithmic.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

We use the decomposition of $\overline{\text{BDn}}/\text{UNC}$ in order to discuss the variation of the resolution and reliability terms with L_c for the extreme values l_n = 1 and l_n = 49 (Figs. 4b,c). REL/UNC remains negligible compared with 1 − GRES/UNC for all these experiments, as they are all unbiased for the event under consideration (Bn = 1 is also plotted in Figs. 4b,c). In fact, forecasts and observations are drawn from the same statistical distribution for each experiment. When l_n = 1, there is no variation in the normalized scores with L_c. This is no longer the case for l_n = 49, since taking into account forecasts and observations correlated over greater or lesser distances influences the value of 1 − GRES/UNC and therefore $\overline{\text{BDn}}/\text{UNC}$ and $\overline{\text{BDnSS}}$. Note that it is the GRES resolution term that improves faster than UNC as L_c increases. This shows that these forecasts are more informative than climatology when L_c increases for a given verification scale l_n.

e. Variation according to the bias of the forecast

Two complementary experiments EXP_B+ and EXP_B− were performed in order to document the impact of a prediction additive bias by taking distributions of the prediction complement given by N(+0.1, 0.2²) and N(−0.1, 0.2²), respectively, and a distribution of the observation complement given by N(0.0, 0.2²). All the other parameters are the same as for EXP_ref. The positive or negative additive bias corresponds nearly to ±10% of the standard deviation of the observations. We quantify the impact of these additive biases by calculating for the three experiments the frequency bias Bn for the event corresponding to the threshold 1.72: Bn = 1.22 for EXP_B+, Bn = 1.0 for EXP_ref, and Bn = 0.81 for EXP_B−.

For the biased experiments, we observe $\overline{\text{BDnSS}}$ grows as a function of l_n as for EXP_ref but the maximum values reached are lower (Fig. 5a). The $\overline{\text{BDnSS}}$ values of experiments EXP_B+ and EXP_B− are close. We have also plotted in Fig. 5a the evolution of $\overline{\text{FSS}}$ for the three experiments as a function of the length scale of the neighborhood l_n. We see that $\overline{\text{FSS}}$ has the same behavior as $\overline{\text{BDnSS}}$ with better scores for the unbiased experiments and also a saturation for the largest neighborhoods. The $\overline{\text{FSS}}$ values are stronger than $\overline{\text{BDnSS}}$ because the normalization used by $\overline{\text{FSS}}$ is easier to beat than the climatological forecast used by $\overline{\text{BDnSS}}$.

View Full Size

(a) Variation of the $\overline{\text{BDnSS}}$ (full lines) and $\overline{\text{FSS}}$ (dashed-dotted lines) as a function of the neighborhood length scale l_n for experiments EXP_ref (black line), EXP_B+ (orange line), and EXP_B− (green line) for L_c = 80. (b) The same parameters are plotted for the same experiments except L_c = 2. Variations of the terms of the $\overline{\text{BDnSS}}$ decomposition as a function of the neighborhood length scale l_n: (c) GRES/UNC and (d) REL/UNC in full lines for L_c = 80 and dotted lines for L_c = 2. Each symbol corresponds to a numerical experiment.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

The $\overline{\text{BDnSS}}$ decomposition reveals which term is responsible for this improvement as l_n increases. The GRES/UNC for the three experiments remains very close when l_n increases (Fig. 5c) because the three predictions follow by construction the observation when it varies, in spite of the systematic shift when the bias is different from 0. The GRES/UNC improves as l_n increases, while REL/UNC remains almost constant (Fig. 5d). We can conclude that it is still the resolution term that is the source of the improvement in forecast performance over climatological forecasting, thanks to the correlation of forecast values within the neighborhood when l_n < L_c.

We now consider the case of biased forecasts for which we take a neighborhood size l_n for verification that is large compared with L_c. We choose L_c = 2 and plot $\overline{\text{BDnSS}}$ as a function of l_n in Fig. 5b. The $\overline{\text{BDnSS}}$ variations are very important in comparison to the case L_c = 80 only for the biased experiments since the saturation of $\overline{\text{BDnSS}}$ toward a value close to 1 is replaced, for the range of l_n explored, by a growth toward a maximum for low values of l_n followed by a decrease for high values of l_n. For L_c = 2, the maximum of $\overline{\text{BDnSS}}$ is reached for a value of l_n = 7 (value obtained thanks to complementary experiments to those plotted in Fig. 5). The decrease in $\overline{\text{BDnSS}}$ for high l_n values is explained by the inclusion in the large neighborhoods of uncorrelated observations whose average effect in terms of averaged frequency of observations on is to reduce the variability by getting closer to the climatological value. The expression $\overline{\text{BDn}}(\text{climatology})=\text{UNC}$ tends therefore toward 0 for very large neighborhoods for L_c = 2. This tendency to get closer to the climatology also applies to forecasts that follow observations by construction even when they are systematically offset. This is quite visible in the sharpness diagram for L_c = 2 (Fig. 6) giving the distribution of the predicted frequencies fn averaged in the neighborhood that cluster around the climatological frequency of the considered event.

View Full Size

Sharpness diagram with the same legend as Fig. 2a but for an experiment with L_c = 2, σ_c = 1, σ_o = 0.2, μ_f = 0, and σ_f = 0.2.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

The experiment with μ_f = 0.1 is of lower quality than the equivalent experiment with μ_f = −0.1 when comparing $\overline{\text{BDnSS}}$ for L_c = 2 because BDn is not an equitable score as has been noted by Ben Bouallègue et al. (2018) in their Fig. 4. This is a very different situation when compared with the case where L_c = 80, for which the difference between $\overline{\text{BDnSS}}$ for positively and negatively biased experiments remained very small. On the contrary, $\overline{\text{FSS}}$ shows little change from the L_c = 80 case with monotonic growth of $\overline{\text{FSS}}$ with l_n. Since both $\overline{\text{BDnSS}}$ and $\overline{\text{FSS}}$ use $\overline{\text{BDn}}$ as the numerator, the difference in behavior comes from normalization. The bias of the prediction is taken into account in the normalization term of the $\overline{\text{FSS}}$ [Eq. (6)], and this counterbalances part of the variation of the numerator ($\overline{\text{BDn}}$) with the bias.

The $\overline{\text{BDnSS}}$ decomposition shows REL/UNC increases strongly for the biased experiments as a function of l_n (Fig. 5d) with L_c = 2 when it remains negligible for the unbiased experiment. Even if averaging over the spatial neighborhood strengthens the correspondence between fn and on, this leads to a weaker impact than on UNC for the biased experiments. The evolutions of GRES/UNC as a function of l_n are comparable for all the three experiments for L_c = 2 and similar to the case L_c = 80. It explains the increase of $\overline{\text{BDnSS}}$ from l_n = 1 to l_n = 7. Then, the decrease in $\overline{\text{BDnSS}}$ follows the decrease in REL/UNC for larger l_n.

To go further, we use an ergodicity assumption to find the asymptotic values of the observed and predicted mean frequencies over the neighborhood for the case of the very large neighborhoods (l_n ≫ L_c) and then derive $\overline{\text{BDn}},\overline{\text{FSS}}$, and $\overline{\text{BDnSS}}$. We then have the following asymptotic expressions for l_n → ∞:

$\begin{array}{l}on\to \overline{o},\\ fn\to \overline{o}Bn,\\ \overline{\text{BDn}}\to {\overline{o}}^2{(Bn-1)}^2,\\ \overline{\text{FSS}}\to \frac{2Bn}{1+B{n}^2},\\ \overline{\text{BDn}}(\text{climatology})\to 0,\\ \overline{\text{BDnSS}}\to 1-\frac{{\overline{o}}^2{(Bn-1)}^2}{\overline{\text{BDn}}(\text{climatology})}.\end{array}$(15)

We use the previous experiments with L_c = 2 to compute explicitly the scores for the different values of μ_f and compare them in Table 1 with their asymptotic formulations.

Table 1.

Different scores computed for the five experiments labeled by the μ_f value. The other parameters are the same for the five experiments: L_c = 2, σ_c = 1, σ_o = 0.2, σ_f = 0.2, and l_n = 49. The asymptotic scores refer to Eq. (15).

View Table

Table 1 shows that the asymptotic behavior for large neighborhoods reproduces the large variations of these five experiments. The term $\overline{\text{FSS}}$ depends only on the bias Bn of the forecast, and its formula [Eq. (15)] is invariant to the change Bn replaced by 1/Bn. This corresponds to a symmetrical penalty in $\overline{\text{FSS}}$ for the cases μ_f = 0.1 and μ_f = −0.1 because their biases are the inverse of each other.

The BDn(climatology) measures the residual variability of on around by the value of $\overline{o}$ and is still nonzero for l_n = 49. We can thus see by comparing it with the asymptotic or explicitly calculated values of $\overline{\text{BDn}}$ that the forecast deteriorates faster than the climatological forecast when it is biased. These conclusions are confirmed by the decomposition of $\overline{\text{BDnSS}}$, which shows that it is the reliability term that degrades with l_n (Fig. 5d). It should be borne in mind that for a given scale, ranking forecasts according to their $\overline{\text{BDn}}$ or $\overline{\text{BDnSS}}$ values is possible because $\overline{\text{BDn}}$ is a proper score. The values of $\overline{\text{BDnSS}}$ will tend toward −∞ when the size of the neighborhood tends toward infinity, as climatological forecasting tends toward perfect forecasting for our numerical setup. This simply indicates that the forecasts considered in this case are still subject to error. This behavior for large neighborhoods is not relevant for real meteorological cases because the ergodic assumption does not hold, as there is variability on all spatial and temporal scales.

To illustrate the difference in behavior between $\overline{\text{FSS}}$ and $\overline{\text{BDnSS}}$, we consider the extreme cases with μ_f = 10 and μ_f = −10. They correspond in fact to, respectively, “always forecast the event” and “never forecast the event” for all the members of the ensemble. The strongly positively biased forecast is much more penalized by $\overline{\text{BDnSS}}$ than the strongly negatively biased forecast in the case of our event with a base rate 0.045 less than 0.5. The bias values of these extreme forecasts also explain the $\overline{\text{FSS}}$ values [refer to Mittermaier (2021) for her discussion of $\overline{\text{FSS}}$ for extreme cases]. We can see also that the ranking of the two forecasts is reversed with respect to that given by $\overline{\text{BDn}}$ and $\overline{\text{BDnSS}}$ due to normalization.

To conclude, we have illustrated by these three series of experiments that $\overline{\text{BDnSS}}$ allows the quantification of the quality of the ensemble forecast with a proper score when they are used at a larger scale than the original forecasts and observations. The term $\overline{\text{BDnSS}}$ rewards event predictions close to the locations where they were observed as has been illustrated for cases where spatially correlated observations were to be forecast. The decomposition of this score is useful for the understanding of how it is split between the reliability and the generalized resolution terms as in the case of the classical Brier skill score without neighborhood.

a. Description of the models and observations

For this study, we use the same four forecasting systems operational at Météo-France as in Stein and Stoop (2022):

1. The global deterministic hydrostatic model ARPEGE uses a stretched horizontal grid ranging from 5 km over France to 24 km over New Zealand (Courtier et al. 1991). Its initial conditions are produced by 4DVAR assimilation cycles over a 6-h time window (Desroziers et al. 2003). In this study, we use outputs on a regular latitude–longitude grid oversampled to 0.025° over Europe.

2. The ensemble version of the ARPEGE model, called Prévision d’Ensemble Action de Recherche Petite Echelle Grande Echelle (PEARP), consists of 35 members with a horizontal resolution reduced to 7.5 km over France and 32 km over New Zealand (Descamps et al. 2015). The different initial conditions are obtained by a mixture of singular vectors and perturbed members of a set of 50 assimilations. The model error is represented by a random draw for each member among a set of 10 different coherent physics. We use in this study the outputs on a regular latitude–longitude grid oversampled to 0.025° over Europe.

3. The deterministic nonhydrostatic limited-area model AROME (Seity et al. 2011) uses a 1.3-km horizontal grid over western Europe. The lateral boundary conditions are provided by the ARPEGE model, and the initial conditions stem from a cycle of hourly 3DVAR assimilations. In this study, we use the outputs on a regular latitude–longitude grid at 0.025° over western Europe.

4. The ensemble version of the AROME model, called Prévision d’Ensemble AROME (PEAROME), consists of an ensemble of 16 perturbed members with a horizontal resolution of 2.5 km over western Europe. Their lateral boundary conditions are provided by a selection of PEARP members. Their initial conditions come from an assimilation ensemble of 25 members at 3.5 km centered around the 3DVAR operational analysis of the deterministic AROME model (Bouttier and Raynaud 2018). The tendencies of some physical parameterizations are perturbed by a random multiplicative factor in order to model the model error. Outputs on a regular latitude–longitude grid at 0.025° over western Europe are used in this study.

Ground rainfall observations cumulated over 6 h are provided by the Analyse par Spatialisation Horaire des Précipitations (ANTILOPE) data fusion product (Laurantin 2008) between radar data from the French radar network and rain gauges. The horizontal resolution of this analysis is 1 km, and the data are averaged on the scale of the forecast outputs to be comparable, i.e., on the latitude–longitude grid at 0.025°.

The comparison between observations and forecasts is performed with the 0000 UTC networks of ARPEGE and PEARP as well as the 0300 UTC networks of AROME and PEAROME, which use asynchronous coupling files from ARPEGE and PEARP of 0000 UTC. The two events chosen for this study are cumulative rainfall over 6 h above the 0.5- and 5-mm thresholds. We will use two different neighborhoods: 0.025° corresponding to a neighborhood reduced to one point of the verification grid and 0.525° which involves 21 × 21 = 441 points. The verification period is spread over a period of 12 months from 1 January 2020 to 31 December 2020.

No restriction on the minimum number of observations present in the chosen neighborhood is imposed, but the mask of observed missing rainfall data is also applied to the forecasts in order to keep the same control points for forecasts and observations at all times. The verification domain therefore corresponds to metropolitan France, extended from the range of the radars (Fig. 7). The observed and forecast frequencies averaged in a neighborhood are therefore computed only for points which are not masked in this neighborhood. To evaluate the statistical significance of the observed differences between two forecasting systems, a block-bootstrap technique (Hamill 1999) is applied to the time series by randomly drawing blocks of 3 days from the 366 days of the verification period and the starting point among the nine possible ones so as to calculate with disjoint neighborhoods.

View Full Size

Occurrences of the event: (a) accumulated rainfall greater than 0.5 mm in 6 h observed by the ANTILOPE analysis at 1200 UTC 27 Jan are colored purple. Masked areas where no radar data are available are hatched. (b) The mean frequencies over the 0.525° neighborhood of this event computed from the ANTILOPE analysis are plotted with the color palette given at the bottom of the figure. The mean probabilities of the event predicted by PEARP are plotted with the same color palette (c) without neighborhood and (d) with a neighborhood of 0.525°. (e),(f) The same mean probabilities are plotted in the same way for PEAROME. Rectangular boxes are drawn in black to facilitate comparison of the different panels.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

b. Comparison on a single case: 27 January 2020

In this section, we illustrate the calculation of $\overline{\text{BDn}}$ cases on a real case. We choose to look at the rainfall of 27 January 2020, accumulated between 0600 and 1200 UTC and greater than 0.5 mm. Figure 7 shows the occurrence of this event analyzed by ANTILOPE (frequency of occurrence 40%). On this day, a cold front passed rapidly over France, bounded by zones A-1 and A-2; behind it, a rear sky approached the English Channel and England. The probabilities predicted for this event by the PEARP at 0000 UTC 27 January 2020 and the PEAROME at 0300 UTC 27 January 2020 are also shown in Fig. 7, and BDn with no neighborhood or with a neighborhood of 0.525° is presented in Fig. 8. Overall, both models have well forecast the occurrence of this event. PEARP predicts higher probabilities on average (47%) than those predicted by PEAROME (42%). This is related to the tendency of ARPEGE physics, also used in PEARP, to predict small rainfalls too often (Stein and Stoop 2019). In detail,

• Ahead (A-1) and behind (A-2) the cold front, the spatial dispersion of PEARP members is more marked than that of PEAROME. PEAROME predicts finer structures than PEARP. Without a neighborhood, PEAROME’s BDn is worse than PEARP’s BDn, PEAROME being penalized by the double penalty. Applying a neighborhood of 0.525° reduces the impact of the double penalty more for PEAROME than for PEARP.

• Over the English Channel (B), PEAROME predicts higher probabilities with finer structures than PEARP, but PEARP probabilities are closer to observed frequencies than those of PEAROME. PEAROME therefore has poorer BDn than PEARP because its probabilities are too high. The application of a neighborhood preserves this ranking between PEAROME and PEARP.

• Over Brittany (C), PEAROME models well a zone with little probability of precipitation, corresponding to a respite between the passage of the front and the arrival of the rear sky. Conversely, the majority of PEARP members forecast precipitation over this zone and therefore high probabilities. PEAROME is closer to the observed frequency than PEARP, as can be seen on BDn with and without neighborhood.

• Over the Pisa region (D), PEARP predicts significant probabilities over a fairly widespread area. PEAROME predicts high probabilities in a more localized manner with a slight northward shift compared with reality. PEAROME is closer to reality than PEARP, as BDn shows. PEAROME’s small location errors are totally redeemed by the 0.525° in contrast to PEARP’s false alarms.

• Over the northeast of the control domain, both models predict high probabilities over an extended area, but nothing is observed. Both models are in outright false alarm with a BDn of 1. The application of a neighborhood does not redeem these false alarms.

View Full Size

BDn for the accumulated rainfall event greater than 0.5 mm in 6 h at 1200 UTC 27 Jan for PEARP calculated (a) without neighborhood and (b) with a neighborhood of 0.525°. The same BDn fields are plotted for PEAROME (c) without neighborhood and (d) with a neighborhood of 0.525°. The observations are provided by the ANTILOPE analysis. Masked areas where no radar data are available are hatched, and the same rectangular boxes are drawn as in Fig. 7.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

Over the whole domain, the $\overline{\text{BDn}}$ of PEARP without a neighborhood is 0.088, while that of PEAROME is 0.092; PEAROME is more affected by the double penalty than PEARP, particularly in zones A-1 and B. After applying a neighborhood, $\overline{\text{BDn}}$ for PEARP is 0.052, while $\overline{\text{BDn}}$ for PEAROME is 0.049. The neighborhood corrected the double penalty more for PEAROME than for PEARP, allowing PEAROME to have a better $\overline{\text{BDn}}$ than PEARP.

c. Comparison of two ensemble forecasts during 1 year

The PEAROME forecasts evaluated without neighborhood against the ANTILOPE analysis degrades with the forecast time as shown by the temporal evolution of $\overline{\text{BDn}}$ (Figs. 9a,c). These temporal evolutions also show a very pronounced diurnal cycle for light (rain accumulated during 6 h > 0.5 mm) and moderate (rain accumulated during 6 h > 5 mm) rainfall. The maximum error is at 1800 UTC which corresponds to the daily maximum of convective rainfall over France. The uncertainty term UNC for light rainfall is higher than for moderate rainfall by a factor of 4 (Fig. 9) in about the same proportion as the frequencies of occurrence of these rainfalls (not shown). The decrease in UNC also drives the decrease in $\overline{\text{BDn}}$ for moderate rainfall relative to light rainfall.

View Full Size

The $\overline{\text{BDn}}$ (full lines) and uncertainty (dashed black lines) averaged over the year 2020 as a function of the lead time for the rainfall thresholds 0.5 mm in 6 h (a) using no neighborhood and (b) using a neighborhood of l_n = 0.525°. The lower panels correspond to the thresholds 5 mm in 6 h (c) using no neighborhood and (d) using a neighborhood of l_n = 0.525°. The PEAROME results are drawn with purple lines and the PEARP results with orange lines. D and D1 correspond to the first and second day’s forecast. The observations are provided by the ANTILOPE analysis.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

Taking into account a neighborhood of 0.525° in $\overline{\text{BDn}}$ leads to a decrease in UNC because of pooling, which reduces the spatial variability of observations at the verification points and also the diurnal cycle of the convection at that scale more for moderate rain that for light rain (which are more widespread). The term $\overline{\text{BDn}}$ follows this decrease (Fig. 9) and amplifies it as can be seen in the temporal evolution of $\overline{\text{BDnSS}}$ (Fig. 10): the skill is therefore improved. The $\overline{\text{BDnSS}}$ of PEAROME for l_n = 0.525° is higher than that for l_n = 0.025°. Thus, the forecasts are more skillful when considered at scales larger than those of the forecast calculation grid.

View Full Size

The $\overline{\text{BDnSS}}$ (full lines), normalized reliability term (dashed lines), and normalized generalized resolution term (dotted lines) averaged over the year 2020 as a function of the lead time for the rainfall threshold 0.5 mm in 6 h (a) using no neighborhood and (b) using a neighborhood of l_n = 0.525°. The lower panels correspond to the threshold 5 mm in 6 h (c) using no neighborhood and (d) using a neighborhood of l_n = 0.525°. The PEAROME results are drawn with purple lines and the PEARP results with orange lines. The power of the statistical test on the difference in $\overline{\text{BDn}}$ is set at 5% and is represented on the $\overline{\text{BDnSS}}$ curve of PEAROME by the symbols: purple triangle up if PEAROME is better than PEARP, orange triangle down if PEARP is better than PEAROME, and circle if the difference is not significant at the 5% level. The observations are provided by the ANTILOPE analysis.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

The decomposition of $\overline{\text{BDnSS}}$ [Eq. (12)] using 36 bins shows that PEAROME is almost reliable for the four cases since the reliability error REL/UNC is small for this ensemble forecast when compared to the generalized resolution term GRES/UNC. The impact of the neighborhood is significant with a gain of $\overline{\text{BDnSS}}$ which is due to the generalized resolution term. Another advantage of the decomposition is that we can construct a reliability diagram connecting the different coordinate points ($\overline{f{n}_k},\overline{o{n}_k}$) as in Duc et al. (2013). We plot it at 1800 UTC in Fig. 11. We can thus visualize the small deviations of the PEAROME forecast from the diagonal except for the least frequent values where $\overline{f{n}_k}$ is close to 1. We also note that the PEAROME reliability is improved for the case l_n = 0.525° compared to the case l_n = 0.025° for each bin. This allows us to verify once again that PEAROME is almost reliable even if there remains a small conditional bias (Jolliffe and Stephenson 2011). We can also see that the sharpness diagram (Fig. 11) shows the strong reduction of the population of the different decomposition bins of $\overline{\text{BDn}}$ when we take a neighborhood of 0.525° but keep roughly the same shape as that without a neighborhood (no accumulation around the observed frequency for the large neighborhoods as in the previous idealized cases with small L_c). This reduction results in stronger sampling noise for l_n = 0.525° than for l_n = 0.025° in the reliability diagram.

View Full Size

(a) Reliability diagram (including a sharpness diagram) over the year 2020 at 1800 UTC on the first day of simulation for the threshold 0.5 mm in 6 h for PEAROME (purple lines) and PEARP (orange lines) using no neighborhood (full lines) and using a neighborhood of l_n = 0.525° (dashed lines). The horizontal and vertical colored lines correspond to $\overline{on}$ and $\overline{fn}$, respectively. (b) The threshold 5 mm in 6 h. The observations are provided by the ANTILOPE analysis.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

PEARP presents lower quality rainfall forecasts against the ANTILOPE analysis than PEAROME since its $\overline{\text{BDn}}$ is higher than PEAROME’s (Fig. 9) or its $\overline{\text{BDnSS}}$ is lower than PEAROME’s (Fig. 10) for all lead times for both thresholds and both neighborhoods. The decomposition of $\overline{\text{BDnSS}}$ using 36 bins shows that PEARP’s GRES/UNC is as good as PEAROME’s GRES/UNC and sometimes slightly improved and that PEARP is not as reliable as PEAROME. PEARP’s REL/UNC improves with time and is better for moderate rainfall than for light rainfall. This point is related to the parameterization of convection used in the PEARP global models, which overestimates light rainfall and underestimates heavy rainfall. This bias in the forecast can be seen in the PEARP reliability plot (Fig. 11a) which for light rainfall shows that neighborhood-averaged probabilities $\overline{f{n}_k}$ are higher than neighborhood-averaged observations $\overline{o{n}_k}$ for all bins in the decomposition or averaged over all bins ($\overline{fn}>\overline{on}$). It should also be noted that for both thresholds, the reliability error for PEARP is increased by taking into account a neighborhood of l_n = 0.525° but to a lesser extent than the improvement of the generalized resolution term (Fig. 10). We thus obtain an improvement of $\overline{\text{BDnSS}}$ with the increase of the neighborhood for PEARP. Figure 11 shows that the reliability of PEARP is improved by applying a neighborhood only for high probabilities but this occurs much less often than lower probabilities. On average, the reliability of PEARP is therefore made worse by applying a neighborhood as shown also in Fig. 10. A remarkable point in the evolution of PEARP scores with lead time is the better quality of the rain forecast at 0600 UTC of day D1 compared to that at 0600 UTC of day D (Fig. 10). The decomposition of $\overline{\text{BDnSS}}$ attributes this improvement to the reduction of the reliability term for longer lead times (Fig. 10). It is explained by the overdispersion of the PEARP for light rainfall brought about by the perturbation of the initial conditions, which is very strong (not shown) and which decreases during the forecast.

Comparing these two ensemble forecasts against the ANTILOPE analysis, we can conclude that PEAROME provides better rainfall forecasts for these two thresholds. In the case without neighborhood averaging, the difference is more important for light rainfall than for moderate rainfall and is mainly due to the reliability error of PEARP which is negligible for PEAROME. It is also important to note that the difference between the two ensemble forecasts increases when they are compared at the scale of a 0.525° neighborhood averaging. For light rainfall, we find that the impact of the neighborhood on the reliability term of PEARP on the $\overline{\text{BDnSS}}$ is negative, while it remains negligible for PEAROME. For moderate rainfall, we have the same growth with the neighborhood of the PEARP reliability error as for light rainfall. We can also note that the generalized resolution term for PEAROME and PEARP is close, while the generalized resolution term for PEARP was slightly better without a neighborhood. These variations correspond to the expected impact of the mitigation of the double penalty by the spatial tolerance brought by the use of a neighborhood which is maximal for an unbiased forecast as illustrated previously on idealized cases.

We return to 1800 UTC on the first day of simulation in order to analyze the variation of $\overline{\text{BDnSS}}$ as a function of the size of the neighborhood l_n. We have access to the distribution of $\overline{\text{BDnSS}}$ by plotting the minimum, maximum, and its mean value on the nine realizations obtained by varying the starting point of the disjoint neighborhood calculations (Fig. 12). We note that $\overline{\text{BDnSS}}$ is not as well estimated for large values of l_n due to the decrease in the number of verification points used. Both ensemble forecasts improve with l_n in a continuous way. Nevertheless, the conclusions of the comparison between PEAROME and PEARP are confirmed for both thresholds.

View Full Size

(a) The $\overline{\text{BDnSS}}$ over the year 2020 at 1800 UTC on the first day of simulation as a function of the neighborhood length scale l_n for PEAROME (purple lines) and PEARP (orange lines) for the threshold 0.5 mm in 6 h. (b) The threshold 5 mm in 6 h. The result of the statistical test is represented in the same way as in Fig. 10. The observations are provided by the ANTILOPE analysis.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

d. Comparison of an ensemble forecast with a deterministic forecast during 1 year

We now use the limiting case of the deterministic AROME forecast. In the neighborhood-free case and in the deterministic limit,

$\overline{\text{BDnSS}}=1-\frac{1-\text{PC}}{\overline{o}(1-\overline{o})}.$

The $\overline{\text{BDnSS}}$ for AROME against the ANTILOPE analysis is clearly worse than the $\overline{\text{BDnSS}}$ of PEAROME against the ANTILOPE analysis (Fig. 13) especially for moderate rainfall where AROME is even less accurate than the forecast made by the sample climatology.

View Full Size

As in Fig. 10, but for PEAROME (purple lines) and AROME (blue lines).

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

In the case of the 0.525° neighborhood, the AROME forecast is transformed into a pooled probabilistic forecast in the neighborhood corresponding to a 441-member ensemble forecast. The $\overline{\text{BDnSS}}$ decomposition of AROME is then performed on the 17 bins also used for the PEAROME. We note a significant increase in $\overline{\text{BDnSS}}$ as a function of the neighborhood size for AROME which is much faster than that for PEAROME leading to a reduction in the relative gap between the two forecasts. This shows that the correction of the double penalty by introducing a neighborhood is more active for a deterministic forecast than for an ensemble forecast. Both reliability and generalized resolution terms are improved by a factor of two by taking into account the neighborhood for AROME. Consequently, PEAROME is of better quality for the two thresholds and the two neighborhoods.

The growth of $\overline{\text{BDnSS}}$ with l_n (Fig. 14) shows that PEAROME is better than AROME for all verification scales even though the difference in quality measured by $\overline{\text{BDnSS}}$ decreases with l_n. This indicates that deterministic forecasts are more sensitive to the double penalty than forecast ensembles. It also validates a posteriori the method of neighborhoods which makes the assumption that there is probabilistic information of high quality to be recovered from the forecasts of the points close to the observation point as was shown in Mittermaier (2014).

View Full Size

As in Fig. 12, but for PEAROME (purple lines) and AROME (blue lines).

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

e. Comparison of two deterministic forecasts during 1 year

For the 0.5-mm threshold, we observe in Fig. 15 that AROME obtains better results than ARPEGE in terms of $\overline{\text{BDnSS}}$ against the ANTILOPE analysis when no neighborhood is taken into account. The decomposition using 17 bins shows that the generalized resolution terms are very close but that the reliability is better for AROME than for ARPEGE. The likely explanation for this shortcoming of ARPEGE comes from the positive bias of ARPEGE which is similar to PEARP for light convective rainfall due to its precipitating convection scheme (Stein and Stoop 2019). If we compare their performance at a scale of 0.525°, we see that both $\overline{\text{BDnSS}}$ improve significantly. The generalized resolution terms improve slightly for both models, but the AROME model is now always better than the ARPEGE model. Moreover, the reliability terms are also greatly improved (and thus reduced). We can see that the difference between the $\overline{\text{BDnSS}}$ of AROME and ARPEGE increases when comparing them on a larger scale because the effects of the double penalty are better corrected by the spatial tolerance introduced thanks to pooling in the case of an unbiased model like AROME.

View Full Size

As in Fig. 10, but for AROME (blue lines) and ARPEGE (red lines).

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

For the 5-mm threshold, the neighborhood-free $\overline{\text{BDnSS}}$ for both AROME and ARPEGE are both negative (Fig. 15) which shows that the sample climatology is a hard-to-beat forecast for this rarer event than for the 0.5-mm threshold. The decomposition in this case provides resolution and reliability terms which remain close for both models. This probably corresponds to the fact that ARPEGE does not have a strong bias in the case of moderate rainfall as does PEARP and is thus closer to AROME (Stein and Stoop 2019). The comparison at a larger scale of 0.525° shows that the correction of the double penalty in the calculation of $\overline{\text{BDnSS}}$ is more effective for AROME than for ARPEGE by improving both the resolution and reliability terms.

In the end, AROME predicts both light and moderate rainfall better than ARPEGE at both comparison scales.

f. Comparison of $\overline{\text{BDnSS}}$ and $\overline{\text{FSS}}$ during 1 year

The $\overline{\text{BDnSS}}$ and $\overline{\text{FSS}}$ differ only in their normalizations in forming a skill score. The first normalization $\overline{o{n}^2}-{\overline{on}}^2$ corresponds to the climatology of the sample, and the second normalization $\overline{o{n}^2}+\overline{f{n}^2}$ uses a forecast increasing the forecast error. These two scores as well as the values of the two reference forecasts are plotted for comparison on real cases of ensemble forecasts of light and moderate rainfall (Fig. 16). We see that on our real cases, $\overline{\text{BDnSS}}$ is always smaller than $\overline{\text{FSS}}$ because the reference provided by the sample climatology forecast makes an error (measured by $\overline{\text{BDn}}$) smaller than the reference forecast used for $\overline{\text{FSS}}$ since their difference in $\overline{\text{BDn}}$ is equal to $-({\overline{on}}^2+\overline{f{n}^2})$ and is therefore always negative. The sample climatology is always a harder-to-beat forecast than the reference forecast for $\overline{\text{FSS}}$. We also notice that the reference for $\overline{\text{FSS}}$ is not the same for PEAROME and PEARP because of the $\overline{f{n}^2}$ term. This results in the ranking of the models according to $\overline{\text{FSS}}$ not being the same as the one provided by $\overline{\text{BDn}}$ (Fig. 9) on which it is based and which is a proper score. The rankings provided by $\overline{\text{BDnSS}}$ and $\overline{\text{BDn}}$ are identical since the normalization by the BDn of the climatology does not depend on the forecast. The reversal of the ranking of these ensemble forecasts by $\overline{\text{FSS}}$ depends on their biases Bn. One way of getting around the problem is to remove these biases, but it is not clear how a debiasing of an ensemble forecast should be done or whether it will restore consistency between the rankings based on $\overline{\text{FSS}}$ and $\overline{\text{BDn}}$. On the other hand, we can see that both scores $\overline{\text{FSS}}$ and $\overline{\text{BDnSS}}$ limit the impact of the double penalty more significantly for PEAROME than for PEARP. Nevertheless, this relative improvement of $\overline{\text{FSS}}$ for PEAROME compared to PEARP is not sufficient for the restoration of the ranking provided by $\overline{\text{BDn}}$.

View Full Size

The $\overline{\text{BDnSS}}$ (full lines), $\overline{\text{FSS}}$ (dashed lines), UNC (dotted black lines), and $\overline{o{n}^2}+\overline{f{n}^2}$ (dotted-dashed lines) averaged over the year 2020 as a function of the lead time for the rainfall thresholds 0.5 mm in 6 h (a) using no neighborhood and (b) using a neighborhood of l_n = 0.525°. The lower panels correspond to the thresholds 5 mm in 6 h (c) using no neighborhood and (d) using a neighborhood of l_n = 0.525°. The PEAROME results are drawn with purple lines and the PEARP results with orange lines. The power of the statistical test on the difference in $\overline{\text{BDn}}$ is set at 5% and is represented on the $\overline{\text{BDnSS}}$ curve of PEAROME by symbols: purple upward triangle if PEAROME is better than PEARP, orange downward triangle if PEARP is better than PEAROME, and circle if the difference is not significant at the 5% level.

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide

The comparison between $\overline{\text{BDnSS}}$ and $\overline{\text{FSS}}$ is applied to the deterministic forecasts of AROME and ARPEGE (Fig. 17). We find the same conclusions as for the case of ensemble forecasts with a reference forecast that is easier to beat for $\overline{\text{FSS}}$ and which depends on the forecast as already noted by Amodei et al. (2015). It can be seen that in this case, the correction of the double penalty through the use of a 0.525° neighborhood is sufficiently effective for AROME to achieve better forecasts of light and moderate rainfall whereas this was not the case without taking into account the neighborhood.

View Full Size

As in Fig. 16, but for AROME (blue lines) and ARPEGE (red lines).

Citation: Monthly Weather Review 152, 5; 10.1175/MWR-D-22-0235.1

• Download Figure
• Download figure as PowerPoint slide