1. Introduction
Very short-term forecasts up to a lead time of about 6 h have become an important tool for issuing high-impact weather warnings. Commonly referred to as nowcasting, such forecasts cover a time range where the persistence of the most recent observations can produce significant skill, while the performance of numerical weather prediction (NWP) can suffer from outdated initial conditions, spinup issues, and model approximations. In this sense, nowcasting can be seen as a postprocessing step aimed at updating NWP forecasts with real-time observations. The term seamless is often used in this context to indicate a consistent prediction regardless of location, lead time or forecasting procedure (World Meteorological Organization 2015). To obtain a seamless forecast that does not degrade the quality of its individual components, the combination should be based on an objective procedure that explicitly accounts for the respective forecast uncertainties.
In the context of quantitative precipitation forecasting, nowcasting systems are typically based on observations from a network of weather radars. The radar quantitative precipitation estimation (QPE) consists of a data processing chain that transforms raw polar-coordinate measurements of backscattered signal from volumes of hydrometeors at a certain height in the atmosphere into estimates of precipitation intensity at the ground. For hydrological applications, the radar QPE is usually presented as a sequence of fields on a regular Cartesian grid at 1–4-km and 1–6-min resolutions, down to 100 m and 1 min in case of urban catchments (Thorndahl et al. 2017). By assuming persistence of precipitation patterns in moving coordinates, known as the Lagrangian persistence assumption (Zawadzki et al. 1994), the radar observations can be used to produce extrapolation forecasts whose skill rapidly decreases with lead time as the persistence assumption weakens. Because the lifetime of precipitation relates to its spatial scale (Venugopal et al. 1999; Seed 2003), predictability in the Lagrangian persistence approach can range from several hours at the scale of hundreds of kilometers down to time scales less than 1 h when considering the extent of a localized thunderstorm (e.g., Germann et al. 2006; Foresti and Seed 2014).
Thanks to the improvements in horizontal resolution that have occurred in the past decades, NWP precipitation forecasts are increasingly able to meet nowcasting requirements in terms of skill at the convective scale. Sun et al. (2014) give an overview of the achievements and challenges in this field. While diabatic initialization techniques offer an efficient way to assimilate radar reflectivities, reduce precipitation spinup problems, and increase the NWP skill in the nowcasting range (e.g., Jacques et al. 2018), their success is limited by the capacity to simulate and support the observed rainfall intensities. Stephan et al. (2008) link such problems to imperfections in the physics parameterization and in the accuracy of the large-scale environment conditions provided by the NWP. The direct assimilation of radar volume scans is also possible (e.g., Johnson et al. 2015), but the approach is particularly challenging owing to the strong nonlinear relationship between radar observations and model state (Tong and Xue 2005; Dowell et al. 2011).
Beyond the challenges related to the assimilation of radar observations, the computational cost of numerical simulations limits the update frequency of NWP runs and hence their ability to exploit real-time radar observations for initializing a new forecast. In fact, only a few national weather services can currently afford to operationally run rapid update cycles, and this usually applies to deterministic runs only. Hence, today, the potential of high-spatiotemporal-resolution measurements from weather radars is largely undertapped in NWP schemes.
Precipitation nowcasting systems have traditionally relied on deterministic blending schemes, in most cases using a simple linear-weighting function in time to produce a smooth transition from the radar extrapolation to a deterministic NWP forecast. A first example of an operational scheme is the NIMROD system at the Met Office (Golding 1998). In this approach, weights are typically precomputed based on long-term forecast error statistics and then applied in real time. For example, the blending system currently in operations at MeteoSwiss (INCA; Haiden et al. 2011) performs the blending by starting to give weight to the NWP forecast at +1 h and then linearly increasing the weight given to the NWP model until +4 h. The technique is simple and robust, which explains its appeal for operational use. On the other hand, it cannot account for variability in both the NWP and the radar extrapolation skills, resulting in a suboptimal blended forecast. Moreover, linear blending generally produces weights that are only a function of lead time, ignoring other dependencies such as location or spatial scales.
More advanced nowcasting blending schemes have thus been developed in recent years: for example, the Short-Term Ensemble Prediction System (STEPS; Bowler et al. 2006; Seed et al. 2013). STEPS quantifies the skill of the NWP forecast in real time and uses the information to adjust the weights that combine the radar-based extrapolation and NWP forecast as a function of lead time and spatial scale. As the actual forecast skill is unknown, the blending weights at analysis time are gradually relaxed to climatological skill values.
Following the success encountered in the NWP community (Bauer et al. 2015), a number of nowcasting systems have introduced the ensemble approach by means of stochastic perturbations as a mean to quantify the forecast uncertainty (e.g., Bowler et al. 2006; Berenguer et al. 2011; Atencia and Zawadzki 2014). The definition of forecast error as the spread of the ensemble members represents an uncertainty estimate that can be readily propagated into several applications—for example, hydrological forecasting (Zappa et al. 2010)—and ultimately used for informed decision-making (Todini 2018). The same nowcast error models can be also used to find the optimal combination of radar extrapolation and NWP forecasts. In fact, Bayes’s theorem represents an attractive mathematical framework for the combination of information in the presence of uncertainty. In essence, one needs to represent the two precipitation forecasts as probability density functions to construct the prior and likelihood distributions and thus compute their posterior distribution. Bayesian filtering refers to the Bayesian formulation of optimal filtering, which seeks to estimate the state of a time-varying system based on observations. For this class of methods, “optimal” is intended in the sense of a minimum mean-squared error, thus corresponding to the least squares solution, while “filtering” indicates the estimation of the current state given past observations.
The widely used Kalman filter (KF; Kalman 1960) algorithm is a recursive solution to the linear Gaussian optimal filtering problem that can be formulated from a purely Bayesian point of view, that is, to produce a solution for the posterior Gaussian distribution (e.g., Särkkä 2013). In this sense, the KF becomes a powerful tool to implement the Bayesian update for a time-varying system under the assumption of a linear Gaussian model. To relax the linear assumption and in order to allow its application to high-dimensional state vectors, Evensen (1994) introduced the ensemble Kalman filter (EnKF), which is essentially a Monte Carlo implementation of the KF equations that allows us to sample the error distribution without the need to fully represent it.
It should be noted that the use of precipitation data in an EnKF system is not straightforward. This can be related to the presence of non-Gaussian error statistics and the limited ensemble size that is available in most applications. An important question of this study is therefore whether a Bayesian approach based on the EnKF can produce reasonable results despite such challenges.
The main contribution of this study is the formulation of the precipitation nowcasting and blending with NWP in a Bayesian framework through the use of an ensemble Kalman filter. The idea is to apply the EnKF from a nowcasting perspective: the prediction step is computed as an extrapolation nowcast, while the correction step uses NWP precipitation forecasts as pseudomeasurements. Thus, the usual application of EnKF in data assimilation is reversed, as observations represent the state to be updated with NWP pseudomeasurements.
Following a similar approach, Buil (2017) investigated the use of a local ensemble transform Kalman filter (LETKF) to improve ensemble precipitation nowcasts produced with the SBMcast model (Berenguer et al. 2011). Our work generalizes the same concept to the use of NWP ensembles and introduces two technical solutions that are necessary to deal with high-resolution precipitation fields. These include the use of principal component analysis (PCA) to perform a reduced-space EnKF and a resampling technique addressing the non-Gaussianity of precipitation fields. In formulating this approach, we have tried to present a clean mathematical development that can be generalized to analogous problems and datasets.
The paper is organized as follows. Section 2 formulates the Bayesian nowcasting system based on the reduced-space ensemble Kalman filter. The developed approach is tested using synthetic experiments in section 3 and real precipitation fields in section 4. Section 5 concludes the paper and discusses possible future developments.
2. Formulation of the Bayesian nowcasting technique
Sequential Bayesian filtering consists of three main steps:
Initialization step, where a vector characterizing the initial state of the model and a covariance matrix describing errors in the estimation of the model state are defined.
Prediction step, where a state transition function is used to move the system forward in time, while process noise can be included to account for the model uncertainty.
Correction step, where an observation is used to update the prediction according to the relative importance of the observation error and prediction error in a Bayesian fashion.
The algorithm is iterated between steps 2 and 3. Each of these main steps is now detailed: first in the terms of the original Kalman filter, then by looking at the approximations introduced by the EnKF, and finally by formulating the nowcasting application of this study. Figure 1 summarizes the algorithm and its application in the nowcasting context.
The flowchart summarizes the basic equations of the Bayesian nowcasting system. Detailed explanation and notation can be found in the text (section 2).
Citation: Monthly Weather Review 147, 3; 10.1175/MWR-D-18-0258.1
a. Initialization
The true but unknown model state vector 
where 
In precipitation nowcasting, the state variable of interest is the mean precipitation intensity in millimeters per hour, with time step 
Following the ensemble approach introduced in the EnKF, the uncertainty in the radar estimates can be included by means of multiple realizations of the QPE. In this way, observation errors are explicitly represented and propagated throughout the forecast. There exists a considerable research effort in the quantification of radar QPE uncertainty with the work of, among others, Ciach et al. (2007), Germann et al. (2009), or more recently Cecinati et al. (2017); a review can be found in Mandapaka and Germann (2010). A quantification of the QPE uncertainty is nevertheless outside the scope of this study, and it is thus assumed to be zero, meaning that all members of the nowcast ensemble are initialized with the same QPE field 
b. Prediction
For the time update step of the filter, a prediction model is used to move the state forward in time to produce a first guess of the new state of the system, namely, the prior. We now include a time index and denote a prior estimate with the superscript f for forecast.
where 
The term 
The term on the left-hand side of (4) represents the ith member of the extrapolated ensemble for grid point j and time 
In (5), 
c. Correction
The correction step of the KF performs the update of the prior distribution 
The innovation term in (9) is computed with a perturbed version of the observation 
It is also possible to include in (11) a covariance inflation factor, as is commonly done for practical applications of the EnKF in data assimilation (Hunt et al. 2007). Covariance inflation techniques can account for biases in the estimation of both the model state and the observation uncertainty by tuning the inflation factor against some measure of the analysis performance. For the sake of simplicity, this study does not include an inflation factor, and we thus assume that 
Finally, 
d. Reduced-space EnKF by means of principal component analysis
The high-resolution precipitation fields used in nowcasting can be seen as samples from high-dimensional probability distributions, where each dimension corresponds to one of the m grid points in the domain. Because we only have n members, the 
The idea of computing a more efficient EnKF analysis in a space of reduced dimensionality has been already exploited in data assimilation methods [see, e.g., Ott et al. (2002) and references therein]. The definition of such subspace is, of course, the main question. The ensemble transform Kalman filter (Bishop et al. 2001; Hunt et al. 2007) makes direct use of the prior members in order to define the basis of an ensemble space wherein to perform the analysis. A limitation of this approach is that the basis is not linearly independent, so other approaches have used the singular vectors of 
In this way, now 
The approach is illustrated in Fig. 2 for a radar-based nowcast at lead time +40 min trying to assimilate a NWP EPS forecast that was initialized almost 10 h before. The data in this example belong to case study ID01; see section 4b. In this figure, two of the precipitation fields in 
Example of reduced-space EnKF for one particular time step of a nowcast (+40 min; event ID01). The first two columns include the first and last members of the prior 
Citation: Monthly Weather Review 147, 3; 10.1175/MWR-D-18-0258.1
e. Dealing with non-Gaussian precipitation data
Precipitation exhibits a bimodal distribution, being either zero or lognormally distributed. Such non-Gaussianity is problematic to the reduced-space EnKF technique described above mainly in two ways. First, it introduces biases in the estimation of the sample covariance matrices during the Bayesian update. Second, it causes an overestimation of the wet area and underestimation of maximum intensity when precipitation fields are linearly combined to produce the analysis. We refer to the latter effect as smoothing.
To account for the skewed distribution of precipitation, positive rain rates R are log-transformed to produce units of dBR according to the following formula: 
To adjust the bias in the wet area and the underestimation of maximum precipitation intensity that is found in the analysis, we introduce a final resampling and probability matching step. Similar approaches can be found in many applications including recent developments in nonlinear data assimilation [e.g., the probability mapping of Poterjoy (2016)]. Our implementation is inspired by the procedure described by Ebert (2001) for the computation of probability matched ensemble medians. The basic idea is first to build a new empirical posterior distribution of precipitation intensities by resampling the prior and pseudo-observation empirical distributions. Finally, the original posterior distribution is matched to the resampled distribution. The algorithm can be described as follows.
- For the current lead time t, compute the resampling weight p aswhich can be interpreted as the average probability of sampling the pseudo-observation
Sort the m rain rates from
For rank
Assign the value of the highest rank
Repeat steps 2–4 for the remaining ensemble members
f. Summary of the algorithm
The basic steps of the Bayesian nowcasting algorithm are summarized by the flowchart in Fig. 1. As the diagram shows, the last available radar QPE ensemble, called 
3. Synthetic data experiments
The Bayesian nowcasting technique is first applied to synthetic data in a series of numerical experiments. Synthetic data use prescribed parameters and data distributions, thus simplifying the interpretation of the simulation results.
a. Synthetic data
where 
The synthetic pseudo-observations 
b. Simulation results
Figure 3 shows the results of the Bayesian nowcast from a set of 100 synthetic simulations using 100 ensemble members. Please note that the posterior estimate at lead time t is computed based on the ensembles of prior and pseudo-observations at the same lead time. Since this is a recursive algorithm, the prior at lead time t is based on the the posterior of time 
Results of 100 synthetic simulations to test the Bayesian nowcasting based on reduced-space EnKF. The mean value over a 5 × 5 pixel target box is shown as a function of time step for a stochastic persistence forecast (red), synthetic NWP ensemble (blue), and Bayesian nowcasting (green). The graph represents at each lead time the median and the 10%–90% interval. All simulations used 100 member ensembles, 
Citation: Monthly Weather Review 147, 3; 10.1175/MWR-D-18-0258.1
Figure 4 presents the results of two experiments that verify the ability of the Bayesian nowcast to account for flow dependence, here intended as the capacity to adapt to the uncertainty of both the pseudo-observations and radar-based nowcast. The root-mean-square errors (RMSE) are computed with respect to the simulated true model state 
Results from two numerical experiments with the Bayesian nowcast based on reduced-space EnKF. (a) The skill of the Bayesian nowcast ensemble mean (solid line) is verified for three levels of temporal persistence of the simulated true model state (dotted line; left to right: 
Citation: Monthly Weather Review 147, 3; 10.1175/MWR-D-18-0258.1
4. Real data experiments
The Bayesian nowcasting technique is now applied and verified with a set of real precipitation events over the Swiss Alps.
a. Datasets and study domain
The weather radar observation dataset was produced by the fourth generation of the MeteoSwiss radar network. The radar QPE is a composite image derived from measurements of five dual-polarization, C-band weather radars distributed across the Swiss territory (white triangles in Fig. 5). The data processing chain is specifically conceived to address the many challenges that exist in operating weather radars in an Alpine region. The design of the fourth-generation Swiss radar network is presented by Germann et al. (2015, 2016, 2017). The precipitation product is based on a fixed Z–R relationship between measured reflectivity and rainfall intensity 
The 512 km × 512 km study area (solid red line) and the 300 km × 300 km verification domain (dashed red line) are centered over Switzerland. The five radars are indicated with white triangles. The composite radar mask, the topography, and the national borders are also included as references.
Citation: Monthly Weather Review 147, 3; 10.1175/MWR-D-18-0258.1
A control experiment is produced through stochastic radar extrapolation. This ensemble forecast is based solely on the stochastic Lagrangian persistence approach described in section 2b. Therefore, it represents a benchmark against which the benefits of the Bayesian update can be assessed.
The mesoscale EPS forecasts that are included in this study are produced by the 21-member COSMO-E model that has been in operation at MeteoSwiss since 2016 (Klasa et al. 2018). In this study, the control member is treated as an additional member of the ensemble (i.e., 20 perturbed members and one unperturbed control). The COSMO-E model provides twice-daily (initialization times at 0000 and 1200 UTC) medium-range forecasts (up to 120 h) for the Alpine region at convection-permitting resolution (2.2-km mesh size). The ensemble of initial conditions is produced by the kilometer-scale ensemble data assimilation system (KENDA; Schraff et al. 2016), based on an LETKF (Hunt et al. 2007). The lateral boundary conditions are provided by the ECMWF ensemble model. The radar reflectivities are assimilated using a latent heat nudging scheme (Leuenberger and Rossa 2007; Stephan et al. 2008). Finally, the uncertainty of the model parameterizations is included in COSMO-E with stochastically perturbed physics tendencies (SPPT; Buizza et al. 1999).
b. Case studies
Four precipitation events between 2016 and 2017 were selected. They are listed in Fig. 6, where their total accumulated precipitation is represented.
Maps of total accumulated radar precipitation corresponding to the events considered in section 4. The timestamp indicates the start of the event, while the event duration is given within brackets.
Citation: Monthly Weather Review 147, 3; 10.1175/MWR-D-18-0258.1
The first precipitation event ID01 (Fig. 6a) is linked to a frontal system causing convective activity to be organized in lines with a SW–NE orientation, while the general flow is predominately west-southwest.
The second event ID02 (Fig. 6b) concerns a winter precipitation case that is produced by a strong northwesterly flow bringing moist air toward the Alps. As a consequence, precipitation mainly occurs on the northern slopes while it is effectively blocked by the Alpine chain, leading to the formation of a rain shadow south of the Alps.
Event ID03 (Fig. 6c) is a summertime situation with widespread and localized convective activity under a westerly flow. Thunderstorms have a scattered distribution but only occurred in the southern half of the domain.
The last precipitation case ID04 (Fig. 6d) is again a summertime event, but this time the flow is from the southwest, resulting in an interesting case with prefrontal convective activity on the southern side of the Alps and an active frontal passage with organized convection on the northern side of the Alps.
c. Verification method
where 
FSS conveniently depends on spatial scale and intensity, and it is thus particularly suitable for the verification of precipitation fields. To generalize the FSS to the ensemble skill, we applied the same approach described by Zacharov and Rezacova (2009), which consists of computing the FSS for each individual ensemble member and then averaging it to obtain the mean ensemble skill.
d. Bayesian precipitation nowcast examples
Figure 7 shows an example of a nowcast started at 2100 UTC 11 July 2016 (ID01). Each row represents one selected lead time (+00, +60, +120, or +180 min). Each column includes one of the forecasts, namely, the COSMO-E control member (COE), one random member from the Bayesian nowcast (BAY), or stochastic radar extrapolation (EXTR) ensembles. The verifying observations from the radar QPE are also included in the first column (OBS).
An example for a forecast started at 2100 UTC 11 Jul 2016 (ID01) with (a) verifying radar QPE at 2100, 2200, 2300, and 0000 UTC; (b) the COSMO-E control run; (c) one realization (member) of the Bayesian nowcast; and (d) one member of the stochastic radar extrapolation ensemble at +00, +60, +120 and +180 min after 2100 UTC. The COSMO-E forecast was initialized at 1200 UTC. The shaded area indicates the radar QPE mask. All fields are 10-min equivalent precipitation intensities.
Citation: Monthly Weather Review 147, 3; 10.1175/MWR-D-18-0258.1
An organized line of convection moving eastward with a southwest–northeast orientation is clearly visible in the radar images in the first column. In the second column, COSMO-E was initialized at 1200 UTC (i.e., 9 h before the start of this nowcast). In a real-time situation, this is currently the most recent EPS run that would be available for forecasting precipitation. The initialization time of the NWP EPS run is implicitly accounted for in the Bayesian nowcast, as this adapts to the spreads in both the NWP and extrapolation nowcasts, which also depend on their initialization time. We refer to this behavior as flow dependence. One can notice a delay in the initiation of convective activity in the COSMO-E control run, where the storms do not yet appear as developed and organized as in the observation. It is in such situations that the use of a nowcasting system is particularly valuable.
The third column presents the result of the Bayesian nowcasting introduced in this study. One can expect this forecast to be a combination of the forecasts in the second and fourth columns. In fact, the first and last frames are identical and very close to the extrapolation and COSMO-E fields, respectively, while in between the correction step of the EnKF uses the COSMO-E forecast to update the extrapolation forecast. The fourth column contains one member of the stochastic radar extrapolation ensemble forecast. The last observations are advected while being progressively replaced by correlated noise in order to simulate the loss of predictability of the Lagrangian persistence forecast. The radar mask is also advected along with the radar field in order to illustrate the influence of the domain boundaries on the extrapolation technique.
A more quantitative illustration of the Bayesian nowcasting technique is given in Fig. 8, where the mean precipitation intensity over the 300 km × 300 km verification box (Fig. 5) is shown as a function of forecast lead time. As in Fig. 3, the plot shows the ensemble median estimates with their 10%–90% prediction intervals, while the observed mean intensity over the box is also included as a black dotted line. It is interesting to note that not only does the mean precipitation intensity seamlessly merge from observations into the NWP forecast, but also the variability within the ensemble grows consistently with its transition from the observations to the numerical model.
Mean areal precipitation intensity in dBR over the 300 km × 300 km verification box as a function of forecast lead time for the nowcast started at 2100 UTC 11 Jul 2016 (as in Fig. 7). The median and the 10%–90% prediction interval are visualized.
Citation: Monthly Weather Review 147, 3; 10.1175/MWR-D-18-0258.1
A second example of a nowcast is presented in Fig. 9. This time, the nowcast is started at 1000 UTC 31 January 2017 (ID02), and it concerns the winter precipitation event with pronounced orographic precipitation, as can be seen in the radar images.
As in Fig. 7, but for the nowcast started at 1100 UTC 31 Jan 2017 (ID02). The COSMO-E forecast was initialized at 0000 UTC of the same day.
Citation: Monthly Weather Review 147, 3; 10.1175/MWR-D-18-0258.1
The COSMO-E run was initialized at 0000 UTC (i.e., 10 h before the start of the nowcast). While the general structure of the precipitation event seems well captured, the control run of the ensemble is missing most of the higher precipitation intensities, as seen by the radar observation.
By design, the stochastic extrapolation model conserves the rain-rate distribution over the whole domain (it is a persistence forecast). It is also interesting to observe how a simple extrapolation approach can be challenging in regions with complex orography, as it can happen that precipitation is unrealistically advected across a mountain ridge to areas that should remain cloud free. There are ongoing research efforts in trying to account for additional sources of predictability, such as the diurnal cycle (Atencia et al. 2017) or orographic forcing (Panziera and Germann 2010; Foresti et al. 2018) for the correction of extrapolation nowcasts.
The Bayesian nowcasting at +60 min has already picked up the large-scale structure from COSMO-E, but at the same time, it still manages to produce higher intensity values, consistently with the distribution of rain rates at observation time. At +180 min, it has almost entirely converged to the numerical model.
The seamless transition from the radar observation to the COSMO-E forecast is perhaps easier to notice in Fig. 10, where the mean precipitation intensity over the 300 km × 300 km verification box (Fig. 5) is shown as a function of forecast lead time. It is now clearer from this chart that the Bayesian nowcast starts exactly from the last radar observation and then moves toward the COSMO-E forecast. After 3 h, the Bayesian nowcast is very close to COSMO-E. Another interesting aspect to note is how the spread of the Bayesian nowcast ensemble is equal to zero by definition at the beginning of the nowcast (assuming no observation error) and increases to closely match the COSMO-E ensemble spread. In this sense, the Bayesian nowcasting can be considered seamless in terms of both mean and spread.
As in Fig. 8, but for the nowcast started at 1100 UTC 31 Jan 2017 (ID02).
Citation: Monthly Weather Review 147, 3; 10.1175/MWR-D-18-0258.1
It is important to remember that the Bayesian update is based on the distribution of the PC scores (cf. Fig. 2). The spread in terms of PC scores is illustrated in Fig. 11, where the average ratio between the extrapolation and NWP prediction intervals is shown for the first five PC scores and as function of lead time. The ratio becomes larger than 1.0 (black dotted line) when the prediction interval of a given PC score in the extrapolation is larger than the corresponding prediction interval for the NWP. For the examples included in this study, this happens on average between 1.5 and 2 h.
The spread of the extrapolation nowcast (PIEXTR) relative to the COSMO-E spread (PICOE) in terms of the first five PC scores. The results are averaged over all the case studies. The spread PI is defined as the 10%–90% prediction interval.
Citation: Monthly Weather Review 147, 3; 10.1175/MWR-D-18-0258.1
e. Verification results
For each event, a 4-h Bayesian nowcast was started every hour. Given that all events last 12 h, the number of runs per event is equal to 9. Therefore, there are in total 4 events × 9 runs × 4 h = 144 h that are included in this verification. If one further considers the time resolution of 10 min (hence, six precipitation fields for every hour), the size of the ensembles (21 members) and the number of forecasting procedures that are compared (i.e., radar-based nowcasting, NWP, and Bayesian estimate), there is a total of 144 h × 6 fields h−1 × 21 members × 3 procedures = 54 432 precipitation fields that are verified.
The verification of the four precipitation events (ID01–04) is presented in Fig. 12, which illustrates the mean ensemble skill in terms of FSS at 32-km scale and 2.0 mm h−1. These scale and intensity parameters are those typically used in comparison analysis of precipitation forecasts (e.g., Simonin et al. 2017). Scores are computed within the verification box as defined in Fig. 5. For each of the three ensemble forecasts, the average skill and its variability among all runs are represented.
Mean ensemble FSS as a function of lead time for a 32-km scale, threshold 2.0 mm h−1, and computed within the verification box (see Fig. 5) for all four events (ID01–04). The mean ensemble FSS is the average of the individual FSSs of each ensemble member. The bars represent the variability among the individual runs within a single event. The three ensembles are the COSMO-E in blue (COE), the Bayesian nowcasting in green (BAY), and the stochastic radar extrapolation in red (EXTR).
Citation: Monthly Weather Review 147, 3; 10.1175/MWR-D-18-0258.1
We will refer to crossover time to indicate the lead time when the skill of the numerical model becomes better than the skill of the radar extrapolation, which is easily found in the verification plots as the point where the blue line crosses the red line. Hence, the challenge for the Bayesian nowcast is to correctly estimate the crossover point using the relative spread of the two ensemble forecasts.
The first verification concerns the convective event presented above (ID01; see Figs. 7 and 8). Interestingly, the skill of the NWP ensemble shows a considerable variability within the precipitation event, particularly at short forecast lead times, where it spans from 0.8 to less than 0.1. This is an indication that for this particular event, the NWP skill strongly varied among model runs. The crossover time is on average between +2 and +2.5 h, and it is well captured by the Bayesian nowcasting system. However, there is a tendency to converge a bit too early to COSMO-E.
The second verification case concerns the winter stratiform event (ID02; see Figs. 9 and 10). In Fig. 12b, the average ensemble skill for COSMO-E is higher and has a lower variability compared to event ID01, while the skill of the extrapolation drops more rapidly. Consequently, the average crossover time appears to occur already after about 1.5–2.0 h. In this case, the Bayesian approach is correctly weighting the contribution of the numerical model to produce an optimal combined forecast.
The intense and localized precipitation of event ID03 translates into the low skill of COSMO-E, as can be seen in Fig. 12c. This seems to reflect the fact that the numerical model cannot correctly forecast individual thunderstorms at a spatial scale of 32 km. As a consequence, the crossover point is not reached within the 4-h lead times included in the verification. The Bayesian nowcast manages to identify the poor skill of the numerical model, and as a result, the convergence is also occurring after a lead time of 4 h. However, the performance of the Bayesian nowcast remains lower than the stochastic radar extrapolation for most of the forecast range, in particular between 1- and 3-h lead time, an indication that the skill of COSMO-E is somewhat overestimated.
Concerning the last precipitation case study (ID04) in Fig. 12d, it can be seen that the large-scale organization of this event leads to a slower decay of the skill of the radar extrapolation compared with event ID03. COSMO-E also exhibits a reasonably good accuracy, and the average crossover is found at about 2.5–3 h, which is also well captured by the Bayesian nowcast.
The four examples illustrate how a combination of flow-dependent NWP and radar extrapolation skills requires a blending scheme that can recognize and then adapt to each situation accordingly. In fact, the observed crossover time for this limited sample of events and the specific definition of skill used here varied from 1.5 to more than 4 h. The Bayesian nowcasting has been shown to effectively use the ensemble spread as an estimate for the forecast uncertainty to derive optimal weights to integrate the NWP forecasts with the radar-based nowcasts. The blended product is a better forecast overall, since the total skill of the blended nowcast is equal to or larger than the total of both the radar-based and NWP-based nowcast skills.
Despite these promising results, here we did not observe the same improvement in terms of crossover time that was found in the numerical experiments based on synthetic Gaussian data (Fig. 4). Besides a larger variability due to the limited sample size, we probably also need to attribute such outcome to the challenging nature of real precipitation data.
5. Summary and conclusions
Nowcasting can be seen as a forecasting procedure that combines NWP forecasts with real-time observations in the very short-term range (from 0 to about 6 h). The result should be one seamless forecast that optimally integrates all available sources of predictability in a consistent way.
Such a nowcasting system should use the information from ensembles, as these can convey uncertainty and at the same time respect the spatiotemporal structure of the variable of interest (Berenguer et al. 2011). Already recognized as an established technique in data assimilation, the Kalman filter is an attractive tool for the nowcasting problem, too, thanks to its recursive implementation of the Bayesian update equations. However, its application to high-resolution precipitation fields is not trivial and requires a number of adjustments to relax some of its basic assumptions and efficiently work with large datasets. In this study, a reduced-space ensemble Kalman filter was introduced as a tentative answer to such requirements.
Synthetic data experiments were run in order to test the methodology. The results have demonstrated that the Bayesian nowcast correctly captures the flow dependence of both the NWP forecast and the Lagrangian persistence of the radar observations. The result is an adaptive blending scheme that depends on the forecast skill as estimated by the forecast ensemble spread.
Four experiments were run with real data produced by the Swiss weather radar network and the COSMO-E model. This represented an application of the EnKF blending approach with actual precipitation fields and a relatively small ensemble size (21 members). The four meteorological situations spanned from a stratiform wintertime precipitation case to very localized summertime convection. The first two case studies were detailed in a sequence of images for a qualitative assessment of the system. The skill for all four cases was quantified in terms of fractions skill score at 32-km scale and 2.0 mm h−1. Scores were computed for different scales and intensities (not shown), yielding consistent results. The results indicate that the system can cope with realistic conditions to produce precipitation forecasts that perform as well as extrapolation nowcasts for short lead times and then converge to the skill of NWP at a plausible rate.
Compared to traditional blending schemes based on fixed linear weights, the Bayesian approach does not need a training period, and it can adapt to various meteorological situations and changes in the forecast skill. Furthermore, it provides a blending that is a function not only of lead time, but also of location and spatial scale, thanks to its formulation into the space of principal components. Its computational cost remains within the needs of a nowcasting system: with the data used in this study, the correction step takes approximately 1 s on a 2.6-GHZ core, while one prediction step demands approximately 6 s. In total, a 4-h Bayesian nowcast takes about 2 min of computations. The only parallelization achieved in this study is the one currently integrated in the Python library numpy via OpenMP.
The framework that was presented in this study has the potential to serve for other variables than precipitation. In principle, any observed meteorological parameter can be used as a state variable of a Bayesian nowcast, provided that an error model for its persistence forecast is available. Moreover, the nowcast of multiple variables, including their interdependencies, can be foreseen. For example, future research could consider the joint integration of multiple fields (e.g., precipitation, temperature, wind) by means of a combined PCA approach (CPCA; Wilks 2011a). In this sense, the Bayesian nowcasting approach can open the door to a truly integrated, multivariable, seamless, probabilistic nowcasting system.
One additional aspect calling for future research is the Gaussian assumption of the EnKF in presence of non-Gaussian quantities such as precipitation. The non-Gaussian nature of precipitation questions the use of an EnKF approach. In principle, a nonparametric approach such as the particle filter (PF) would be more suitable, as it employs a nonlinear and non-Gaussian update. On the other hand, PF requires a very large ensemble size, which limits its application to high-resolution data (Snyder et al. 2008). For this reason, the present study included a resampling and probability matching step to correct for the biases in wet area and maximum intensities resulting from the EnKF analysis. However, recent developments in data assimilation using local particle filters (LPF; see, e.g., Penny and Miyoshi 2016; Poterjoy 2016) and hybrid local ensemble Kalman particle filters (Robert et al. 2018) are showing promising results and hence could also represent valuable developments for seamless forecasting systems.
Acknowledgments
This study was supported by the Swiss National Science Foundation Ambizione project Precipitation attractor from radar and satellite data archives and implications for seamless very short-term forecasting (PZ00P2_161316). Discussions with Isztar Zawadzki on the use of PCA on precipitation fields have been of great inspiration. We are grateful to Heini Wernli, Nikola Besic, Martin Rempel, and the three anonymous reviewers for their valuable comments and corrections to the manuscript.
REFERENCES
Anderson, J. L., 2001: An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev., 129, 2884–2903, https://doi.org/10.1175/1520-0493(2001)129<2884:AEAKFF>2.0.CO;2.
Atencia, A., and I. Zawadzki, 2014: A comparison of two techniques for generating nowcasting ensembles. Part I: Lagrangian ensemble technique. Mon. Wea. Rev., 142, 4036–4052, https://doi.org/10.1175/MWR-D-13-00117.1.
Atencia, A., I. Zawadzki, and M. Berenguer, 2017: Scale characterization and correction of diurnal cycle errors in MAPLE. J. Appl. Meteor. Climatol., 56, 2561–2575, https://doi.org/10.1175/JAMC-D-16-0344.1.
Bauer, P., A. Thorpe, and G. Brunet, 2015: The quiet revolution of numerical weather prediction. Nature, 525, 47–55, https://doi.org/10.1038/nature14956.
Berenguer, M., D. Sempere-Torres, and G. G. Pegram, 2011: SBMcast—An ensemble nowcasting technique to assess the uncertainty in rainfall forecasts by Lagrangian extrapolation. J. Hydrol., 404, 226–240, https://doi.org/10.1016/j.jhydrol.2011.04.033.
Bishop, C. H., B. J. Etherton, and S. J. Majumdar, 2001: Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects. Mon. Wea. Rev., 129, 420–436, https://doi.org/10.1175/1520-0493(2001)129<0420:ASWTET>2.0.CO;2.
Bowler, N. E., C. E. Pierce, and A. W. Seed, 2006: STEPS: A probabilistic precipitation forecasting scheme which merges an extrapolation nowcast with downscaled NWP. Quart. J. Roy. Meteor. Soc., 132, 2127–2155, https://doi.org/10.1256/qj.04.100.
Buil, A., 2017: Nowcasting probabilístico basado en observaciones de lluvia con radar meteorológico. Ph.D. thesis, Universitat Politècnica de Catalunya, 175 pp., https://upcommons.upc.edu/handle/2117/111231.
Buizza, R., M. Milleer, and T. N. Palmer, 1999: Stochastic representation of model uncertainties in the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 125, 2887–2908, https://doi.org/10.1002/qj.49712556006.
Cecinati, F., M. A. Rico-Ramirez, G. B. Heuvelink, and D. Han, 2017: Representing radar rainfall uncertainty with ensembles based on a time-variant geostatistical error modelling approach. J. Hydrol., 548, 391–405, https://doi.org/10.1016/j.jhydrol.2017.02.053.
Ciach, G. J., W. F. Krajewski, and G. Villarini, 2007: Product-error-driven uncertainty model for probabilistic quantitative precipitation estimation with NEXRAD data. J. Hydrometeor., 8, 1325–1347, https://doi.org/10.1175/2007JHM814.1.
Dowell, D. C., L. J. Wicker, and C. Snyder, 2011: Ensemble Kalman filter assimilation of radar observations of the 8 May 2003 Oklahoma City supercell: Influences of reflectivity observations on storm-scale analyses. Mon. Wea. Rev., 139, 272–294, https://doi.org/10.1175/2010MWR3438.1.
Ebert, E. E., 2001: Ability of a poor man’s ensemble to predict the probability and distribution of precipitation. Mon. Wea. Rev., 129, 2461–2480, https://doi.org/10.1175/1520-0493(2001)129<2461:AOAPMS>2.0.CO;2.
Ebert, E. E., and Coauthors, 2013: Progress and challenges in forecast verification. Meteor. Appl., 20, 130–139, https://doi.org/10.1002/met.1392.
Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99, 10 143–10 162, https://doi.org/10.1029/94JC00572.
Evensen, G., 2003: The ensemble Kalman filter: Theoretical formulation and practical implementation. Ocean Dyn., 53, 343–367, https://doi.org/10.1007/s10236-003-0036-9.
Foresti, L., and A. Seed, 2014: The effect of flow and orography on the spatial distribution of the very short-term predictability of rainfall from composite radar images. Hydrol. Earth Syst. Sci., 18, 4671–4686, https://doi.org/10.5194/hess-18-4671-2014.
Foresti, L., I. V. Sideris, L. Panziera, D. Nerini, and U. Germann, 2018: A 10-year radar-based analysis of orographic precipitation growth and decay patterns over the Swiss Alpine region. Quart. J. Roy. Meteor. Soc., 144, 2277–2301, https://doi.org/10.1002/qj.3364.
Fortin, V., M. Abaza, F. Anctil, and R. Turcotte, 2014: Why should ensemble spread match the RMSE of the ensemble mean? J. Hydrometeor., 15, 1708–1713, https://doi.org/10.1175/JHM-D-14-0008.1.
Germann, U., I. Zawadzki, and B. Turner, 2006: Predictability of precipitation from continental radar images. Part IV: Limits to prediction. J. Atmos. Sci., 63, 2092–2108, https://doi.org/10.1175/JAS3735.1.
Germann, U., M. Berenguer, D. Sempere-Torres, and M. Zappa, 2009: REAL—Ensemble radar precipitation estimation for hydrology in a mountainous region. Quart. J. Roy. Meteor. Soc., 135, 445–456, https://doi.org/10.1002/qj.375.
Germann, U., M. Boscacci, M. Gabella, and S. Maurizio, 2015: Peak performance: Radar design for prediction in the Swiss Alps. Meteor. Technol. Int., 42–45, https://www.meteoswiss.admin.ch/content/dam/meteoswiss/en/Mess-Prognosesysteme/doc/peak-performance-radar-design-for-prediction.pdf.
Germann, U., J. Figueras i Ventura, M. Gabella, A. Hering, I. Sideris, and B. Calpini, 2016: Triggering innovation. Meteor. Technol. Int., 62–65, https://www.meteoswiss.admin.ch/content/dam/meteoswiss/de/Mess-und-Prognosesysteme/Atmosphaere/Radarnetz/doc/MTI-April2016-Rad4Alp.pdf.
Germann, U., D. Nerini, I. Sideris, L. Foresti, A. Hering, and B. Calpini, 2017: Real-time radar: A new Alpine radar network. Meteor. Technol. Int., 88–92, https://www.meteoswiss.admin.ch/content/dam/meteoswiss/en/Mess-Prognosesysteme/Atmosphaere/doc/MTI-April2017-Rad4Alp.pdf.
Golding, B. W., 1998: Nimrod: A system for generating automated very short range forecasts. Meteor. Appl., 5, 1–16, https://doi.org/10.1017/S1350482798000577.
Haiden, T., A. Kann, C. Wittmann, G. Pistotnik, B. Bica, and C. Gruber, 2011: The Integrated Nowcasting through Comprehensive Analysis (INCA) system and its validation over the eastern Alpine region. Wea. Forecasting, 26, 166–183, https://doi.org/10.1175/2010WAF2222451.1.
Hunt, B. R., E. J. Kostelich, and I. Szunyogh, 2007: Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter. Physica D, 230, 112–126, https://doi.org/10.1016/j.physd.2006.11.008.
Jacques, D., D. Michelson, J.-F. Caron, and L. Fillion, 2018: Latent heat nudging in the Canadian regional deterministic prediction system. Mon. Wea. Rev., 146, 3995–4014, https://doi.org/10.1175/MWR-D-18-0118.1.
Johnson, A., X. Wang, J. R. Carley, L. J. Wicker, and C. Karstens, 2015: A comparison of multiscale GSI-based EnKF and 3DVar data assimilation using radar and conventional observations for midlatitude convective-scale precipitation forecasts. Mon. Wea. Rev., 143, 3087–3108, https://doi.org/10.1175/MWR-D-14-00345.1.
Kalman, R. E., 1960: A new approach to linear filtering and prediction problems. J. Fluids Eng., 82D, 35–45, https://doi.org/10.1115/1.3662552.
Klasa, C., M. Arpagaus, A. Walser, and H. Wernli, 2018: An evaluation of the convection-permitting ensemble COSMO-E for three contrasting precipitation events in Switzerland. Quart. J. Roy. Meteor. Soc., 144, 744–764, https://doi.org/10.1002/qj.3245.
Leuenberger, D., and A. Rossa, 2007: Revisiting the latent heat nudging scheme for the rainfall assimilation of a simulated convective storm. Meteor. Atmos. Phys., 98, 195–215, https://doi.org/10.1007/s00703-007-0260-9.
Lien, G. Y., E. Kalnay, and T. Miyoshi, 2013: Effective assimilation of global precipitation: Simulation experiments. Tellus, 65A, 19915, https://doi.org/10.3402/tellusa.v65i0.19915.
Mandapaka, P. V., and U. Germann, 2010: Radar-rainfall error models and ensemble generators. Rainfall: State of the Science, Geophys. Monogr., Vol. 191, Amer. Geophys. Union, 247–264, https://doi.org/10.1029/2010GM001003.
Nerini, D., N. Besic, I. Sideris, U. Germann, and L. Foresti, 2017: A non-stationary stochastic ensemble generator for radar rainfall fields based on the short-space Fourier transform. Hydrol. Earth Syst. Sci., 21, 2777–2797, https://doi.org/10.5194/hess-21-2777-2017.
Ott, E., B. R. Hunt, I. Szunyogh, M. Corazza, E. Kalnay, D. Patil, and J. A. Yorke, 2002: Expoliting local low dimensionality of the atmospheric dynamics for efficient ensemble Kalman filtering. 2002 Fall Meeting, San Francisco, CA, Amer. Geophys. Union, Abstract NG71A-06, http://abstractsearch.agu.org/meetings/2002/FM/NG71A-06.html.
Panziera, L., and U. Germann, 2010: The relation between airflow and orographic precipitation on the southern side of the Alps as revealed by weather radar. Quart. J. Roy. Meteor. Soc., 136, 222–238, https://doi.org/10.1002/qj.544.
Penny, S. G., and T. Miyoshi, 2016: A local particle filter for high-dimensional geophysical systems. Nonlinear Processes Geophys., 23, 391–405, https://doi.org/10.5194/npg-23-391-2016.
Poterjoy, J., 2016: A localized particle filter for high-dimensional nonlinear systems. Mon. Wea. Rev., 144, 59–76, https://doi.org/10.1175/MWR-D-15-0163.1.
Pulkkinen, S., D. Nerini, and L. Foresti, 2018: pySTEPS: The nowcasting initiative. pySTEPS, https://pysteps.github.io/.
Robert, S., D. Leuenberger, and H. R. Künsch, 2018: A local ensemble transform Kalman particle filter for convective-scale data assimilation. Quart. J. Roy. Meteor. Soc., 144, 1279–1296, https://doi.org/10.1002/qj.3116.
Roberts, N. M., and H. W. Lean, 2008: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev., 136, 78–97, https://doi.org/10.1175/2007MWR2123.1.
Särkkä, S., 2013: Bayesian Filtering and Smoothing. Cambridge University Press, 232 pp., https://doi.org/10.1017/CBO9781139344203.
Schiemann, R., R. Erdin, M. Willi, C. Frei, M. Berenguer, and D. Sempere-Torres, 2011: Geostatistical radar-raingauge combination with nonparametric correlograms: Methodological considerations and application in Switzerland. Hydrol. Earth Syst. Sci., 15, 1515–1536, https://doi.org/10.5194/hess-15-1515-2011.
Schraff, C., H. Reich, A. Rhodin, A. Schomburg, K. Stephan, A. Periáñez, and R. Potthast, 2016: Kilometre-scale ensemble data assimilation for the COSMO model (KENDA). Quart. J. Roy. Meteor. Soc., 142, 1453–1472, https://doi.org/10.1002/qj.2748.
Seed, A. W., 2003: A dynamic and spatial scaling approach to advection forecasting. J. Appl. Meteor., 42, 381–388, https://doi.org/10.1175/1520-0450(2003)042<0381:ADASSA>2.0.CO;2.
Seed, A. W., C. E. Pierce, and K. Norman, 2013: Formulation and evaluation of a scale decomposition-based stochastic precipitation nowcast scheme. Water Resour. Res., 49, 6624–6641, https://doi.org/10.1002/wrcr.20536.
Simonin, D., C. Pierce, N. Roberts, S. P. Ballard, and Z. Li, 2017: Performance of Met Office hourly cycling NWP-based nowcasting for precipitation forecasts. Quart. J. Roy. Meteor. Soc., 143, 2862–2873, https://doi.org/10.1002/qj.3136.
Snyder, C., T. Bengtsson, P. Bickel, and J. Anderson, 2008: Obstacles to high-dimensional particle filtering. Mon. Wea. Rev., 136, 4629–4640, https://doi.org/10.1175/2008MWR2529.1.
Stephan, K., S. Klink, and C. Schraff, 2008: Assimilation of radar-derived rain rates into the convective-scale model COSMO-DE at DWD. Quart. J. Roy. Meteor. Soc., 134, 1315–1326, https://doi.org/10.1002/qj.269.
Sun, J., and Coauthors, 2014: Use of NWP for nowcasting convective precipitation: Recent progress and challenges. Bull. Amer. Meteor. Soc., 95, 409–426, https://doi.org/10.1175/BAMS-D-11-00263.1.
Thorndahl, S., T. Einfalt, P. Willems, J. E. Nielsen, M.-C. ten Veldhuis, K. Arnbjerg-Nielsen, M. R. Rasmussen, and P. Molnar, 2017: Weather radar rainfall data in urban hydrology. Hydrol. Earth Syst. Sci., 21, 1359–1380, https://doi.org/10.5194/hess-21-1359-2017.
Tipping, M. E., and C. M. Bishop, 1999: Mixtures of probabilistic principal component analyzers. Neural Comput., 11, 443–482, https://doi.org/10.1162/089976699300016728.
Todini, E., 2018: Paradigmatic changes required in water resources management to benefit from probabilistic forecasts. Water Secur., 3, 9–17, https://doi.org/10.1016/j.wasec.2018.08.001.
Tong, M., and M. Xue, 2005: Ensemble Kalman filter assimilation of Doppler radar data with a compressible nonhydrostatic model: OSS experiments. Mon. Wea. Rev., 133, 1789–1807, https://doi.org/10.1175/MWR2898.1.
Venugopal, V., E. Foufoula-Georgiou, and V. Sapozhnikov, 1999: Evidence of dynamic scaling in space-time rainfall. J. Geophys. Res., 104, 31 599–31 610, https://doi.org/10.1029/1999JD900437.
Wilks, D., 2011a: Principal component (EOF) analysis. Statistical Methods in the Atmospheric Sciences, D. S. Wilks, Ed., International Geophysics Series, Vol. 100, Academic Press, 519–562, https://doi.org/10.1016/B978-0-12-385022-5.00012-9.
Wilks, D., 2011b: Time series. Statistical Methods in the Atmospheric Sciences, D. S. Wilks, Ed., International Geophysics Series, Vol. 100, Academic Press, 395–456, https://doi.org/10.1016/B978-0-12-385022-5.00009-9.
World Meteorological Organization, 2015: Seamless Prediction of the Earth System: From Minutes to Months. WMO, 483 pp.
Zacharov, P., and D. Rezacova, 2009: Using the fractions skill score to assess the relationship between an ensemble QPF spread and skill. Atmos. Res., 94, 684–693, https://doi.org/10.1016/j.atmosres.2009.03.004.
Zappa, M., and Coauthors, 2010: Propagation of uncertainty from observing systems and NWP into hydrological models: COST-731 Working Group 2. Atmos. Sci. Lett., 11, 83–91, https://doi.org/10.1002/asl.248.
Zawadzki, I., J. Morneau, and R. Laprise, 1994: Predictability of precipitation patterns: An operational approach. J. Appl. Meteor., 33, 1562–1571, https://doi.org/10.1175/1520-0450(1994)033<1562:POPPAO>2.0.CO;2.
