a. Datasets

In this study, we make predictions at 131 weather station locations covering Switzerland, as shown in Fig. 1. We consider forecast data from COSMO-E (Klasa et al. 2018) for the postprocessing task that spans January 2017–December 2022. COSMO-E is a limited-area weather forecasting model used for the operational weather forecasts for Switzerland by MeteoSwiss. It is operated as a 21-member ensemble with a 2.2-km horizontal resolution. It runs two cycles per day (0000 and 1200 UTC) with a time horizon of 5 days. We interpolate the nearest model grid cell to the selected station locations, using the k-d tree method. We retain model runs initialized at 0000 UTC and consider lead times between +3 and +24 h with hourly steps. We use observational data from the SwissMetNet network (MeteoSwiss 2022) to define our “ground truth.” The predictors and predictands used in this study are summarized in Table 1. They represent instantaneous measurements with hourly granularity. We note that while using both dewpoint temperature and dewpoint deficit is redundant, this is necessary to guarantee a fair comparison between the different model architectures in section 3. In contrast to prior postprocessing studies, we train a single model for all lead times and use the lead time as a predictor, thereby increasing the variance in the training set. This choice was motivated by the ease of implementation, allowing us to focus on the main aspect of the proposed methodology, namely, the physical constraints.

View Full Size

The location of the 131 weather stations across Switzerland considered in our study. Stations are colored on the basis of their elevation above sea level. We show the topography of Switzerland and its surroundings in the background. Map tiles are by Stamen Design, under CC BY 3.0. Data are by OpenStreetMap, under an Open Database License (ODbL).

Citation: Artificial Intelligence for the Earth Systems 2, 4; 10.1175/AIES-D-22-0089.1

• Download Figure
• Download figure as PowerPoint slide

Table 1.

List of predictors and predictands considered in this study.

View Table

b. Cross validation and random seed

To consider the variability due to the data used during optimization, each model configuration was trained on multiple cross-validation splits. When dealing with time series, it is important to design the cross-validation strategy in a way that ensures independence between sets. At the same time, in our setup it is desirable that samples of different sets are roughly equally distributed throughout the year. To do so, we opted for a simple fourfold cross validation with a holdout set for testing. Specifically, of our five years of data we used 20% for testing, 60% for training and 20% for validation, and we removed values between sets in a gap of five days in order to ensure independence. A total of four years was considered for training and validation, meaning each of the four cross-validation folds considered a different year in the dataset. The dataset partitioning is shown in Fig. 1 in the online supplemental material. For each cross-validation split, we applied a standard normalization to the inputs of each set based on the training set’s mean and standard deviation. Moreover, to account for the stochasticity of neural network (NN) optimization (due to weights initialization and gradient descent), each model configuration was trained with three different random seeds. In total, for any given approach, 12 trials were conducted with the trained models and subsequently all evaluated on the same holdout test dataset.

c. Multitask neural networks for postprocessing

To keep our DL framework general, the basic building block used for all models in this study is a fully connected neural network (see Fig. 2) that takes as inputs a vector containing the predictors in Table 1 and a vector with station identifers (IDs). Station IDs are mapped into an n-dimensional, real-valued vector z ∈ $\mathbb{R}$⁶ via an embedding layer and concatenated with the predictors. This approach, here referred to as “unconstrained setting,” is the same proposed by Rasp and Lerch (2018) and may be regarded as state-of-the-art for the case of local postprocessing of surface variables (Schulz and Lerch 2022). It is convenient because it allows for training a single model to do local postprocessing at many stations, instead of training one model for each station, making its operational implementation easier. We note that our $\mathbb{R}$⁶ embedding has more dimensions than the $\mathbb{R}$² found in Rasp and Lerch (2018): this is likely due to the fact that we target five variables simultaneously and to the complex topography of the Alps (as compared with German stations). The forecasts are deterministic, and we train a model to predict multiple target variables simultaneously. As such, we are dealing with a case of multitask learning (see Crawshaw 2020, for a survey on the subject), where the individual errors of each task contribute to the objective loss function. Kendall et al. (2017) observed that the relative weighting of each task’s loss had a strong influence on the overall performance of such models. This is fairly intuitive in our application because tasks have different scales (due to different units) and uncertainties. The authors proposed a weighting scheme based on the homoscedastic uncertainty of each task, where the weights are learned during optimization. To the best of our knowledge, this approach for multitask learning is new in the postprocessing field. If jointly postprocessing multiple meteorological parameters simultaneously becomes more common, it will be important to design optimal weighting schemes. We use the mean-square error (MSE) for the loss function $\mathcal{L}$_k of each task k. For a predicted value ${\widehat{y}}_{k,i}$ in physical units (we will use that notation for predicted values throughout the rest of the paper) and an observed value y_k,I, the task loss $\mathcal{L}$_k is hence defined as

${\mathcal{L}}_k\stackrel{\text{def}}{=}{\displaystyle \sum _{i=1}^{N_{\text{samples}}}\frac{{(y_{k,i}-{\widehat{y}}_{k,i})}^2}{N_{\text{samples}}}},$(1)

where N_samples is the number of samples. For p tasks, we define the combined loss $\mathcal{L}$ as

$\mathcal{L}\stackrel{\text{def}}{=}{\displaystyle \sum _{k=1}^p\left[\frac{1}{2}\frac{{\mathcal{L}}_k}{{\sigma }_k^2}+\log {\sigma }_k\right]}.$(2)

Each task’s loss $\mathcal{L}$_k is scaled by the homoscedastic uncertainty represented by ${\sigma }_k^2$, and a regularizing term logσ_k is added to prevent degenerating toward a zero-weighted loss. In practice, for improved numerical stability, we learn the log variance ${\eta }_k\stackrel{\text{def}}{=}\log {\sigma }_k^2$ for each task k, and Eq. (2) becomes

$\mathcal{L}={\displaystyle \sum _{k=1}^p\frac{{\mathcal{L}}_k\exp (-{\eta }_k)+{\eta }_k}{2}},$(3)

where the division by 2 can be ignored for optimization purposes because it does not influence the minimization objective. To avoid having to learn large biases, we initialize the bias vector in the model’s output layer using the training set-averaged output vector, which facilitates optimization. The optimization is insensitive to the initial values of this bias vector as long as it has the same order of magnitude as the mean output.

View Full Size

Summary of the models used in this study. (a) The basic building block of all models, a fully connected network preceded by an embedding layer. (b) The unconstrained setting used as a baseline, where all target variables are predicted directly. (c) The architecture-constrained setting, including a physical constraints layer that takes a subset of the target variables as inputs and returns the complete prediction. (d) The loss-constrained neural network, in which physical consistency is enforced by adding a physics-based penalty $\mathcal{P}$ to the conventional loss $\mathcal{L}$. (e) An offline-constrained neural network, where constraints are only applied after training using the constraints layer.

Citation: Artificial Intelligence for the Earth Systems 2, 4; 10.1175/AIES-D-22-0089.1

• Download Figure
• Download figure as PowerPoint slide

d. Enforcing analytic constraints in neural networks

We proceed to implement and compare two approaches on how to enforce physical constraints in our networks:

1. In the architecture-constrained setting, the constraints are enforced by using a layer that uses Eqs. (4) to derive RH and r from T, T_def, and P; T_def is the dewpoint deficit, to which we apply a rectified linear unit (ReLU) activation function to ensure positivity before computing dewpoint temperature as $T_d\stackrel{\text{def}}{=}T-T_{\text{def}}$. This additional step allows us to enforce that T ≥ T_d and RH ∈ [0, 100], two desirable properties in this case. We show the constrained architecture in Fig. 2c. The trainable part of the model has the same number of layers and units as the unconstrained architecture, but directly predicts only a subset of variables. In our case, which has five outputs and two constraints, we directly predict three variables and derive the last two via a custom-defined layer encoding Eqs. (4). An important point is that the choice of which variables are computed first and which are derived analytically is arbitrary. Given n = 5 variables and q = 2 constraints, the total number of possibilities in our case is n!/[q!(n − q)!] = 10. Nevertheless, there are differences in the actual implementation that point in favor of some configurations. For example, one may want the analytic constraints arranged in a way such that numerical stability is not a concern, for example, avoiding division by zero or asymptotes of logarithmic functions.

2. In the loss-constrained setting, the constraints are enforced by using an additional physics-based loss term that includes a penalty term $\mathcal{P}$ based on residuals from our set of analytic equations. As for the architecture-constrained approach, the variables that we choose to calculate the residuals are arbitrary. Based on Eq. (4), we define the following constraints:

   $\{\begin{array}{c}{\mathcal{P}}_{RH}\stackrel{\text{def}}{=}\widehat{\text{RH}}-f(\widehat{T},\widehat{T_d})=0\\ {\mathcal{P}}_r\stackrel{\text{def}}{=}\widehat{r}-g(\widehat{P},\widehat{T_d})=0\end{array},$(8)

   where physical violations result in nonzero residuals. Using the L2-norm for consistency with our MSE loss, we formulate the penalty term $\mathcal{P}$ used in the loss function as

   $\mathcal{P}\stackrel{\text{def}}{=}\frac{{({\mathcal{P}}_{\text{RH}})}^2}{{\sigma }_{\text{RH}}^2}+\frac{{({\mathcal{P}}_r)}^2}{{\sigma }_r^2},$(9)

   where we square the residuals $\mathcal{P}$_RH and $\mathcal{P}$_r to penalize larger violations more, and then scale by the variance of the observed values ${\sigma }_{\text{RH}}^2$ and ${\sigma }_r^2$ in order to normalize the contribution of the two terms. Last, the physical penalty term is added to the conventional loss and our training objective becomes minimizing the physically constrained loss function ${\mathcal{L}}_{\mathcal{P}}$:

   ${\mathcal{L}}_{\mathcal{P}}\stackrel{\text{def}}{=}(1-\alpha )\mathcal{L}+\alpha \mathcal{P},$(10)

   where α ∈ [0, 1] is a hyperparameter used to scale the contribution of the physical penalty term. Note that in contrast with the hard constraints in the architecture, with the soft constraints in the loss, we have no guarantee that $\mathcal{P}$ = 0 because stochastic gradient descent does not generally lead to a zero loss.

Last, to assess whether enforcing constraints during optimization is advantageous, we additionally introduce an “offline-constrained” setting (see Fig. 2e) in which constraints are enforced after training. Specifically, we train a model to minimize the MSE of T, T_d (derived from T and T_def), and P. The RH and r are then calculated after training so as to exactly enforce our physical constraints.

e. Libraries, hyperparameters, and training

We use the PyTorch deep learning library (Paszke et al. 2019) to implement our models, the Ray Tune library (Liaw et al. 2018) for hyperparameter tuning, and Snakemake (Mölder et al. 2021) to manage our workflow (the code is available online: https://www.github.com/frazane/pcpp-workflow). For training we use the Adam optimizer (Kingma and Ba 2014) with the exponential decay rates set to β₁ = 0.99 and β₂ = 0.999 for the first and second moments, implementing an early stopping rule based on a validation loss to avoid overfitting. We use the aggregated normalized mean absolute error (NMAE) aggregated over all five outputs as our validation loss:

$\text{NMAE}\stackrel{\text{def}}{=}\frac{1}{5}{\displaystyle \sum _{k=1}^5{\displaystyle \sum _{i=1}^{N_{\text{samples}}}\frac{\left|y_{i,k}-\widehat{y_{i,k}}\right|}{{\sigma }_k}}},$(11)

and halt training after five epochs in the absence of improvements in the validation loss. This metric was chosen for the validation and early stopping because it proved to be more robust to sudden model changes during training, when compared with the training loss. We save the model state with the lowest validation loss of all training epochs. The hyperparameters used to produce the main results of this study are shown in Table 2. We chose them after running a hyperparameter tuning algorithm for the unconstrained model that considered the aggregated loss of all cross-validation splits. The best performing hyperparameters configuration was then applied to all models. The loss-constrained model also required a hyperparameter to scale the influence of the physics-based penalty term. After testing different values, α was set to 0.995. We discuss this choice in the next section.

Table 2.

Hyperparameters used to train the models, along with their optimal value and the search space used for tuning the unconstrained model, represented as the range or set of possible values followed by the sampling method. After selecting the five best configurations automatically, we chose the one with the lowest number of trainable parameters for parsimony.

View Table

a. Predictive performance and physical consistency

We use two metrics to evaluate the overall performance of our models: the mean absolute error (MAE) and the mean-square skill score (MSSS) calculated with respect to the raw NWP forecast, defined as

$\text{MSSS}=1-\frac{{\text{MSE}}_{\text{PP}}}{{\text{MSE}}_{\text{NWP}}},$(12)

where MSE_PP and MSE_NWP represent the MSE for our postprocessed forecasts and the NWP forecast, respectively. The MSSS presented here was first computed on each station individually and then averaged. Values closer to 1 are better, and negative values indicate a decrease in performance. The overall results are presented in Table 3 and Fig. 3a for each variable. For both MAE and MSSS these results show that the performance is comparable for all NN architectures. There are small differences in performance of each setting on the different tasks, which we hypothesize are related to the influence of the constraints coupled with the multitask loss weighting. Overall, we observe slightly better results for the loss- and architecture-constrained approaches.

Table 3.

Two performance metrics: The mean absolute error (MAE; lower values are better) and mean-square skill score (MSSS; higher values are better) for each considered NN architecture (rows) and target variable (columns), averaged over the test set. Boldface type indicates the best performance for each metric and variable.

View Table

View Full Size

(a) MAE for each target variable and approach, where the boxplot distribution represents the nine trials using different cross-validation splits and random seeds. Note that the ranges of these distributions are relatively small in comparison with the absolute values of the error metric. (b) Scatterplot representing the distribution of physical violations $\mathcal{P}$_RH in RH units, as a function of RH, using all samples of all trials, where points are color coded by density. Physical violations are deviations from the zero line.

Citation: Artificial Intelligence for the Earth Systems 2, 4; 10.1175/AIES-D-22-0089.1

• Download Figure
• Download figure as PowerPoint slide

Table 4 displays Diebold–Mariano predictive performance test results for MAE, applied individually to each station and lead time following the implementation of Schulz and Lerch (2022). The reported values show the percentage of tests where an approach significantly outperformed another. Overall, these results align with Table 3, without a clear winner, but the offline constrained approach appears generally worse, except for pressure. Our models achieved MAEs of approximately 1.35°C, whereas an altitude-corrected NWP forecast, often used as a reference, gave 1.63°C (using a fixed lapse rate of 6°C km⁻¹).

Table 4.

For each target variable, the percentage of tests for which an approach (rows) produces significantly better forecasts than another (columns), according to Diebold–Mariano statistical tests performed with the MAE. Also reported is the average percentage of “wins” or “losses” for each approach. Tests are applied to each station and lead time individually.

View Table

We note that the MSSS values are surprisingly high, especially for pressure. These high MSSS values are due to the large errors in the NWP model. For variables that are strongly tied to elevation, such as pressure and temperature, the differences in the NWP model elevation and the true station elevation result in consistently large biases. These elevation differences can be larger than 100 m. Taking the example of pressure, the mean bias at certain stations is almost 90 hPa, which is reduced to almost 0 hPa by the postprocessing models, explaining the high MSSS values.

Figure 3b depicts the physical consistency of the predictions. On the vertical axis is Π_RH, which is the difference between the predicted RH and its physically consistent counterpart derived from the constraint function f(T, T_d) [Eq. (4)], while we show predicted values on the horizontal axis. Deviations from zero are therefore considered physical violations, as are values that are not between 0 and 100. Relative to the unconstrained approach, we observe a noticeable decrease of violations using the loss-constrained approach, although large violations still occur at the ends of the RH distribution, where values larger than 100% still occur. The architecture-constrained models bring physical violations to zero to within machine precision. These results are consistent with Beucler et al. (2021). As a side remark, we note that in the unconstrained approach larger violations tend to occur more at the tails of the distribution, which could indicate that it is more difficult to converge to a physically consistent solution if the samples are scarce.

To choose an optimal value for the α hyperparameter, we tested several values and compared both the NMAE and the physical consistency of the predictions. The results are shown in Fig. 4 for the following values: α ∈ {0.0, 0.2, 0.5, 0.8, 0.9, 0.95, 0.99, 0.995, 0.999, 0.9999, 0.999 99}. We observe that the trade-off between physical constraints and performance is nonlinear: up to α = 0.995, there is little to no drawback in terms of performance. In contrast, for higher values of α, the NMAE starts to increase. We note two things: first, the choice of this hyperparameter α can be chosen based on how much one wishes to prioritize physical consistency over error reduction. Second, one should be aware that the choice of the learning rate has a significant influence on α’s impact (and vice versa) and thus on this trade-off, although this was not further investigated in this study. We limit ourselves to observe that, from a practical standpoint, this relationship is inconvenient as it makes model selection harder, and we consider it to be a drawback of the loss-constrained approach.

View Full Size

Effect of the hyperparameter α on both the overall physical violation $\mathcal{P}$ (in red) and the NMAE (in blue) for the test dataset.

Citation: Artificial Intelligence for the Earth Systems 2, 4; 10.1175/AIES-D-22-0089.1

• Download Figure
• Download figure as PowerPoint slide

b. Robustness to data scarcity

Among the potential advantages of constraining neural networks using physical knowledge are improved robustness to data scarcity. The rationale is that because we reduce the hypothesis space of the model to a subset of physically consistent solutions, we could expect the physics-constrained models to learn with fewer training samples, or to require fewer parameters. We have therefore designed an experiment in which we retrained all models (with fewer parameters in order to reduce the chance of overfitting; see Table 2 in the online supplemental material) on increasingly reduced training datasets, namely, 20%, 5%, and 1% of the full dataset. The reduction is applied by station to ensure that all stations are still equally represented in the dataset. For instance, with the 1% reduction, we trained with roughly 200 samples for each station. The results, shown in Fig. 5a, seem to indicate a relatively smaller decrease in performance for the architecture-constrained approach when the data is scarce, although this difference is rather small. Importantly, the added value of enforcing physical consistency in data scarce situations is emphasized by Fig. 5b. For the unconstrained model in particular, the physical inconsistencies increase as we reduce the number of training samples. Conversely, for the architecture-constrained approach, physical inconsistencies are always zero by construction.

View Full Size

(a) NMAE for each reduction and approach, where the boxplot distribution represents the nine trials using different cross-validation splits and random seeds. As the size of the training dataset reduces, constrained models perform relatively better. (b) Boxplot showing the physical penalty term $\mathcal{P}$’s distribution as a function of training data size for the unconstrained and loss-constrained settings. The architecture- and offline-constrained approaches have zero penalty by construction.

Citation: Artificial Intelligence for the Earth Systems 2, 4; 10.1175/AIES-D-22-0089.1

• Download Figure
• Download figure as PowerPoint slide

c. Generalization ability

A common finding of physics-informed ML is that physically constraining models could help them generalize to unseen conditions (Willard et al. 2022). To test the ability of our models to generalize to unseen weather situations, we designed an experiment in which models are trained on a dataset that excludes the warm season (June–August), and then tested on the warm season only. This choice was motivated by the increasing relevance of record-shattering heat extremes in a warming climate. In such situations, the robustness of postprocessing models is put to test as they have to process and predict values never seen during training. We present our results in Fig. 6 and observe that physical constraints do not seem to impact the generalization capabilities of the model to unseen temperature extremes. This result, consistent with Beucler et al. (2020), suggests that the constraints of Eq. (4) are insufficient to guarantee generalization capability for our mapping.

View Full Size

NMAE for the test dataset containing samples from June–August, conditioned on different quantiles of the univariate temperature distribution. As expected, the error increases as temperature are more extreme, but the relative performance of the considered architectures does not change significantly.

Citation: Artificial Intelligence for the Earth Systems 2, 4; 10.1175/AIES-D-22-0089.1

• Download Figure
• Download figure as PowerPoint slide