1. Introduction
California has the largest interannual variability in precipitation of any state, receiving almost the entirety of its yearly precipitation during winter months. Future climate projections suggest that California’s interannual variability in precipitation may increase (Gershunov et al. 2019). Nearly 25%–50% of the state’s rainfall and snowpack during this time are directly related to atmospheric rivers (Dettinger et al. 2011; Ralph et al. 2013; Rutz et al. 2014). Atmospheric rivers (ARs) have been shown to be associated with 92% of the West Coast’s heaviest 3-day rain events (Ralph and Dettinger 2012) and are responsible for the largest major floods in West Coast rivers (Ralph et al. 2006; Neiman et al. 2011; Konrad and Dettinger 2017; Corringham et al. 2019). ARs also supply nearly half of California’s water supply, end droughts in the West Coast, and provide water to wetlands, floodplains, and fisheries (Dettinger 2013; Florsheim and Dettinger 2015).
Several studies have evaluated how well operational numerical weather prediction (NWP) models forecast AR characteristics in the western United States (Wick et al. 2013; Lavers et al. 2016; Nardi et al. 2018). However, it has also been shown that forecast location, intensity and temporal evolution of AR events can change substantively in the 72-h period before landfall (Martin et al. 2019). AR-related flooding is prevalent in the Russian River basin in Northern California, which is embedded within the complex terrain of the coastal mountains where landfalling ARs interact with topography, resulting in heavy precipitation (Neiman et al. 2002; Colle 2004; Ralph et al. 2006; Dettinger et al. 2011; Kunz and Wassermann 2011; Picard and Mass 2017; Cao et al. 2019). Millions of people are also at risk during high-impact precipitation events in the San Francisco Bay region immediately south of the Russian River basin (Bridger et al. 2019; Ralph et al. 2018).
Gowan et al. (2018) and others have shown that current high-resolution models outperformed other operational models with lower resolution for wintertime precipitation in the western contiguous United States (western CONUS). This study focuses on the ability of two mesoscale models undergoing continual development to forecast heavy precipitation in the California region during the 2018/19 winter season: 1) the National Centers for Environmental Prediction’s (NCEP) High-Resolution Rapid Refresh model (hereafter referred to simply as the HRRR) and 2) the Naval Research Laboratory’s (NRL) Coupled Ocean–Atmosphere Mesoscale Prediction System (COAMPS) (COAMPS). Comparisons are made of forecast skill from those two models to the NCEP’s frozen operational North American Mesoscale Forecast System (NAM-3km) of the NCEP (Rogers et al. 2017).
The HRRR is an hourly updated, convection-allowing model (Benjamin et al. 2016). Its 3-km grid spacing provides high-resolution detail over the CONUS with more than 1.9 million grid points. The HRRR has been shown to have the highest accuracy among other high-resolution models for deterministic and probabilistic precipitation forecasts (Bytheway et al. 2017; Gowan et al. 2018; Caron and Steenburgh 2020). However, Darby et al. (2019) suggest that the HRRR has a tendency to underpredict precipitation along the west coast based on their analysis of the model’s skill during the 2015/16 winter. English et al. (2021) found that operational and experimental versions of the HRRR during five 2019 precipitation events also exhibited a dry bias in the San Francisco Bay region of central California and wet bias over the Sierra Nevada Mountains.
COAMPS is the Navy’s high-resolution, convection-allowing model with 4-km grid spacing (Hodur 1997). COAMPS is used operationally for naval defense operations around the world. Model output is publicly available for a coastal California domain used primarily for studies on air–sea interactions, including the system’s ability to forecast oceanic coastal circulations, offshore surface winds and low clouds, marine boundary layer structure, and sea surface temperatures (e.g., Haack and Burk 2001; Hsu et al. 2007; Haack et al. 2008; Neveu et al. 2016).
Stone et al. (2020) focused on the improvement of forecasts along the west coast due to dropsonde observations added to the Navy Global Environmental Model, the parent model for COAMPS. Although no published studies validate COAMPS forecasts for regional California precipitation, errors have been noted in the model’s treatment of precipitation amounts, structure and duration compared to observations in other regions as well as errors in forecasted precipitation and wind speeds in areas of complex terrain (Nachamkin and Jin 2017; Reynolds et al. 2019; Doyle et al. 2019).
Although studies have highlighted the skill of NWP for AR characteristics, there has been limited investigation of precipitation forecast skill by high resolution models more generally for heavy precipitation episodes in California. Although very important contributors to extreme precipitation totals and coastal flooding, ARs explain less than 50% of California’s total precipitation (Dettinger et al. 2011). The objective of this study is to contribute to future model improvements for the HRRR and COAMPS by diagnosing the ability of the modeling systems to accurately forecast heavy precipitation periods in the California region. The complexity of terrain-flow interactions and orographic enhancement in the coastal and interior regions of the state are challenging for operational models and relevant to other regions for which the models are used.
The operational models, Stage-IV precipitation analyses, and methodology are described in section 2. Characteristics of the model precipitation forecasts are illustrated using a heavy precipitation event during 6–8 December 2019. Initial 12-h precipitation forecasts (F01–F12) from the HRRR, COAMPS, and NAM-3km are validated in section 3 relative to the Stage-IV gridded precipitation analyses during the 1 December 2018–28 February 2019 season, which was California’s second wettest winter season during the past 20 years. In addition, the Fraction skill score metric is used across ranges of spatial scales and precipitation values to assess the relative model performance as a function of precipitation intensity and location. A summary of the results from this research is found in section 4.
2. Data and methods
a. Precipitation data
The 6-h Stage-IV precipitation gridded analyses produced by the NCEP (Lin and Mitchell 2005) are used in this study. These precipitation analyses are a national multisensor analysis at 4-km grid spacing over the CONUS that is blended from 1–6-h precipitation estimates from River Forecast Centers. The California-Nevada and Northwest River Forecast Centers use the PRISM/Mountain Mapper approach to produce gauge-based analyses (Hou et al. 2014).
Although not shown here, the Stage-IV analyses were evaluated by Dougherty (2020) using precipitation observations from the National Weather Service (NWS), Remote Automated Weather Stations (RAWS), and Hydrometeorological Automated Data System (HADS) station networks. Concerns about the representativeness of station observations at high elevation in California and the many sources of uncertainty for both the observations and analyses in such regions are well recognized (Henn et al. 2020; English et al. 2021). In total, 864 precipitation measuring stations were judged after automated and subjective quality control to provide reasonable 12-h precipitation totals during the 1 December 2018–28 February 2019 winter season. Dougherty (2020) found that the Stage-IV analyses generally were within 1 mm on average of the observations at lower elevation locations. However, the analyses at some locations in the coastal ranges and higher elevations of the interior mountains were greater than those observed, which is consistent with prior studies that found Stage-IV analyses to overestimate observations in California during winter (Nelson et al. 2016). Overall, Dougherty (2020) found for 12-h precipitation totals, the Stage-IV analyses were useful for evaluating the model precipitation forecasts in California and are used for validation of model forecasts here.
Although the southwestern United States has been experiencing extensive drought during the past two decades (Williams et al. 2020), California experienced many heavy precipitation events and flooding episodes during the 2016/17 and 2018/19 winters. However, only 1–2 heavy precipitation events occurred during the recent 2019/20 and 2020/21 winters. Figure 1 shows the average Stage-IV 12-h measured precipitation from 0000 UTC 1 December 2018 to 1200 UTC 28 February 2019 computed for the land areas in Fig. 2b. The lower areal-averaged precipitation totals during December 2018 were followed later in the season by several major episodes with widespread heavy precipitation and flooding. The 12-h periods with domain-averaged precipitation exceeding 2.5 mm (0.01 in.) are defined as the study’s 48 precipitation periods during 9 events lasting from 12 to 96 h.
Average precipitation (mm) within the analysis domain from Stage-IV 12 h analyses from 0000 UTC 1 Dec 2018 to 1200 UTC 28 Feb 2019. Periods in excess of 2.5 mm (above red line) are analyzed as heavy precipitation periods.
Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-20-0182.1
(a) HRRR and NAM-3km CONUS domains are in red. COAMPS regional domain is in blue. (b) HRRR topography is based on the color bar below.
Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-20-0182.1
b. Models
The HRRR has been developed over many years by the Earth Systems Research Laboratory (Benjamin et al. 2016). Operational versions of the HRRR are managed by NCEP’s Environmental Modeling Center. The operational model version examined in this study is HRRRv3 that produced forecasts from F00 to F18 every hour except at 0000, 0600, 1200, and 1800 UTC when the forecasts extended from F00 to F36. NCEP completed implementation of HRRRv4 in December 2020.
The HRRR is a convection-allowing, hourly updated model that relies on the Advanced Research Weather Research and Forecasting dynamical core with 3-km grid spacing over the CONUS as shown in Fig. 2. The HRRR uses the NOAA Gridpoint Statistical Interpolation data assimilation process (Wu et al. 2002; Whitaker et al. 2008; Kleist et al. 2009) as well as an assimilation of radar reflectivity every 15 min over 1 h (Benjamin et al. 2016). Model characteristics are summarized in Table 1. We relied on the archive of model analyses and forecasts available from the HRRR Pando archive system at the University of Utah’s Center for High-Performance Computing described by Blaylock et al. (2017). The HRRR archive at the University of Utah focuses on storing two-dimensional analysis and forecast fields and three-dimensional analysis fields that have been used by hundreds of researchers (e.g., McCorkle et al. 2018; Blaylock et al. 2018; Blaylock and Horel 2020; Moore et al. 2020).
Characteristics of forecast models.
COAMPS has also undergone substantive development over the years by the Marine Meteorological Division of the NRL and is run operationally by the Fleet Numerical Meteorology and Oceanography Center. The model uses 4-km grid spacing and can be applied as a nested grid anywhere on the globe (Hodur 1997). COAMPS can be configured as a standalone atmosphere or ocean forecast model or run as a coupled modeling system. This research relies on coupled model simulations run routinely over a Northern and Central California domain with forecasts extending out to 48 h (Fig. 2). The model is initialized daily at 0000, 0600, 1200, and 1800 UTC using the NRL’s Atmospheric Variational Data Assimilation System (NAVDAS; Daley and Barker 2001) with 60 sigma-z levels from 10 m to approximately 50 km. Model characteristics are summarized in Table 1. While COAMPS can assimilate radar reflectivity, it was not activated for the Central California domain used in this study. Since output from the 0600 and 1800 UTC model runs are not archived, only the 0000 and 1200 UTC COAMPS forecasts are available for this research with special efforts made to download the precipitation forecasts for the entire 2018/19 winter season.
The NAM-3km underwent its final upgrade in 2017 yet is continuing to be run operationally by the NCEP (Rogers et al. 2017). The NAM-3km is used in this research to compare the skill of HRRR and COAMPS to a frozen model whose performance characteristics are often familiar to operational forecasters. The NAM-3km has a horizontal resolution of 3 km over the same domain as HRRR. It is initialized daily at 0000, 0600, 1200, and 1800 UTC and produces hourly forecasts out to 60 h. Many of the model specifications for the NAM-3km are the same as the HRRR as summarized in Table 1. Selected fields from the NAM-3km are downloaded from the NOAA Operational Model Archive and Distribution System (NOMADS) and saved locally on servers maintained by the Center for High Performance Computing.
c. 6–8 December 2019
Precipitation analyses and model forecasts are illustrated using a widespread precipitation event across California during 6–8 December 2019. We selected pragmatically this case (one of the wettest during recent years) based on data available to us as well as desiring an event independent from the sample of events during the 2018/19 winter. The progression of a midlatitude cyclone across the state with an embedded atmospheric river contributed to more than 100 mm (3.64 in.) of precipitation falling in the coastal and interior mountain ranges in Northern California (Fig. 3). The maximum amount observed in this region during this period was 204 mm (8 in.) in the northern Coast Range. Localized areas in the northern Coast Range were analyzed to receive more than 150 mm (5.9 in.) of accumulated precipitation during the 72-h period between 1200 UTC 5 December and 1200 UTC 8 December 2019 (Fig. 3f).
Stage-IV analyses of 12-h accumulated precipitation (mm) beginning with the 1200 UTC 5 Dec 2019–0000 UTC 6 Dec 2019 period and continuing until the 0000 UTC 8 Dec 2019–1200 UTC 8 Dec 2019 period shaded according to the color bar.
Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-20-0182.1
Other notable features in Fig. 3 include the large precipitation amounts to the north of the Central Valley, which will be referred to as the Mt. Shasta area. The coastal ranges south of San Francisco also saw substantial accumulations of precipitation with lesser amounts in Southern California. Areas in the Sierra Nevada above 2500 m received between 30 and 60 cm (1–2 ft) of snow. Two flood advisories were issued by the Sacramento NWS Weather Forecasting Office for Shasta and El Dorado Counties during the afternoon of 7 December 2019 when the heaviest precipitation rates were observed (Castellano et al. 2019). This storm resulted in localized flooding throughout Northern California including flooded houses in a San Francisco neighborhood and rockslides along coastal highways south of San Francisco.
Initial 12-h accumulated precipitation forecasts from the HRRR and COAMPS during 6–9 December 2019 are shown in Figs. 4 and 5. Both models captured the general precipitation features with well-defined maxima along the northern Coast Range, Mt. Shasta area, and Sierras. Lesser precipitation amounts are forecasted to develop during the latter stages of the event in Southern California. However, HRRR forecasts tended to under forecast the orographic precipitation in the coastal ranges and Mt. Shasta area as the event evolved (e.g., compare Figs. 3d and 4d) while COAMPS tended to overpredict precipitation amounts in those areas as well as the Sierra Nevada (e.g., compare Figs. 3d and 4d). The narrow offshore banded features evident in the model forecasts from both models indicate the progression of an AR. However, the precipitation accumulations onshore result from many additional factors involving multiscale terrain-flow interactions. This event, and others not shown, was the impetus to investigate objectively the skill of the HRRR and COAMPS forecasts during the prior December 2018–February 2019 winter season when a larger number of precipitation events occurred.
The 12-h accumulated precipitation (mm) beginning with HRRR forecast initialized at 1200 UTC 5 Dec 2019 for the 1200 UTC 5 Dec 2019–0000 UTC 6 Dec 2019 period and continuing until HRRR forecast initialized at 0000 UTC 8 Dec 2019 for the 0000 UTC 8 Dec 2019–1200 UTC 8 Dec 2019 period. Shading is according to the color bar.
Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-20-0182.1
As in Fig. 4, but for 12-h accumulated precipitation (mm) from COAMPS forecasts initialized from 1200 UTC 5 Dec 2019 to 0000 UTC 8 December 2019.
Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-20-0182.1
d. Verification methods
Statistical metrics (frequency bias, hit rate, false alarm ratio, and equitable threat score, ETS) as a function of precipitation thresholds are used to evaluate the deterministic precipitation forecasts (Wilks 2011). The frequency bias is the ratio of the total number of periods forecasted above a specified threshold to the total number of periods observed above that threshold. The hit rate and false alarm ratio are the fractions of occurrences that were correctly forecasted and that were predicted, but did not occur, respectively. The ETS is the proportion of observed and/or forecasted periods that were correctly forecasted, adjusted for the frequency of hits expected by chance. To calculate these scores, model values were interpolated to the Stage-IV grid using nearest-neighbor interpolation. Bilinear interpolation was also applied and produced nearly identical results.
While those traditional point-based statistical measures provide insight on model skill at the resolution of the model grid spacing, they often miss the complexities arising from differences in model and actual terrain or the inherent spatial and temporal predictability issues associated with precipitation processes. The fractions skill score (FSS) is an approach to assess the relative agreement between observed and forecasted precipitation within neighborhoods without expecting their precise locations or amounts to match (Roberts and Lean 2008; Mittermaier and Roberts 2010; Wolff et al. 2014; Skok 2016; Blaylock and Horel 2020). FSS is calculated here iteratively centered on every grid point within circular neighborhoods from 3- to 70-km radius. The FSS ranges from zero (no skill) to one (perfect forecast). However, FSS calculated over large neighborhoods asymptote to values that are proportional to the frequency bias (Roberts and Lean 2008). Hence, we focus on FSS calculated using radial distances of ∼30 km or less, i.e., scales comparable to that of the width of many California hydrologic basins. For FSS statistics aggregated over the entire domain, the number of spatially independent values vary from ∼30–15 000 for the range of 70–3-km radial neighborhoods, with roughly 150–300 spatially independent values for 20–30-km radial distances (Blaylock and Horel 2020).
As many have done, we compute FSS in a manner similar to ETS where a successful forecast is one where the intensity of the observed and forecasted precipitation only needs to exceed a lower threshold. For example, if the lower threshold is 1 mm, then FSS would be high if forecast and analyzed values have similar coverage amounts between 1 mm and the maximum recorded in that time interval (in some cases over 75 mm during the 12-h periods). To illustrate an alternative approach, we add an upper limit such that an accurate forecast must have similar coverage amounts within broad ranges (e.g., the 10–50-mm range) of analyzed values.
3. 2018/19 season statistics
a. Bias
Accumulated precipitation forecasted by the HRRR, COAMPS, and NAM-3km are evaluated for the periods from 0000–1200 to 1200–0000 UTC initialized at those starting hours during the 90 days of the 2018/19 winter season. The seasonal averages of measurable 12-h precipitation totals from the Stage-IV analyses and from the 12-h forecasts from the three models are shown in Fig. 6. The highest 12-h average precipitation is analyzed to be in the coastal mountains of Northern California and southern Oregon (Fig. 6a). HRRR and COAMPS 12-h forecasts (Figs. 6b,c) subjectively compare well with the Stage-IV analyses, with differences noticeable in the northern Sierras and along the northern coast of California. The NAM-3km average forecasted precipitation (Fig. 6d) differs from that analyzed or forecasted by the other models with accumulations much higher throughout most of the domain, which is consistent with the model’s recognized wet bias. All three models also tend to forecast more precipitation in Nevada and eastern Oregon compared to the Stage-IV analyses.
Mean 12-h precipitation totals for (a) Stage-IV, (b) HRRR, (c) COAMPS, and (d) NAM-3km during DJF 2018/19 shaded according to the color bar.
Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-20-0182.1
The seasonal bias ratio for each model is shown in Fig. 7 and calculated by dividing the total of the 12-h precipitation forecasts (Figs. 7b–d) by the analyzed seasonal total precipitation (Fig. 7a). The bias ratio is most relevant over and to the west of the Sierra Nevada where areas receive ample amounts of seasonal rainfall. The large wet and dry biases in the lee of the Sierras, Nevada, and eastern Oregon result primarily from relatively small discrepancies of the model forecasts from the small seasonal precipitation totals in those regions.
(a) Bias ratio for 12-h precipitation totals for the HRRR relative to the Stage-IV analysis totals during DJF 2018/19 shaded according to the color bar. (b) As in (a), but for COAMPS. (c) As in (a), but for the NAM-3km model.
Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-20-0182.1
Of the three models, HRRR exhibits the smallest bias ratios in most areas. HRRR has a dry bias along the coast stretching from the Monterey Bay area north into southern Oregon, which corresponds to the region with the highest precipitation amounts analyzed (Fig. 7a). The HRRR also has a dry bias ratio in the northernmost sections of the Sacramento Valley and a wet bias ratio in the southern Sierra Nevada Mountains and isolated mountain peaks in Northern California. These are generally high elevation areas likely influenced by substantive orographic enhancement. COAMPS has a large positive bias ratio, especially in the Central Valley where limited precipitation was recorded during this season. The wet bias extends into the central and southern Sierra Nevada. COAMPS also exhibits a dry bias along the northern coast of California similar to HRRR, however, it is not as extensive. There is also a strong dry bias along the Transverse Ranges in Southern California. The NAM-3km exhibits a substantial wet bias across much of the domain compared to both the HRRR and COAMPS.
b. Skill scores during heavy precipitation episodes
Deterministic skill scores are applied to the model forecasts during 48 12-h periods. Overall, HRRR shows the highest skill independent of the metric used and precipitation range during the 48 periods (Fig. 8). All these metrics are computed for 7 bins defined in the panels by the lower thresholds, i.e., the leftmost bins consider all measurable 12-h precipitation totals from the Stage-IV analyses (above 0.25 mm, 0.01 in.) while the rightmost bins are restricted to heavy precipitation amounts only (≥75 mm, ∼3 in.). The slightly above one HRRR frequency bias scores indicate a tendency for overprediction that likely results from the large areal coverage of the unrealistic values over Nevada and southeastern Oregon (Fig. 8a). COAMPS and NAM-3km forecasts tend to have similar skill scores for precipitation amounts up to 10 mm (0.4 in.).
Model skill scores for 48 12-h model precipitation periods relative to Stage-IV analyses for (a) equitable threat score, (b) frequency bias, (c) hit rate, and (d) false alarm ratio. The x-axis position of the filled circles indicates the starting threshold value used to calculate the skill score. Blue, green, and orange lines indicate HRRR, COAMPS, and NAM-3km, respectively.
Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-20-0182.1
On the basis of the FSS metric, the top row of Fig. 9 highlights the ability of the models to predict the areas experiencing precipitation as a function of spatial scale and precipitation threshold. As discussed in section 2, FSS is calculated separately for over 15 000 grid points iterating over neighborhoods with radial distances from 3 km (28 km2 area) to 70 km (15 394 km2). The relative ability of the models to have high FSS values on the mesoscale, which corresponds to that of many California hydrologic basins (radial distances of ∼20–30 km), is of greatest interest for this study. FSS values for the lowest threshold (0.254 mm) correspond to evaluating the coverage of measurable precipitation, i.e., the extent to which forecasted areas of measurable precipitation overlap with those analyzed. Based on the typical overall coverage of analyzed precipitation across the domain, FSS values in excess of 0.6 can be considered “useful” here (Roberts and Lean 2008; Blaylock and Horel 2020).
FSS computed for (a) the HRRR, (b) COAMPS, and (c) the NAM-3km model. The x axis indicates the radial distance (km). Thresholds of 12-h precipitation totals used to calculate FSS are indicated in the legend. (d)–(f) As in (a)–(c), but FSS was computed using threshold ranges.
Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-20-0182.1
All three models are able to identify areas where measurable or greater precipitation has been analyzed on the mesoscale (e.g., 20–30 km) with FSS values in excess of 0.8 (Fig. 9 top row). For thresholds up to 25 mm (∼1 in.), HRRR and NAM-3km models have better correspondence between areal coverage of precipitation in excess of those thresholds than that from COAMPS.
Although of greatest importance for public safety, none of the models would be particularly useful at forecasting the specific locations of heavy precipitation in the Stage-IV analyses during the 12-h periods (areas with amounts in excess of 50 mm within 30-km radial distances). There were ∼21 000 grid points that measured precipitation greater than 50 mm during the season, which is ∼1% of the 12-h Stage-IV precipitation values within this domain during the entire season. Relative to the other two models, the HRRR has higher FSS values for the even smaller sample (∼2000) grid points where the analyzed 12-h precipitation totals exceed 75 mm (∼3 in.).
As discussed in the methods section, the FSS approach used in the top row of Fig. 9 evaluates the coverage of forecast and analyzed precipitation values exceeding a threshold. Hence, when a small threshold is used for FSS, a skillful forecast may have large discrepancies between what is forecasted and analyzed within the region. To address this issue, the FSS shown in the bottom row of Fig. 9 is calculated using specified precipitation ranges: no measurable precipitation within the 12-h period (0–0.254 mm), light (0.254–5 or 0.254–10 mm), moderate (5–25 or 10–50 mm), and heavy (25–75 or greater than 50 mm).
As evident from the bottom row of Fig. 9, all models do better at forecasting the coverage of moderate 12-h precipitation amounts (5–25 or 10–50 mm). All models have less accuracy explicitly forecasting the coverage of no- or light-precipitation amounts or, as already evident in the top row of Fig. 9, where precipitation is analyzed to be greater than 50 mm. FSS from the HRRR model in the 10–50-mm range is ∼0.87 for 20–30-km radial neighborhoods, which is substantively higher than that from COAMPS and the NAM-3km. This result may be an important finding since forecasters faced with an episode of sustained moderate precipitation might have higher confidence that flooding could occur.
To examine the spatial variability in model forecast skill, Fig. 10 examines FSS within a 28-km radius separately for every grid point during the 48 12-h periods. The top row in Fig. 10 shows FSS for each model for precipitation in excess of 10 mm. All models have low skill in the southern San Joaquin Valley and to the east of the Sierras with the highest skill in most regions across Northern California. The HRRR has higher skill throughout most of the domain relative to the other two models except in regions with large bias ratio scores (Fig. 7) and likely large false alarm rate (Fig. 8). The HRRR and COAMPS have noticeably higher skill than the NAM-3km in the southern half of the state (equatorward of 38°N) over the coastal ranges, Sierras, and Los Angeles and San Diego regions.
(a)–(c) Average FSS for 28-km radius during all 48 periods for precipitation amounts above 10 mm. (d)–(f) As in (a)–(c), but for precipitation amounts between 10 and 50 mm.
Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-20-0182.1
The panels in the lower row of Fig. 10 are computed similarly to those in the upper row after removing the ∼1% of times when the analyzed precipitation values exceeded 50 mm during the 12-h periods, i.e., restricting the analysis to moderate analyzed precipitation amounts between 10 and 50 mm (∼0.4–2 in.). While the differences between the upper and lower panels are typically small since few locales received over 50 mm frequently, the lower panels help quantify the models’ relative skill for substantive precipitation amounts. The lower panels also illustrate the HRRR’s apparent greater skill statewide with noticeably higher skill in the coastal regions south of the San Francisco Bay region and southern Sierras.
5. Summary
While HRRR winter precipitation forecasts in California and the western states have been investigated (e.g., Gowan et al. 2018; Caron and Steenburgh 2020; English et al. 2021), COAMPS precipitation forecasts have not received such attention. HRRR and COAMPS initial 12-h precipitation forecasts were compared to the Stage-IV analyses during the 6–8 December 2019 precipitation event to highlight subjectively model features that were found to be common within a larger independent 48-member forecast sample during the 2018/19 season.
Overall, the accuracy of HRRR 12-h precipitation forecasts during the 2018/19 season was higher than that relative to both the COAMPS and the NAM-3km model. On the basis of all 90 of the 12-h forecasts during the season, NAM 3-km had a strong wet bias and a high orographic precipitation enhancement compared to COAMPS or HRRR. COAMPS forecasts were excessive in the San Joaquin Valley while the HRRR underestimated precipitation amounts in the northern coastal ranges more than the other models. During the 48 heavy precipitation periods and considering all grid points, HRRR had smaller overall bias ratios, higher ETS scores, lower false alarm rates, and higher FSS for all precipitation ranges. The improvement in ETS or bias scores provided by the HRRR relative to COAMPS or the NAM-3km model was most pronounced for the rare occasions when precipitation exceeded 50 mm (2 in.).
Based on FSS computed within precipitation ranges, all models tended to predict moderate amounts of precipitation (10–50 mm) better than smaller and larger amounts of precipitation with the HRRR exhibiting the greatest skill. When comparing the results of the FSS on scales typical for California river basins, the models tended to do better for the many locations where moderate precipitation amounts were more frequent (i.e., coastal and interior mountain ranges of California) and were poor where those were rare (San Joaquin valley and east of the Sierra mountains).
While the total number of heavy winter precipitation events in California has been lower during recent years, their impacts remain high. Improving high-resolution models for such heavy precipitation events is vital for weather forecasters, hydrologists, and emergency managers to enhance public safety in California (Corringham et al. 2019). These episodes also provide rich opportunities to understand storm-scale structure of storms making landfall, formation of low-level jets along the coast and within interior valleys, and terrain-flow interactions. Future work will need to examine the complex interactions of the model dynamics, particularly low-level jet structure, and physical treatment of stratiform and convective precipitation, with particular attention on both warm and mixed-phase precipitation processes. Sensitivity to the data assets available as part of the models’ assimilation procedures and impacts of the nearby western boundary of the regional models should also be explored. Our examination of the HRRRv3 remains highly relevant for operational uses of the HRRRv4 as the improvements introduced with the latest release of that model are ones less likely to affect substantively the model’s performance for heavy precipitation events (English et al. 2021). Of particular interest for the future will be to examine research versions of HRRR and COAMPS that provide ensemble forecasts in order to better understand the predictability of these storms.
Acknowledgments
We thank the Naval Research Enterprise Internship Program (NREIP) for funding an internship at the Naval Research Lab–Marine Meteorology Division (NRL-MMD) where this work was partially completed. We would also like to thank the NRL-MMD for supplying computer resources as well as COAMPS forecast data to complete this study. The work at the University of Utah was supported by the Collaborative Science, Technology, and Applied Research (CSTAR) Program Award (55500146). Stage IV data access was provided by the Earth Observing Laboratory web service. Computational hardware and software were provided by the University of Utah Center for High Performance Computing (CHPC) to complete this research. We also appreciate the comments of the anonymous reviewers who helped us to focus the final manuscript.
Data availability statement.
Metadata related to the model output from the HRRR are available in the University of Utah Hive data repository: doi: 10.7278/S5JQ0Z5B. HRRR model grids used in this study were archived at http://hrrr.chpc.utah.edu/ (Blaylock et al. 2017) and now available as part of the Amazon Web Service Open Data Program and its partnership with the National Oceanic and Atmospheric Administration Big Data Program. Other data used in this study are available from the authors upon request.
REFERENCES
Aligo, E. A., B. Ferrier, and J. R. Carley, 2018: Modified NAM microphysics for forecasts of deep convective storms. Mon. Wea. Rev., 146, 4115–4153, https://doi.org/10.1175/MWR-D-17-0277.1.
Benjamin, S. G., and Coauthors, 2016: A North American hourly assimilation and model forecast cycle: The Rapid Refresh. Mon. Wea. Rev., 144, 1669–1694, https://doi.org/10.1175/MWR-D-15-0242.1.
Blaylock, B. K., and J. D. Horel, 2020: Comparison of lightning forecasts from the High-Resolution Rapid Refresh model to Geostationary Lightning Mapper observations. Wea. Forecasting, 35, 401–416, https://doi.org/10.1175/WAF-D-19-0141.1.
Blaylock, B. K., J. D. Horel, and S. T. Liston, 2017: Cloud archiving and data mining of High-Resolution Rapid Refresh forecast model output. Comput. Geosci., 109, 43–50, https://doi.org/10.1016/j.cageo.2017.08.005.
Blaylock, B. K., J. D. Horel, and C. Galli, 2018: High-Resolution Rapid Refresh model data analytics derived on the open science grid to assist wildland fire weather assessment. J. Atmos. Oceanic Technol., 35, 2213–2227, https://doi.org/10.1175/JTECH-D-18-0073.1.
Bridger, A. F., D. Nguyen, and S. Chiao, 2019: Developing spatially accurate rainfall predictions for the San Francisco Bay area through case studies of atmospheric river and other synoptic events. Atmosphere, 10, 541, https://doi.org/10.3390/atmos10090541.
Bytheway, J. L., C. D. Kummerow, and C. Alexander, 2017: A features-based assessment of the evolution of warm season precipitation forecasts from the HRRR model over three years of development. Wea. Forecasting, 32, 1841–1856, https://doi.org/10.1175/WAF-D-17-0050.1.
Cao, Q., A. Mehran, F. M. Ralph, and D. P. Lettenmaier, 2019: The role of hydrological initial conditions on atmospheric river floods in the Russia River basin. J. Hydrometeor., 20, 1667–1686, https://doi.org/10.1175/JHM-D-19-0030.1.
Caron, M., and W. J. Steenburgh, 2020: Evaluation of recent NCEP operational model upgrades for cool-season precipitation forecasting over the western conterminous United States. Wea. Forecasting, 35, 857–877, https://doi.org/10.1175/WAF-D-19-0182.1.
Castellano, C., C. Hecht, B. Kawzenuk, and F. M. Ralkd, 2019: CW3E event summary: 6–8 December 2019. Center for Western Weather and Water Extremes, accessed 20 November 2021, https://cw3e.ucsd.edu/cw3e-event-summary-6-8-december-2019.
Colle, B. A., 2004: Sensitivity of orographic precipitation to changing ambient conditions and terrain geometries: An idealized modeling perspective. J. Atmos. Sci., 61, 588–606, https://doi.org/10.1175/1520-0469(2004)061<0588:SOOPTC>2.0.CO;2.
Corringham, T., F. Ralph, A. Gershunov, and D. Cayan, 2019: Atmospheric rivers drive flood damages in the western United States. Sci. Adv., 5, eaax4631, https://doi.org/10.1126/sciadv.aax4631.
Daley, R., and E. Barker, 2001: NAVDAS: Formulation and diagnostics. Mon. Wea. Rev., 129, 869–883, https://doi.org/10.1175/1520-0493(2001)129<0869:NFAD>2.0.CO;2.
Darby, L. S., A. B. White, D. J. Gottas, and T. Coleman, 2019: An evaluation of integrated water vapor, wind, and precipitation forecasts using water vapor flux observations in the western United States. Wea. Forecasting, 34, 1867–1888, https://doi.org/10.1175/WAF-D-18-0159.1.
Dettinger, M. D., 2013: Atmospheric rivers as drought busters on the U.S. West Coast. J. Hydrometeor., 14, 1721–1732, https://doi.org/10.1175/JHM-D-13-02.1.
Dettinger, M. D., F. M. Ralph, T. Das, P. J. Neiman, and D. Cayan, 2011: Atmospheric rivers, floods, and the water resources of California. Water, 3, 445–478, https://doi.org/10.3390/w3020445.
Dougherty, K. J., 2020: Evaluation of the High-Resolution Rapid Refresh and the coupled ocean-atmosphere mesoscale prediction system during atmospheric river events in California. M.S. thesis, Department of Atmospheric Sciences, The University of Utah, 70 pp., https://hive.utah.edu/concern/file_sets/rf55z7756.
Doyle, J. D., C. A. Reynolds, and C. Amerault, 2019: Adjoint sensitivity analysis of high-impact extratropical cyclones. Mon. Wea. Rev., 147, 4511–4532, https://doi.org/10.1175/MWR-D-19-0055.1.
English, J. M., D. D. Turner, T. I. Alcott, W. R. Moninger, J. L. Bytheway, R. Cifelli, and M. Marquis, 2021: Evaluating operational and experimental HRRR model forecasts of atmospheric river events in California. Wea. Forecasting, 36, 1925–1944, https://doi.org/10.1175/WAF-D-21-0081.1.
Florsheim, J., and M. Dettinger, 2015: Promoting atmospheric-river and snowmelt-fueled biogeomorphic processes by restoring river-floodplain connectivity in California’s Central Valley. Geomorphic Approaches to Integrated Floodplain Management of Lowland Fluvial Systems in North America and Europe, P. Hudson and H. Middelkoop, Eds., Springer, 119–141, https://doi.org/10.1007/978-1-4939-2380-9_6.
Gershunov, A., and Coauthors, 2019: Precipitation regime change in Western North America: The role of atmospheric rivers. Sci. Rep., 9, 9944, https://doi.org/10.1038/s41598-019-46169-w.
Gowan, T. M., W. J. Steenburgh, and C. S. Schwartz, 2018: Validation of mountain precipitation forecasts from the convective-permitting NCAR ensemble and operational forecast systems over the western United States. Wea. Forecasting, 33, 739–765, https://doi.org/10.1175/WAF-D-17-0144.1.
Haack, T., and S. D. Burk, 2001: Summertime marine refractivity conditions along coastal California. J. Appl. Meteor., 40, 673–687, https://doi.org/10.1175/1520-0450(2001)040<0673:SMRCAC>2.0.CO;2.
Haack, T., D. Chelton, J. Pullen, J. D. Doyle, and M. Schlax, 2008: Summertime influence of SST on surface wind stress off the U.S. West Coast from the U.S. Navy COAMPS model. J. Phys. Oceanogr., 38, 2414–2437, https://doi.org/10.1175/2008JPO3870.1.
Henn, B., K. N. Musselman, L. Lestak, F. M. Ralph, and N. P. Molotch, 2020: Extreme runoff generation from atmospheric river driven snowmelt during the 2017 Oroville dam spillways incident. Geophys. Res. Lett., 47, e2020GL088189, https://doi.org/10.1029/2020GL088189.
Hodur, R. M., 1997: The Naval Research Laboratory’s Coupled Ocean/Atmosphere Mesoscale Prediction System (COAMPS). Mon. Wea. Rev., 125, 1414–1430, https://doi.org/10.1175/1520-0493(1997)125<1414:TNRLSC>2.0.CO;2.
Hou, D., and Coauthors, 2014: Climatology-calibrated precipitation analysis at fine scales: Statistical adjustment of Stage-IV toward CPC gauge-based analysis. J. Hydrometeor., 15, 2542–2557, https://doi.org/10.1175/JHM-D-11-0140.1.
Hsu, H., L. Oey, W. Johnson, C. Dorman, and R. Hodur, 2007: Model wind over the central and Southern California coastal ocean. Mon. Wea. Rev., 135, 1931–1944, https://doi.org/10.1175/MWR3389.1.
Kleist, D. T., D. F. Parrish, J. C. Derber, R. Treadon, W.-S. Wu, and S. Lord, 2009: Introduction of the GSI into the NCEP Global Data Assimilation System. Wea. Forecasting, 24, 1691–1705, https://doi.org/10.1175/2009WAF2222201.1.
Konrad, C. P., and M. D. Dettinger, 2017: Flood runoff in relation to water vapor transport by atmospheric rivers over the western United States, 1949–2015. Geophys. Res. Lett., 44, 11 456–11 462, https://doi.org/10.1002/2017GL075399.
Kunz, M., and S. Wassermann, 2011: Sensitivity of flow dynamics and orographic precipitation to changing ambient conditions in idealized model simulations. Meteor. Z., 20, 199–215, https://doi.org/10.1127/0941-2948/2011/0221.
Lavers, D. A., D. E. Waliser, F. M. Ralph, and M. D. Dettinger, 2016: Predictability of horizontal water vapor transport relative to precipitation: Enhancing situational awareness for forecasting western U.S. extreme precipitation and flooding. Geophys. Res. Lett., 43, 2275–2282, https://doi.org/10.1002/2016GL067765.
Lin, Y., and K. E. Mitchell, 2005: The NCEP Stage II/IV hourly precipitation analyses: Development and applications. Preprints, 19th Conf. on Hydrology, San Diego, CA, Amer. Meteor. Soc., 1.2, https://ams.confex.com/ams/Annual2005/techprogram/paper_83847.htm.
Liu, M., J. E. Nachamkin, and D. L. Westphal, 2009: On the improvement of COAMPS weather forecasts using an advanced radiative transfer model. Wea. Forecasting, 24, 286–306, https://doi.org/10.1175/2008WAF2222137.1.
Martin, A. C., F. M. Ralph, A. Wilson, L. DeHaan, and B. Kawzenuk, 2019: Rapid cyclogenesis from a mesoscale frontal wave on an atmospheric river: Impacts on forecast skill and predictability during atmospheric river landfall. J. Hydrometeor., 20, 1779–1794, https://doi.org/10.1175/JHM-D-18-0239.1.
McCorkle, T. A., J. D. Horel, A. A. Jacques, and T. Alcott, 2018: Evaluating the experimental High-Resolution Rapid Refresh–Alaska modeling system using USArray pressure observations. Wea. Forecasting, 33, 933–953, https://doi.org/10.1175/WAF-D-17-0155.1.
Mittermaier, M., and N. Roberts, 2010: Intercomparison of spatial forecast verification methods: Identifying skillful spatial scales using the fractions skill score. Wea. Forecasting, 25, 343–354, https://doi.org/10.1175/2009WAF2222260.1.
Moore, B. J., and B. A. White, D. J. Gottas, and P. J. Neiman, 2020: Extreme precipitation events in Northern California during winter 2016–17: Multiscale analysis and climatological perspective. Mon. Wea. Rev., 148, 1049–1074, https://doi.org/10.1175/MWR-D-19-0242.1.
Nachamkin, J. E., and Y. Jin, 2017: An Eulerian framework for event-based pattern verification. Wea. Forecasting, 32, 2027–2043, https://doi.org/10.1175/WAF-D-17-0065.1.
Nardi, K. M., E. A. Barnes, and F. M. Ralph, 2018: Assessment of numerical weather prediction model reforecasts of the occurrence, intensity, and location of atmospheric rivers along the West Coast of North America. Mon. Wea. Rev., 146, 3343–3362, https://doi.org/10.1175/MWR-D-18-0060.1.
Neiman, P. J., F. M. Ralph, A. B. White, D. E. Kinsmill, and P. O. Persson, 2002: The statistical relationship between upslope flow and rainfall in California’s coastal mountains: Observations during CALJET. Mon. Wea. Rev., 130, 1468–1492, https://doi.org/10.1175/1520-0493(2002)130<1468:TSRBUF>2.0.CO;2.
Neiman, P. J., L. J. Schick, F. M. Ralph, M. Hughes, and G. A. Wick, 2011: Flooding in western Washington: The connection to atmospheric rivers. J. Hydrometeor., 12, 1337–1358, https://doi.org/10.1175/2011JHM1358.1.
Nelson, B. R., O. P. Prat, D. Seo, and E. Habib, 2016: Assessment and implications of NCEP Stage-IV quantitative precipitation estimates for product intercomparisons. Wea. Forecasting, 31, 371–394, https://doi.org/10.1175/WAF-D-14-00112.1.
Neveu, E., A. M. Moore, C. A. Edwards, J. Fiechter, P. Drake, W. J. Crawford, M. G. Jacox, and E. Nuss, 2016: An historical analysis of the California Current circulation using ROMS 4D-Var: System configuration and diagnostics. Ocean Modell., 99, 133–151, https://doi.org/10.1016/j.ocemod.2015.11.012.
Picard, L., and C. Mass, 2017: The sensitivity of orographic precipitation to flow direction: An idealized modeling approach. J. Hydrometeor., 18, 1673–1688, https://doi.org/10.1175/JHM-D-16-0209.1.
Ralph, F. M., and M. D. Dettinger, 2012: Historical and national perspectives on extreme West Coast precipitation associated with atmospheric rivers during December 2010. Bull. Amer. Meteor. Soc., 93, 783–790, https://doi.org/10.1175/BAMS-D-11-00188.1.
Ralph, F. M., P. J. Neiman, G. A. Wick, S. I. Gutman, M. D. Dettinger, D. R. Cayan, and A. B. White, 2006: Flooding on California’s Russian River: Role of atmospheric rivers. Geophys. Res. Lett., 33, L13801, https://doi.org/10.1029/2006GL026689.
Ralph, F. M., T. Coleman, P. J. Neiman, R. J. Zamora, and M. D. Dettinger, 2013: Observed impacts of duration and seasonality of atmospheric-river landfalls on soil moisture and runoff in coastal Northern California. J. Hydrometeor., 14, 443–459, https://doi.org/10.1175/JHM-D-12-076.1.
Ralph, F. M., M. D. Dettinger, M. M. Cairns, T. J. Galarneau, and J. Eylander, 2018: Defining “atmospheric river”: How the Glossary of Meteorology helped resolve a debate. Bull. Amer. Meteor. Soc., 99, 837–839, https://doi.org/10.1175/BAMS-D-17-0157.1.
Reynolds, C. A., J. D. Doyle, F. M. Ralph, and R. Demirdjian, 2019: Adjoint sensitivity of North Pacific atmospheric river forecasts. Mon. Wea. Rev., 147, 1871–1897, https://doi.org/10.1175/MWR-D-18-0347.1.
Roberts, N. M., and H. W. Lean, 2008: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev., 136, 78–97, https://doi.org/10.1175/2007MWR2123.1.
Rogers, E., and Coauthors, 2017: Mesoscale modeling development at the National Centers for Environmental Prediction: Version 4 of the NAM Forecast System and scenarios for the evolution to a high-resolution ensemble forecast system. 28th Conf. on Weather Analysis and Forecasting/24th Conf. on Numerical Weather Prediction, Seattle, WA, Amer. Meteor. Soc., 3B.4, https://ams.confex.com/ams/97Annual/webprogram/Paper311212.html.
Rutz, J. J., W. J. Steenburgh, and F. M. Ralph, 2014: Climatological characteristics of atmospheric rivers and their inland penetration over the western United States. Mon. Wea. Rev., 142, 905–921, https://doi.org/10.1175/MWR-D-13-00168.1.
Schmidt, J. M., 2001: Moist physics development for the Naval Research Laboratory’s Coupled Ocean/Atmosphere Mesoscale Prediction System (COAMPS). Battlespace Atmospheric and Cloud Impacts on Military Operations (BACIMO) Conf., Fort Collins, CO.
Skok, G., 2016: Analysis of fraction skill score properties for a displaced rainy grid point in a rectangular domain. Atmos. Res., 169, 556–565, https://doi.org/10.1016/j.atmosres.2015.04.012.
Stone, R. E., C. A. Reynolds, J. D. Doyle, R. H. Langland, N. L. Baker, D. A. Lavers, and F. M. Ralph, 2020: Atmospheric river reconnaissance observation impact in the Navy Global Forecast System. Mon. Wea. Rev., 148, 763–782, https://doi.org/10.1175/MWR-D-19-0101.1.
Thompson, G., and T. Eidhammer, 2014: A study of aerosol impacts on clouds and precipitation development in a large winter cyclone. J. Atmos. Sci., 71, 3636–3658, https://doi.org/10.1175/JAS-D-13-0305.1.
Whitaker, J. S., T. M. Hamill, X. Wei, Y. Song, and Z. Toth, 2008: Ensemble data assimilation with the NCEP Global Forecast System. Mon. Wea. Rev., 136, 463–482, https://doi.org/10.1175/2007MWR2018.1.
Wick, G. A., P. J. Neiman, F. M. Ralph, and T. M. Hamill, 2013: Evaluation of forecasts of the water vapor signature of atmospheric rivers in operational numerical weather prediction models. Wea. Forecasting, 28, 1337–1352, https://doi.org/10.1175/WAF-D-13-00025.1.
Wilks, D. S., 2011: Statistical Methods in the Atmospheric Sciences: An Introduction. 3rd ed. International Geophysics Series, Vol. 100, Academic Press, 704 pp.
Williams, A. P., and Coauthors, 2020: Large contribution from anthropogenic warming to an emerging North American megadrought. Science, 368, 314–318, https://doi.org/10.1126/science.aaz9600.
Wolff, J. K., M. Harrold, T. Fowler, J. H. Gotway, L. Nance, and B. G. Brown, 2014: Beyond the basics: Evaluating model-based precipitation forecasts using traditional, spatial, and object-based methods. Wea. Forecasting, 29, 1451–1472, https://doi.org/10.1175/WAF-D-13-00135.1.
Wu, W. S., R. J. Purser, and D. F. Parrish, 2002: Three-dimensional variational analysis with spatially inhomogeneous covariances. Mon. Wea. Rev., 130, 2905–2916, https://doi.org/10.1175/1520-0493(2002)130<2905:TDVAWS>2.0.CO;2.