a. Selection of the statistical method

To develop the ProxyVis algorithm, we used multiple linear regression because it is one of the simplest available methods that is easy to train and interpret. Further, linear regression is usually more robust than nonlinear ML methods when running the model with data outside of the training set. Thus, our algorithm is more likely to be successful when trained on VIIRS data and applied to multiple geostationary satellite sensors using approximately matching channels, and forecasters can easily understand how the individual channels are being used. Using only VIIRS data for development ensures that the independent variables (IR channels) and the “truth” data (DNB) see objects at the same angle and time from the same platform, while assuming that VIIRS and ABI channels are similar to first order.

b. Selection of channels

We used as predictors the VIIRS IR channels that closely match channels available on all current geostationary satellites. In the first step, similar VIIRS, GOES ABI, Himawari AHI, and MSG SEVIRI channels were identified (Table 2). These include three longwave IR channels: VIIRS channels M14 (8.55 μm), M15 (10.76 μm), and M16 (12.0 μm) corresponding to ABI channels C11 (8.5 μm), C13 (10.35 μm), and C15 (12.3 μm), and to AHI and SEVIRI channels provided in Table 2. The selection of a shortwave VIIRS IR channel matching ABI channel C07 (3.9 μm) and similar AHI and SEVIRI channels was more complicated since VIIRS does not have a channel centered at 3.9 μm, but has three similar channels, M12 (3.7 μm), M13 (4.05 μm), and I04 (3.74 μm). Channel I04 was selected based on the comparison of the spectral response of these VIIRS channels (Fig. 4 from Hillger and Kopp 2019) with the ABI 3.9-μm channel (Fig. 1. From Schmit et al. 2017). The broader I04 band contains much of the ABI’s 3.9-μm band range and the responses to dry air and water vapor are nearly identical. Finally, VIIRS DNB was used as the dependent variable. The DNB center wavelength (0.70 μm) closely matches ABI “red” channel C02 (0.64 μm) that ProxyVis is trying to approximate, and which has been long used for visible satellite applications (Kidder and Vonder Haar 1995).

Table 2.

Visible (Vis, DNB), near-infrared (NIR), and infrared (IR) channels that are similar between Suomi NPP and NOAA-20/21 VIIRS and GOES ABI, Himawari AHI, and MSG SEVIRI. Columns that show VIIRS values are highlighted in bold. The DNB provides dynamic range from day to night, while the ABI C02 and similar AHI and SEVIRI channels do not provide a nighttime visible capability. VIIRS “M” channels have nadir resolutions of 750 m, while VIIRS “I” channels have nadir resolutions of 375 m. VIIRS DNB channel has a resolution of 742 m across the swath. ABI and AHI IR channels have 2-km resolutions, ABI and AHI visible channels C02 and B03 have 0.5-km resolutions, and all SEVIRI channels listed in this table have 3-km resolutions.

View Table

c. Selection of training scenes and preprocessing of VIIRS DNB data

In total 16 ProxyVis training scenes (Table 3) were selected from the CIRA TC-centric VIIRS database (currently 2012–23) that includes Sensor Data Record data collocated with TCs from the ATCF (Sampson and Schrader 2000) final best tracks. The VIIRS database provides ∼10 min of data (∼2000 km) around TC centers for all bands for each available VIIRS/TC overpass. The most important features that we wanted to enhance in the new product were the low-level oceanic clouds. Thus, the training scenes had to satisfy the following three criteria. First, a scene must occur within three days of a full moon to ensure well-lit scenes. Second, it should not contain any land (or land points should be filtered out) to ensure that background emissivity and radiance are more homogeneous. Third, a scene should be composed only of clouds and their surface backgrounds, and not contain significant additional signals such as lightning, moon glint, city lights, ship lights, etc. The scenes satisfying the above three criteria were manually selected from the CIRA archive of TC-centric VIIRS DNB imagery by subjectively evaluating each available image within three days of the full moon. That reduced the number of available training scenes to less than 50 images. The final set of 15 TC training scenes was chosen subjectively by visual inspection of images to contain different cloud types [low, middle, and high level, as defined by WMO (2023)] over a wide range of SSTs, and TCs ranging from pre-TC disturbances and depressions to major hurricanes. One non-TC scene over the Gulf of Mexico was added to increase the number of developmental data points over very warm SST backgrounds.

Table 3.

Details of VIIRS scenes used to develop ProxyVis imagery, including the ATCF storm ID (see Table 1), the storm name, the latest best track intensity, the times associated with DNB data collection, the percent of full moon, and the day of the closest full moon. Storm intensity is provided in knots (kt), and information about the moon phase was obtained from the U.S. Naval Observatory (2023).

View Table

Next, developmental DNB data were scaled to produce reflectance-like images. The DNB radiances span eight orders of magnitude, from 1.0 × 10⁻¹⁰ W cm⁻² sr⁻¹ to at least 2.0 × 10² W cm⁻² sr⁻¹ (Cao 2013), thus the radiances need to be scaled to approximate visible reflectance (Miller and Turner 2009; Seaman and Miller 2015). For this study, we used “gamma DNB” scaling (appendix B) developed at CIRA in 2014 to display TC-centric DNB imagery on RAMMB/CIRA (2019). The “gamma DNB” is normalized between 0 and 1, making it a convenient independent variable for ProxyVis training. Further, the developmental VIIRS data from I-, M-, and DNB channels were regridded to a common grid with 148 samples per degree latitude, which is representative of DNB resolution in the tropics. After regridding, the 16 training scenes (Table 3) contain a total of 123 616 263 pixels, with an average of 7 726 016 pixels per scene.

d. Selection of predictors and derivation of coefficients for multiple regression

The ProxyVis regression equations were derived in several steps. Initial testing was performed using multiple combinations of 2–3 scenes from Table 3 with different prevailing types of clouds (low clouds, deep convection, or cirrus), and different independent variable transformations. By visually comparing the images produced using IR channels with the corresponding DNB images, a combination of the VIIRS IR channels that yielded qualitatively satisfactory results was identified. This qualitative first pass at performance is based on the observation that common error metrics, such as RMSE, MAE, or SSIM might not correctly reflect the visual quality of an image to human analysts (Dosselmann and Yang 2011). The selected channel combination is comprised of 1) channel I04 (3.74 μm), 2) channel difference [M15 (10.76 μm) − M16 (12.01 μm)], and 3) channel difference [I04 (3.74 μm) − M14 (8.55 μm)]. The best visual results were obtained by using nonlinear transforms of those three independent variables, as described in sections 2e and 2g.

After a preliminary form of the regression terms was determined, the exact form of independent variables was derived by minimizing the residuals, again using different combinations of 2–3 developmental scenes at a time. Finally, data from all 16 developmental scenes (>120 million points) were used to determine the regression coefficients. That was the maximum amount of data that could be processed using available computer memory (132 GB); however, given the number and the variety of training scenes and points and the independent verification results that had comparable R² values (section 2h), we do not think that adding more tropical scenes would improve the algorithm. Our training approach helped to ensure that the resulting images appear to the human eye like visible imagery and avoided some pitfalls of automatic variable selection routines that could lead to overfitting, biases, and poor independent performance (Sainani 2013). Further, it was found that qualitatively the resulting imagery can be further improved by using two regression equations, one for low/warm clouds and a second for high/cold clouds. A downside to this approach is that the “boundary” between the two regressions can sometimes be seen in the ProxyVis imagery. However, the overall performance of the resulting algorithm is better compared to using a single regression for all clouds.

e. Regression equations for four versions of the ProxyVis algorithm

In this section, we describe four versions of the ProxyVis algorithm that enable its creation using the global constellation of geostationary satellites as well as its use in multispectral algorithms as a replacement for visible data. Table 4 summarizes the differences, similarities, and suggested use for each version. The comparison and the reasons for using four versions are further discussed in section 2f.

Table 4.

The main differences and the suggested use for different versions of the ProxyVis algorithm. Column 2 shows the ABI channels used for each version of ProxyVis. Mapping of ABI channels to other satellites is shown in Table 2.

View Table

1) Multiple regression—Mathematical formulation

The mathematical form of the multiple regression equations is written in terms of the ABI channels C07 (3.9 μm), C11 (8.4 μm), C13 (10.3 μm), and C15 (12.3 μm) instead of VIIRS channels because the primary operational use of the algorithm is with geostationary data. In the next five subsections, we discuss the regression equations for each of the four versions and then present the postprocessing steps that are applied in the same way to all four versions.³

(i) PrVis—Two regressions

The primary operational ProxyVis formulation, the two-regression version, combines two equations derived separately for “low/warm” and “high/cold” clouds. The boundary between “low” and “high” clouds is defined using an empirically selected threshold of

${\text{BT}}_{\text{tr}}=273\mathrm{K}$(1)

in ABI channel C07 BT. The equations for the “low/warm” clouds, PV_L, and “high/cold” clouds, PV_H, are shown in (2) and (3), where C07, C11, C13, and C15 are BTs (K), normalized by the sample mean and standard deviation (provided in Table 5), and the normalized coefficients are X_L1…3, and X_H1…3 (provided in Table 6):

${\text{PV}}_L=X_{L1}{\left|\mathrm{C}13-\mathrm{C}15\right|}^{0.4}+X_{L2}\ln (\left|\mathrm{C}11-\mathrm{C}07\right|)+X_{L3}\mathrm{C}{07}^5,$(2)

${\text{PV}}_H=X_{H1}{\left|\mathrm{C}13-\mathrm{C}15\right|}^{0.4}+X_{H2}\ln (\left|\mathrm{C}11-\mathrm{C}07\right|)+X_{H3}\mathrm{C}{07}^5.$(3)

The equation for all clouds is obtained by combining (2) and (3) using BT_tr from (1):

$\text{PV}=\{\begin{array}{l}{\text{PV}}_L,\text{if}\mathrm{C}07\ge {\text{BT}}_{\text{tr}}\hspace{0.17em}\hfill \\ {\text{PV}}_H,\text{if}\mathrm{C}07<{\text{BT}}_{\text{tr}}\hfill \end{array}.$(4)

PV_L and PV_H should be unnormalized (converted back to dimensional quantities) after being estimated from (2) and (3). The corresponding values should be also unnormalized after being estimated from (5); (6) and (7); and (9).

Table 5.

Means and standard deviations for variables used in (2)–(9), where BT_tr is defined by (1), and C07, C11, C13, and C15 are BTs (K).

View Table

Table 6.

Variance explained for the developmental sample and normalized coefficients used to estimate PV_L and PV_H (two left column) defined in (2) and (3) for the full PrVis, where the threshold, BT_tr, is defined by (1), and the same for PV_SL and PV_SH (two right columns) defined in (6) and (7) and used in PrVisSimple. C07, C11, C13, and C15 are BTs (K) for the corresponding ABI channels.

View Table

(ii) PrVisA—Single regression

The regression equation for the version of ProxyVis that uses a single regression for clouds at all levels, is given by

${\text{PV}}_A=X_{A1}{\left|\mathrm{C}13-\mathrm{C}15\right|}^{0.4}+X_{A2}\ln (\left|\mathrm{C}11-\mathrm{C}07\right|)+X_{A3}\mathrm{C}{07}^5,$(5)

where X_A1…3 are the normalized coefficients (Table 6), and the means and standard deviations for all terms are listed in Table 5.

(iii) PrVisSimple—Two regressions, single channel

PrVisSimple is a single-channel version of PrVis. Similar to (2)–(4), the equations for the low/warm clouds, PV_SL, and high/cold clouds, PV_SH, are shown in (6) and (7), where C07 is BT in kelvin, normalized by the sample mean and standard deviation (Table 5), and the normalized coefficients are X_SL and X_SH (Table 6):

${\text{PV}}_{\text{SL}}=X_{\text{SL}}\mathrm{C}{07}^5,$(6)

${\text{PV}}_{\text{SH}}=X_{\text{SH}}\mathrm{C}{07}^5.$(7)

The regression equation for all clouds is obtained by combining (6) and (7) using BT_tr from (1):

${\text{PV}}_S=\{\begin{array}{l}{\text{PV}}_{\text{SL}},\text{if}\mathrm{C}07\ge {\text{BT}}_{\text{tr}}\hspace{0.17em}\hfill \\ {\text{PV}}_{\text{SH}},\text{if}\mathrm{C}07<{\text{BT}}_{\text{tr}}\hfill \end{array}.$(8)

(iv) PrVisSimpleA—Single regression, single channel

Finally, PrVisSimpleA is a single-channel version of PrVisA. The regression equation for the simple version of ProxyVis that uses a single regression for clouds at all levels, is given by

${\text{PV}}_{\text{SA}}=X_{\text{SA}}\mathrm{C}{07}^5,$(9)

where X_SA is the normalized coefficient (Table 6), and the predictor mean and standard deviation are listed in Table 5.

f. Comparison of the four versions of ProxyVis

PV_L, PV_H, and PV from (2), (3), and (5) explained 45.7%, 39.6%, and 68.7% of the variance of low, high, and all clouds from the developmental dataset, respectively (Table 6). For the single-regression PV_A for all clouds the R² = 68.2%, which is slightly lower than for two-regression version. Based on those numbers and visual analysis of different versions for multiple ProxyVis scenes (not shown), it was decided to use PrVis (11), the multichannel two-regression version, as the main algorithm that is demonstrated here and used for all ProxyVis figures unless stated otherwise. The threshold BT_tr (1) between PV_L and PV_H (2), (3) was empirically selected to optimize the performance for low clouds, at the expense of the performance for high clouds, consistent with the goal of ProxyVis to emphasize low-level clouds.

The normalized regression coefficients from Table 6 show that the VIIRS 3.74-μm channel I04 provides the largest contribution to PrVis. This enables the development of single-channel simple versions of ProxyVis (PrVisSimple and PrVisSimpleA) using VIIRS channel I04 or a similar 3.9-μm channel common on geostationary sensors. Similar to the main algorithm, the simple version that uses separate equations for low and high clouds performs slightly better. The corresponding R² values for PrVisSimpleA are 65.7% for a single regression for all clouds, and for the two-regression version PrVisSimple, 66.5% for all, 41.7% for low, and 35.2% for high clouds. These values are slightly lower than those of the main ProxyVis algorithms. A visual comparison of different versions of ProxyVis (e.g., Fig. S1 in the online supplemental material) confirmed that despite different statistics (Table 6) for many scenes all four versions of ProxyVis, including the simple single-channel versions (PrVisSimple and PrVisSimpleA) and the main versions (PrVis and PrVisA), look very similar.

Each ProxyVis algorithm version has a specific utility, and users can choose the version of ProxyVis that best matches their application (Table 4). The single regression equation versions are preferable if continuity is essential (e.g., if used as input to other applications, such as ML or other multispectral products) and might work best for observing midlevel clouds. The two-regression versions are preferable if accuracy of the low cloud detection is a priority. The simple single channel versions (PrVisSimple and PrVisSimpleA) are needed so that ProxyVis can be applied to older GOES satellites, to GOES-17 during the time when it is affected by the loop heat pipe issues (STAR 2019; Goodman et al. 2019; Van Naarden and Lindsey 2019), and for cases before the current generation of geostationary satellites. For example, PrVisSimple is being considered for operational implementation for GOES-13 and GOES-15 that are still used by JTWC. Based on NWS forecaster feedback, the simple versions also might work better for MSG SEVIRI. Further, the simple single-channel versions can be used to develop long ProxyVis datasets that combine current and previous generation GOES data and may be needed for some ML applications.

g. Evaluation of the functional form of the prediction equations

The channels used in ProxyVis are commonly used for cloud detection (Schmit et al. 2005). The shortwave IR 3.74-μm (3.9-μm) channel is more sensitive to warm temperatures than the longwave IR (owing to the Planck function), and is also less sensitive to intervening cirrus clouds, which are more forward-scattering at 3.9-μm compared to the longwave IR where cirrus clouds are more absorbing (Baran et al. 1999; Schmit et al. 2005; Velden et al. 2005). The combined effects make the 3.9-μm channel one of the most useful channels for identifying near-surface features such as low-level clouds and underlying SSTs.

The SWD, M15 (10.76 μm) − M16 (12.01 μm), provides information about the lower-tropospheric water vapor distribution (Prabhakara et al. 1974; McMillin 1975; Lindsey et al. 2014). This sensitivity allows the differentiation between dry and moist environments in cloud-free areas. The SWD is positive for clear-sky moist areas and thin cirrus, close to zero for dry areas and thick clouds, and negative for volcanic ash and most blowing dust. The absolute value of SWD used in PrVis cannot differentiate between dust and thin cirrus clouds; however, it can still differentiate between thick clouds (deep convection) and thin cirrus.

Finally, the difference [I04 (3.74 μm) − M14 (8.55 μm)] is related to the difference used in the fog product, [10.7 (11.2) − 3.9 μm; Ellrod 1995] and highlights the emissivity differences for land versus ocean surfaces as well as ice versus liquid water. The fog product is positive for liquid clouds, close to zero for ice clouds and clear areas, and negative for thin cirrus. The absolute value used in PrVis cannot differentiate between thick clouds and cirrus; however, it can still distinguish liquid water and ice clouds. Only 7.6% or less of the SWD and fog product equivalent used in PrVis (Table S1 in the online supplemental material) are negative, thus most of the information is preserved in PrVis.

To assess the independence of the predictors, Table 7 shows the correlation matrix for the terms used in PrVis (11). While all correlations are significant at the 99.9th percentile, no variable explains more than about half of the variance of another variable. The largest correlation, 0.75, is observed between the terms |C13 − C15|^0.4 and ln(|C11 − C07|) for the low clouds. The very low p values for the regression also confirm that all terms are highly significant (at 0.01% or less), with most of the contribution coming from C07⁵, followed by |C13 − C15|^0.4, and with the least contribution from the ln(|C11 − C07|) term. Figure 3 shows the BT of VIIRS channel 3.74 μm (Fig. 3a), the 5th power of that BT (Fig. 3b), and the final PrVis (Fig. 3c) versus scaled DNB for scene 1 in Table 3. The 5th power of 3.9 μm is more linearly related to DNB compared to 3.9 μm, and the relationship between PrVis and DNB is even more linear. Thus, using nonlinear transformation of independent variables allows us to linearize the relationships between our independent variables and the scaled DNB, which improves ProxyVis similarity to DNB.

Table 7.

Correlation matrix for regression terms used in Eqs. (2), (3), and (5).

View Table

View Full Size

Scatterplots of (a) VIIRS BT for channel I04, 3.74 μm, (b) VIIRS (I04)⁵, and (c) PrVis vs scaled DNB, for Tropical Storm Ida on 26 Sep 2015 (scene 1 from Table 3). The plots illustrate how taking the fifth power of BT of channel I04, and further estimating ProxyVis (PrVis) makes the relationship between DNB and the resulting function more linear. The color shading shows the density of the points. To estimate density, all points were split into 1000 bins of the 2D histogram, and the density of points in each bin was evaluated. The points were then sorted to make sure the densest points are plotted last. The color indicates the common logarithm of the number of data points used to make each point on the scatterplot.

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide

h. Independent verification

Independent verification of PrVis was conducted using VIIRS data for four scenes from four different basins (Table 8), selected using the same criteria as for the training scenes (section 2c). Table 9 shows that for all four independent scenes, the R² is higher than for the dependent sample for both low and high clouds, with the values ranging from 50.7% to 70.8% for low and from 50.3% to 66.8% for high clouds (corresponding dependent values are 45.7% and 39.6% for the low and high clouds, respectively). The comparison of VIIRS DNB with PrVis for independent scenes 2–4 is shown in Fig. 4. (The comparison for independent scene 1 is shown later in Fig. 14.) The total R² is higher than the dependent sample for the North Atlantic, western Pacific, and Indian Ocean (77.6%–85.2% versus 68.7% for the dependent sample) and is slightly lower (66.8%) for the eastern Pacific. The comparable or higher R² values for the independent data confirm the strengths of the relationships from the training scenes. Lower R² were found in the eastern Pacific and at higher latitudes in the Atlantic, which helped to identify a limitation of the current algorithm for areas with cold SSTs (see section 4).

Table 8.

As in Table 3, but for VIIRS scenes used for independent verification of ProxyVis imagery.

View Table

Table 9.

Variance explained for the independent VIIRS scenes used for ProxyVis verification.

View Table

View Full Size

Comparison of VIIRS DNB scaled using (a),(c),(e) gamma-DNB scaling with (b),(d),(f) PrVis for independent scenes 2 (Hurricane Kiko), 3 (Typhoon Krosa), and 4 (Tropical Cyclone Vayu) from Table 8. Note that the plots show all available data; however, for verification purposes, we removed land points using the same procedure that was used to remove land points from the training dataset.

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide

i. Steps to combine nighttime ProxyVis with daytime visible imagery

To best display the full disk imagery, the ProxyVis generated from 2-km resolution ABI channels is regridded to the native 0.5-km grid of ABI C02, 0.64 μm. The regridding is done using nearest-neighbor interpolation to avoid generating nonexistent clouds that could be produced when multiple points are used to generate the interpolated value. Further, the visible imagery is adjusted to provide a seamless day/night transition. Following Minnis and Harrison (1984) and Tsonis and Isaac (1985), a gamma correction is applied to visible channel C02, and to brighten visible imagery in low sun angle conditions the entire visible image is normalized using

$\text{VisDisp}={(\frac{\text{Vis}}{\cos \theta })}^{\gamma },$(12)

where θ is the solar zenith angle, γ = 0.5, and Vis is the reflectance for the ABI C02. The final operational GeoProxyVis imagery combines nighttime PrVis and daytime VisDisp into a single product. Both day and night parts are adjusted to be between 0 and 1.3 to match the range of visible C02 in the operational AWIPS2 (2019):

$\text{VisDisp}(\text{VisDisp}>1.3)=1.3,$(13)

$\text{GeoProxyVis}=\{\begin{array}{l}\hspace{0.17em}1.3\hspace{0.17em}\text{PrVis},\text{if}\theta >89°\hfill \\ \text{VisDisp},\text{if}\theta \le 89°\hfill \end{array}.$(14)

As shown in Fig. 5 for a GOES-16 scene primarily containing low-level clouds, the transition between the daytime and nighttime parts of the GeoProxyVis imagery is very smooth, which allows for the tracking of cloud features across the terminator.

View Full Size

Day/night terminator (shown by yellow arrows) as seen on GOES-16 GeoProxyVis at 2210 UTC 29 Jan 2023. The image is centered near 20°N, 65°W. Visible GOES-16 C02 (VisDisp) is on the left, and PrVis is on the right.

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide

a. Comparison with legacy products and DNB with low moonlight

Figure 6 shows a comparison of DNB (Fig. 6a), and nighttime visible-like products estimated from VIIRS IR data, including PrVis (Fig. 6b) and legacy products used for tracking low-level clouds, the fog product (Fig. 6c) and the stretched 3.74 μm (Fig. 6d). PrVis does not exactly replicate the DNB image, but it improves the contrast of the low-level clouds compared to legacy products and shows most of the low-level clouds seen in the DNB image. That detail is highlighted in the area inside the yellow rectangle, where the stretched 3.74 μm saturates, and the fog product struggles due to water vapor masking of low clouds. These differences can be seen more clearly in Fig. 7, which zooms into that area.

View Full Size

VIIRS data at 0425 UTC Sep 2015 (scene 1 in Table 3): (a) scaled DNB, (b) PrVis, (c) fog product (10.76 − 3.74 μm) with low-level clouds corresponding to more positive values shown in black, and (d) 3.74-μm stretched (VIIRS channel similar to 3.9 μm) to enhance low-level clouds, for the same scene as in Fig. 3. The yellow square highlights low-level clouds that are seen in (a) and (b), but are hard to see in (c) and (d). In (c), the BT difference from −20 to +5 K that captures the differences between high and low clouds is shown. The full range of the image in (c) is from −59.8 to 41.4 K, with 0.46% of BT differences less than −20 K and 0.06% of BT differences larger than 5 K.

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 6, but only the part highlighted by the yellow square in Fig. 6 of each panel is zoomed in on.

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide

As described earlier, one of the DNB limitations is that its quality is a strong function of the moon phase. While near the full moon (Figs. 6a,b, the moon is 96% full) DNB and PrVis look similar, near the new moon (Fig. 8, the moon is 33% full) the quality of DNB degrades significantly, while ProxyVis is not affected.

View Full Size

Comparison of (a) scaled VIIRS DNB and (b) VIIRS PrVis for Atlantic Tropical Storm Fernand (2019) at 0801 UTC 4 Sep 2019. It is hard to see the clouds in the VIIRS DNB image because the moon is only 33% full, and, in addition, there is a large number of city lights in the scene. ProxyVis works the same way independently on the moon phase and the presence of city lights and other non-cloud-related light sources. The exact appearance of the DNB image is a function of the scaling chosen for the DNB; however, the visibility of clouds is always degraded closer to the new moon.

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide

b. GOES-16 examples

The main advantage of ProxyVis is that it can be applied to geostationary satellite data from all current and many past satellites to create animations. This subsection presents examples of GeoProxyVis imagery that is used by NWS and JTWC. For these examples, we use GOES-16 data.

Figure 9 shows a GeoProxyVis HovmÖller (1949) diagram demonstrating the advantages of ProxyVis for manual feature tracking. Cloud features highlighted in Fig. 9 include a line of stratocumulus clouds in the eastern North Atlantic moving equatorward in the northeast trade winds, and the cloud lines that provide the direction of the low-level flow in the eastern Pacific. The smaller-scale features are associated with a larger-scale, westward-moving tropical wave. Note that the day/night terminator highlighted in red is not evident.

View Full Size

Longitude vs time cross section of the GOES-16 GeoProxyVis. The images are 3-hourly (top) starting at 1800 UTC 14 Oct and (bottom) ending at 1500 UTC 15 Oct 2018, centered at 13°N. The images have been remapped to a Mercator projection before being added to the individual time panels. The day/night terminator is indicated by the solid red line. Multiple synoptic features can be seen in these images, including some stratocumulus clouds moving equatorward in the northeast trade winds in the eastern part of the scene (yellow arrows), a convective system moving to the west (dashed blue phase line), and a reversal of low-level winds from westerly to southerly indicated by low-level cloud lines in the eastern Pacific (orange arrows).

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide

Another GeoProxyVis example (Fig. 10) shows subregions from the nighttime portion of the full disk image at different zoom levels. The full disk GeoProxyVis (Fig. 10a) shows features that have a similar appearance to visible imagery. These include synoptic-scale midlatitude weather systems (cutoff lows in the Southern Hemisphere, a frontal system stretching across the eastern United States), stratocumulus decks off the South American coast, and a prominent ITCZ in the eastern Pacific. The zoom on the Atlantic (Fig. 10b) again looks similar to visible and shows easterly waves (east of the Lesser Antilles), cold fronts (Texas coast), cloud lines, boundary layer rolls, and mesoscale convergence zones. Finally, in the Gulf of Mexico subsector (Fig. 10c) one can more clearly see the boundary layer rolls (central Gulf of Mexico), cloud lines (e.g., north coast of Cuba), and the ragged appearance of the cold frontal boundary. At this higher resolution, additional features can be seen such as the cold SST wake from Hurricane Michael between Cuba and Apalachicola, Florida, and the cold air behind the cold front. The cold ground signature slightly reduces the contrast with low-level cloud features.

View Full Size

GOES-16 GeoProxyVis for (a) full disk, (b) an Atlantic subsector, and (c) a Gulf of Mexico subsector at 0515 UTC 16 Oct 2018 when the entire full disk is under darkness.

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide

An accurate estimate of the TC center location is critical for determining the initial motion of the system, which serves as an input to the guidance models used for forecasting TC track, and for the accurate estimation of TC intensity via the Dvorak technique, particularly when the shear and embedded center patterns are used (Dvorak 1984). Errors in both the initial position and intensity estimates of the TC can result in poor track and intensity forecasts, especially in the short term. With ProxyVis, the LLCC can be identified almost as well as in DNB, even in the presence of cirrus clouds. Figure 11a shows an example when the LLCC cannot be easily determined from the longwave IR image for Hurricane Otis as it was being sheared from the northeast, but is clearly seen in PrVis (Fig. 11c) and DNB (Fig. 11b). Additional examples of animated GeoProxyVis loops are provided in the supplemental material and are also available in real-time on SLIDER (SLIDER 2023; Micke 2018).

View Full Size

(a) Color-enhanced VIIRS IR image (I05, 11.45 μm) and (b) gamma-DNB-scaled DNB image of the disturbance that became Hurricane Otis at 0845 UTC 12 Sep 2017. A standard IR enhancement does not show the warmer low-level clouds around the exposed LLCC depicted in the DNB image. (c) GOES-16 PrVis at 0900 UTC 12 Sep 2017. The LLCC could be seen in ProxyVis almost as well as in the DNB image. Note that the TC center position is shifted between VIIRS and GOES images due to parallax.

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide

a. Overview of the operational implementation

Due to the simplicity of the formulation, ProxyVis is easy to implement in different programming languages and platforms. That facilitates the transition to NWS operations, which often have significant software constraints. GOES-16 GeoProxyVis imagery first became available at NHC in real-time in NAWIPS in the spring of 2018, where the product was generated at CIRA and sent to NHC via a Local Data Manager feed. Starting with the 2019 hurricane season, GeoProxyVis became part of the standard NHC operational satellite data suite that is used routinely by both TAFB and HSU as one of the four main operational satellite products and replaced the legacy fog product. For the 2020 Atlantic hurricane season, GeoProxyVis was implemented in NHC’s in-house AWIPS2 (2019) via ISATSS (Guillot et al. 2017) for GOES-16/17 full disk. In 2021, 5-min CONUS and 1-min mesoscale sectors were added to ISATSS implementation. NHC now routinely archives the GeoProxyVis data as a replacement for visible imagery and uses it for poststorm analysis and writing their annual TC reports.

OPC and WPC have been using GeoProxyVis provided by CIRA in NAWIPS since 2020. The WPC International Desk has been using GeoProxyVis on SLIDER since 2020. In 2022, GeoProxyVis was implemented in the ATCF via the NRL GeoIPS (GeoIPS 2023; Surratt et al. 2016), and the animated version was demonstrated to JTWC forecasters via SLIDER and AWIPS2. NWS is working on making ProxyVis available to NWS WFOs and other National Centers, including Pacific Region, Alaska Region, and CPHC. The Australian Bureau of Meteorology has also expressed interest in using ProxyVis in its operations. The real-time 10-min (15-min for MSG) full disk animated GeoProxyVis for GOES-16/17(18) has been available on SLIDER since the end of 2017 (2022), for Himawari-8(9) since 2021 (2022), and for Meteosat-9/10/11 since 2022. The operational use of GeoProxyVis was mentioned in the IWTC-10 report (Herndon et al. 2022). Given the differences between geostationary satellites for which ProxyVis is currently implemented, we are confident that ProxyVis can be implemented for Geo-KOMPSAT-2A and MTG, as well as for next-generation geostationary satellites such as GeoXO (NOAA/NESDIS 2022).

b. Use in NHC operations

For HSU forecasters, a critical part of the forecast process is the analysis of the TC center location and its intensity and structure at 0000, 0600, 1200, and 1800 UTC. ProxyVis imagery is one of the most useful imagery products for estimating the TC center location, and offers significant improvements over longwave IR as it allows forecasters to identify low-level cloud lines. Animation of ProxyVis images provides key information for determining the location of the TC’s surface circulation, which is especially helpful for weaker TCs that do not have an eye or an otherwise easily identifiable center in longwave IR imagery. ProxyVis imagery is also very useful in the genesis stage for helping to determine if a disturbance has a closed circulation. NHC forecasters also found that ProxyVis imagery significantly increased situational awareness during the overnight hours; allows tracking the development of small clusters of convection that may be warm topped and not evident on other IR channels; helps to identify convection associated with tropical waves; is useful for determining the location of the ITCZ or weak fronts that do not have deep convection; and is helpful for postseason TC analysis.

Figure 12a shows GeoProxyVis imagery just before 1000 UTC 7 July 2018 over Tropical Depression Three (which later became Hurricane Chris). For this poorly organized system with a broad circulation, ProxyVis was very useful to the NHC forecaster in locating/tracking the center and establishing the initial motion estimate, as noted in the NHC’s TC Discussion issued at 5:00 a.m. EDT 7 July 2018 (NOAA/NHC 2018b). ProxyVis better emphasizes low clouds and looks more like visible imagery compared to the fog product (Fig. 12b) that was operational at NHC before ProxyVis became available. Another example of real-time use of ProxyVis imagery is found in the NHC’s TC Discussion for Tropical Storm Florence (Fig. 13) issued at 0500 AST 7 September 2018, where the NHC forester stated: “Nighttime Proxy-Vis and earlier microwave imagery indicate that Florence has turned westward, with an estimated initial motion of 275°/6 kt” (1 kt ≈ 0.51 m s⁻¹) (NOAA/NHC 2018a).

View Full Size

Example of (a) ProxyVis use at NHC for the Atlantic Tropical Depression TD03 (2018) at 0956 UTC 7 Jul 2018 and (b) the fog product for the same scene. The fog product was operational at NHC before ProxyVis became available. The fog product is plotted using the same BT limits as in Fig. 2 and has also been normalized to match the range of values in the visible part of the image.

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Example of ProxyVis use at NHC showing then Tropical Storm Florence (2018) following a shear-induced period of dramatic weakening at 0800 UTC 7 Sep 2018 when operational intensity estimates were 55 kt.

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide

c. Use in JTWC operations

GeoProxyVis was implemented at JTWC via both AWIPS2 and ATCF (see section 4a) for the latter half of the 2022 TC season. Initially considered an experimental tool to assist in the nighttime positioning of LLCC, ProxyVis quickly became a favored tool in the satellite analyst’s toolbox for cloud feature identification across the Pacific and Indian Oceans. The capability to view looping full disk imagery on AWIPS2, while easily zooming for feature analysis, enhanced JTWC’s ability to interrogate TCs and genesis areas for development.

Feedback from JTWC analysts and forecasters includes accolades such as “When Vis is not available, this is my Go-to.” ProxyVis was found to be a valuable tool to assist when compared to the same analysis perfumed using previously approved procedures, and the GeoProxyVis loop is now utilized day and night.

d. ProxyVis strengths and limitations

Considerable feedback was obtained from NHC, OPC, WPC, and JTWC forecasters during the GOES-R PG (Goodman et al. 2012) evaluation in 2019–22 through surveys and informal communications. Forecaster feedback on ProxyVis was generally very positive and indicated that ProxyVis is a vast improvement over the legacy products, and provides much better temporal and spatial coverage compared to OLS and DNB. According to coauthor John Beven, a senior NHC/HSU forecaster, ProxyVis imagery is “… perhaps the most useful multi-spectral product … which provides almost visible-quality imagery at night.” NHC TAFB forecasters indicated that ProxyVis is useful for multiple meteorological situations, including determining LLCC for Dvorak estimates and identifying the location of fronts and tropical waves and areas with fog and low stratus. Ideas for improvements (summarized below) were usually provided informally. Direct quotes from NHC and JTWC forecasters are included in the online supplemental material.

The original purpose of ProxyVis was to replicate visible imagery at nighttime with the main emphasis on capturing nighttime oceanic low-level clouds that cannot be seen in conventional satellite IR imagery. After ProxyVis became routinely available, forecasters started using it in a variety of conditions and synoptic situations for which the product was not originally designed. For example, forecasters found ProxyVis useful for consistently tracking low-level circulations from daylight to darkness, and for tracking high-latitude cloud features as far north as the Bering Sea. Despite the degraded quality, forecasters also found ProxyVis useful over land.

The primary limitations noticed by forecasters follow from the ProxyVis algorithm design. First, all four ProxyVis algorithms (PrVis, PrVisA, PrVisSimple, and PrVisSimpleA) are only valid during the nighttime and have not been evaluated or verified during the daytime, for θ ≤ 89°. Most of the ProxyVis signal is coming from the 3.9-μm channel (section 2g) that has both reflective and emissive components, thus ProxyVis will behave differently during daytime. GeoProxyVis slightly degrades for large satellite zenith angles; however, most of that degradation comes from the visible part (e.g., Zhuge et al. 2012) of GeoProxyVis imagery. ProxyVis was primarily developed for use by the TC forecast centers, and the training data only included over-the-ocean data and did not include any high-latitude scenes. Thus, ProxyVis performance over land and at higher latitudes is expected to be degraded. Sometimes (not shown), clouds seen over land in the longwave IR imagery cannot be seen in ProxyVis. Further, even though statistical techniques were used for development, ProxyVis imagery was designed as a qualitative product. ProxyVis was trained on dynamically scaled DNB, and the final ProxyVis formulation was selected based on subjective evaluation by the developers, similar to how other multispectral products are created [e.g., RGBs or GeoColor (Miller et al. 2020)]. Using DNB reflectance (Miller and Turner 2009) could be a cleaner way to train the algorithm. The use of two regressions in PrVis improves the display of low clouds at the expense of the degraded presentation of high- and midlevel clouds and sometimes creates a visible boundary between high and low-level clouds. Also, sometimes VisDisp saturates in the transition region between ProxyVis and visible imagery. The limitations described above will be addressed in future work.

The verification on independent VIIRS imagery (section 2h) showed that for independent scenes the R² was comparable to or greater than that for the dependent data. However, when the verification sample was stratified by latitude, R² decreased sharply in the North Atlantic, north of ∼40°N where the SST is quickly decreasing. To investigate that issue further, the Atlantic scene (Table 9) was extended to 50°N, and the R² was calculated for latitudes from 40° to 50°N. Results showed that the R² was much smaller for that region, with values of 4.6% for low clouds, 17.9% for high clouds, and 58.2% for all data. As shown in Fig. 14, in this area there are large cloud-free areas over cold SST that look black in the DNB image but show the SST in ProxyVis, which significantly reduces the correlation. Although not as dramatic as for the North Atlantic region, the R² values for the eastern Pacific scene in Table 9 are also reduced, likely due to much colder SST north of 25°N, and the large amount of marine stratocumulus clouds that were not included in the training scenes.

View Full Size

Comparison of (a) VIIRS DNB and (b) PrVis for VIIRS data near the developing Atlantic Hurricane Humberto at 0630 UTC 14 Sep 2019. Red arrows indicate the region of sharp SST gradients where the ProxyVis image significantly differs from the DNB image.

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide

The SST signal in ProxyVis is a by-product of using the IR channels and is reduced compared to legacy products. This SST signal slightly affects the low-level cloud visibility over cold SSTs (north of 40°N in the Atlantic), but does not affect it at low latitudes (in the Gulf of Mexico or in the eastern Pacific, Fig. 15a). Forecasters noted that SST features can help with increased situational awareness, and in the animated geostationary loops are easy to distinguish from clouds based on the time evolution. The SST signal, however, could be an issue for using ProxyVis as a nighttime replacement for visible channels in multichannel products such as RGBs or GeoColor (Miller et al. 2020). Due to the use of the fifth power of the 3.9-μm channel, additional ProxyVis by-products are the nighttime wildfires, where ProxyVis shows both nighttime wildfires and clouds, and heat islands (not shown). ProxyVis also captures ship tracks shown in Fig. 15b.

View Full Size

(a) SST gradients in the eastern Pacific at 0130 UTC 17 Jan 2020, as seen in GOES-16 PrVis. The SST gradients, indicated by yellow arrows, can be easily distinguished from clouds on animated ProxyVis imagery. (b) Ship tracks in the eastern Pacific at 1500 UTC 13 Dec 2019, as seen in GOES-17 PrVis. The ship tracks, indicated by yellow arrows, are highlighted in ProxyVis similar to or better than in the visible (0.64 μm) channel.

Citation: Weather and Forecasting 38, 12; 10.1175/WAF-D-23-0038.1

• Download Figure
• Download figure as PowerPoint slide