Verification of Daily Thunderstorm Occurrence Using the
National Lightning Detection Network (NLDN)

Since 1983, a daily thunderstorm probability forecast contest has been organized by the Department of Atmospheric and Environmental Sciences at the University at Albany during the summer months of June, July, and August (Bosart and Landin 1994).  The contest involves predicting the probability to the nearest 10% that a thunderstorm (TS) will be reported during a 24 hour period (starting @ 18 UTC) at ten locations across the continental United States (see table below).  The forecasts are verified by standard METAR reports and are scored against consensus and a 16 year thunderstorm climatology (1988–2003) shown in the table below as the percentage of days per month with a METAR TS report.

In recent years however, there have been several instances, particularly during the overnight hours, in which a thunderstorm failed to be reported in the METAR reports despite its occurrence (see 2009 WAF Conference presentation for examples).  During such instances, the forecast contest was verified by contacting the attendant NWS office directly and/or examining WSR-88D and National Lightning Detection Network (NLDN) data.  Given its continuous space and time coverage, an average detection efficiency of ~95% and mean location errors of < 500 m (Cummins and Murphy 2009), using the NLDN to verify thunderstorm occurrence is explored at the original 10 stations of the Albany forecast contest, plus 10 additional cities to cover the full spectrum of summertime, synoptic environments and rates of thunderstorm occurrence across the continental United States.

Locations (by METAR station ID) of the 20 cities examined, overlaid on the 15 year (1995-2009) June, July, August NLDN flash density (in flashes per km2 per year).

The tables and bar charts below show the 15 year (1995-2009) June, July and August (JJA) METAR thunderstorm climatologies (ASOS column) and the climatology of NLDN detected lightning within 5, 10, 15 and 20 km of each station as the fraction of days per month with a TS report or NLDN detected flash.   The red numbers highlight the radial bin with the NLDN climatological value closest to the ASOS value (i.e., for June ALB, .16 at 10 km is the closest to the .15 ASOS value).  Numbers in blue highlight instances where the 5 km NLDN climatological value exceeds the ASOS value.

    As seen in the tables and figures above, 10 km is the radial
    range that best agrees with the METAR TS climatology, in
    agreement with Bosart and Landin's (1994) statement that
    thunder can generally be heard within 10 km of a station.
    There are, however, some distinct exceptions: 1) MIA and
    TPA both have months (July and August) where the number
    of days with flashes within 5 km of the station exceeds the
    number of TS days, and 2) TS reports for stations in the
    western U.S. (DEN, BIS, SLC) tend to agree best with the
    number of NLDN lightning days further from the stations,
    e.g., at the 15 km radius.

The tables and graphs above explore the number of METAR TS and NLDN detected lightning days without taking into account whether these days actually match in the two data sets.   The tables below take date matching into account and address two fundamental questions: 1) If the NLDN detects lightning within X km (where X = 5, 10, 15 or 20) of a station, what percentage of the time does that station report a TS on that day
?, and 2) If a TS is reported by a station, what percentage of the time does the NLDN detect lightning within X km of the station on that date?  Average values are listed at the bottom of the tables and red (blue) numbers denote values one standard deviation above (below) the mean.


The table on the left above shows that GSO, CHS, MSP, OKC, and especially OMA and DEN are excellent verifiers of thunderstorms, e.g., when there is lightning within 10 km of the station, a TS is reported in the METARs.  On the other hand, MIA, TPA, TUS and GTF are very poor verifiers of thunderstorms, i.e. there are many days with lightning within 10 km of the station when no TS is reported.  While one might suspect that the low verification rates at TPA and MIA are due to the fact that lightning is observed nearly every summer day in Florida, the flash density plot above shows that stations such as IAH and CHS have almost as lightning per year as MIA and TPA and verify at rates above the mean.  (Possible explanations for the low verification rate in TUS and GTF will be explored below.)

The table to the right above verifies the strong east/west dichotomy noted above: stations in the western U.S. (DEN, SLC, BIS, GTF) verify TSs with lightning at much farther distances from the stations than locations in the eastern U.S..  Fewer trees and buildings for longer sight lines and less sound absorption are the most likely contributing factors to this difference.

The next factor explored is the interannual variability of the total number of JJA METAR reported TS days (hereon denoted METAR) and total number of JJA days with lightning detected with 10 km of each station
(NLDN10), and their correlation. 

The table to the right shows the Pearson (r) and Spearman rank (rho) correlation coefficients between METAR and NLDN10; the number of years, out of 15, NLDN10 > METAR (mean of 7.3); and the average absolute value of NLDN10-METAR, i.e., the number of days difference between the two measures of thunderstorm activity (mean = 4.9 days, median = 3.3 days). 

In the correlation columns, the light blue, medium blue, dark blue and purple shadings represent cities failing to have a significant positive correlation between METAR and NLDN10 at the 99.5, 99, 97.5 and 95% levels, respectively.  In the last two columns, red (blue) numbers represent values one standard deviation above (below) the mean values indicated in the paragraph above.

A number of interesting results are revealed in the table.  First, MIA, TUS, BNA, MGM, GTF and ALB all fail to have significant positive correlations between the number of JJA METAR reported TS days and the number of days with lightning within 10 km of the stations.  On the other end of the spectrum, OKC, DEN, BWI, and IAH have very high correlations between the two data sets.  Second, the fourth column is another confirmation of western stations having METAR TS days with lightning at much greater distances than eastern stations.  Third, the last column shows that the largest average differences between METAR and NLDN10 are recorded at MIA and TPA, where lightning routinely is detected within 10 km of the stations and no TS is reported, and DEN and TUS, where TSs are verified with lightning at distances > 10 km and monsoonal variability dominates, respectively.  Stations with low differences are DTW, MSP and OKC, indicating 10 km is the ideal radial distance for TS verification at those sites.

The plots below examine the points above in greater detail by showing the interannual variability in METAR (solid lines) and NLDN10 (dashed lines) for six selected cities (plots for the other 14 cities can be found here).  


The plot to the left above shows the interannual variability of METAR and NLDN10 for the cities with the two lowest correlation coefficients between the data sets, MIA and TUS.  The poor correlation in TUS seems to be due to changes in METAR lagging those in NLDN10 by a year.  The strong interannual variability of precipitation in Arizona associated with the North American Monsoon System (NAMS) seems to be the major contributing factor in this difference and is explored further below. 

    The low correlation in MIA is due to METAR remaining flat
    with time, while NLDN10 (which is always larger than
    METAR) exhibits significant year to year variability.  In
    contrast, TPA (shown to the right above), which also always
    has NLDN10 > METAR, does exhibit variability in both
    METAR and NLDN10 and has a significant positive
    correlation between the two.
BWI (above right), DEN and OKC (both left) exhibit three
    of the largest correlation coefficients between METAR and
    NLDN10 despite large differences in number of TS days at
    each station and between METAR and NLDN10 at DEN
    where METAR is always greater than NLDN10.

The next subject explored is the relationship between summertime thunderstorm activity over the U.S. and ENSO.  The table below shows the Spearman rank correlation coefficients between the two measures of TS activity (METAR and NLDN10) and the preceding winter (DJF) and spring (MAM), and concurrent (JJA) Oceanic Niño Index (ONI).  The colored numbers indicate statistical significance at the values listed below the table. 

The city with the strongest and most consistent relationship with ENSO is TUS, exhibiting negative correlations (i.e., warm Pacific waters, less TS activity) across the seasons and measures of TS activity, with 4/6 values being statistically significant. 
DTW, BNA and MSP show positive correlations between winter and spring ENSO values and subsequent JJA TS activity, while GSO, MGM, MAF and DEN all exhibit statistically significant positive correlations between TS activity and ENSO in JJA.  These later relationships, and the one with TUS noted above, are broadly consistent with the known patterns of JJA ENSO precipitation anomalies shown below with less rainfall/TS activity in the midwest, upper Plains and southwest, and more rainfall/TS activity in the southeast and Great Plains (the top panel is from this study with pink/red = positive correlations and blue/navy = negative correlations; and the bottom panel is from CPC).


Finally, relationships involving the amount of TS activity between the 20 cities are examined.  The two tables below show the Spearman rank coefficients for the correlations of NLDN10 (top) and METAR (bottom) between each city with the significant values colored as indicated within the tables. 

The cities with the largest numbers of significant correlations are OKC and IAH, which reveal more TS activity in these cities is positively correlated with activity in cities on the east coast and in the southeast and negatively correlated with TS activity in the upper midwest, northern Plains and western U.S. (MGM and MAF have fewer significant correlations but also reveal the same pattern).  Conversely, SPI TS activity is positively correlated with activity in the upper midwest and northeast, and strongly negatively correlated with activity in the southeastern U.S. and southern Plains.