GHCN Bright: Dark Alleys, Dim Bulbs, and Bright Lights

Introduction

My original post on the ‘brightness’ fields, DMSP: The Stars at Night, They are so Bright …, looked at the DMSP satellite ‘night light’ brightness data as used in GHCN and GISS. The brightness fields were not part of the original GHCN v2 metadata. GHCN adds an A/B/C indicator of brightness. GISS includes that but also adds a numerical value. The GISS value is derived from the DMSP “Radiative Calibrated”, a single data set prepared from data collected 1996-1997

World Stable Lights

I wanted to take a second look at brightness in part because I am slightly more familiar with the GIS tools and also because I noticed that spatial-analyst.net had a ‘night lights’ data set (nlights.zip). Examining the spatial-analyst.net R scripts for data retrieval and processing leads us to a slightly different DMSP data set: “world_stable_lights”. This data set appears to be processed form of the DMSP/OLS data. The date stamps in the tar ball are from 2002, although the readme (not updated?) states that the data was collected from 1994-1995.
http://ngdc.noaa.gov/dmsp/data/web_data/stable_lights/

After downloading and decompressing, I fed the “world_stable_lights.tif” into the GeoTiffDataReader. It returns the following match rate with the GHCN brightness values. This is a ‘cell-centered’ data set and requires the use of an offset. The [a,c] column shows the parametic value used to bin the brightness value where ‘A’ is less than ‘a’ and ‘C’ > c. This values changes as the data ranges differ in differing files. They are set by a manual search for values that maximize the match count.
world_stable_lights.tif (offset required, reso=0.00833, max value=100)
'A' 'B' 'C' total match [a,c]
GHCN: 2169 826 2854 5849 80% [1,60]


I then followed the lead of spatial-analyst.net and rescaled the data into a 0.05deg grid. The rescaling has made this ‘grid-centered’ data set that requires no offset. The rescale command is a follows;
gdalwarp world_stable_lights.tif -t_srs “+proj=longlat +ellps=WGS84” world_stable_lights.tiff -ot Byte -r bilinear -te -180 -90 180 90 -tr 0.05 0.0
world_stable_lights.tiff (rescaled, reso=0.05, max value=100)
'A' 'B' 'C' total match [a,c]
GHCN: 2218 643 2514 5375 74% [1,50]


DMSP/OLS

Returning to the DMSP/OLS datasets, I selected the a data set for the F12 satellite from 1997 to review. The data range differs (0-63) from the world_stable_lights and thus so do the binning values.
http://ngdc.noaa.gov/dmsp/data/web_data/
F121997.v4b_web.stable_lights.avg_vis.tif (reso=0.008333, max value=63)
'A' 'B' 'C' total match [a,c]
GHCN: 2303 479 2877 5659 78% [12,24]


Comparing the previous file to another satellite data set (F14, 1997) from the same year.
F141997.v4b_web.stable_lights.avg_vis.tif (reso=0.008333, max value=63)
'A' 'B' 'C' total match [a,c]
GHCN: 2475 445 2648 5568 76% [12,24]


DMSP/RC

This is the data set which Hansen 2010 explicitly provides a download reference. Hansen 2010 states the resolution is 30 seconds which matches world_avg.tif. The date stamps in the rad_cal.tar show that this tarball was built in 2002.
http://ngdc.noaa.gov/dmsp/data/web_data/rad_cal/
world_avg.tif (reso=0.00833,max=255)
'A' 'B' 'C' total match [a,c]
GHCN: 2515 214 1726 4455 61% [1,10]


world_avg_low_res.tif (reso=0.01666,max=255)
'A' 'B' 'C' total match [a,c]
GHCN: 2506 237 1844 4587 63% [1,10]


Results

Using the ‘world_stable_lights.tif’ file, a GHCN v2 style inventory file is created in fixed width format, a csv file, and a stacked comparison file.

v2.bright.inv.txt
v2.bright.inv.csv
v2.bright.compare.inv.txt

Discussion

One of the problems the previous look was the confusion created by attempting to correlate the brightness data with the R/S/U mappings. Hansen 1999 warns that this did not work well for their initial look-see.

While Hansen 2010 provides a URL to a DMSP/RC data set, using this data set provides the weakest correlation with the GHCN brightness values. References in the Imhoff 1997 abstract indicate that he used a data set with 0.025 resolution. I’m not convinced I know which data set was used for the GHCN A/B/C brightness. The 80% match rate with the world_stable_lights.tif makes a compelling argument.

Rescaling the satellite data has the effect of rescaling the values as they are averaged out over a larger area. This effect is also noted in Hansen 2001.

Resources
http://www.gisdevelopment.net/aars/acrs/1999/ts7/ts7091pf.htm

Image and Data processing by NOAA’s National Geophysical Data Center.
DMSP data collected by the US Air Force Weather Agency.

Preliminary Results From Nighttime Lights Change Detection

Click to access elvidge.pdf

A technique for using composite DMSP/OLS “City Lights” satellite data to map urban area
Marc L.Imhoff, William T. Lawrence, David C. Stutzer, Christopher D. Elvidge, 1997

Temperature Trends of the U.S. Historical Climatology Network Based on Satellite-Designated Land Use/Land Cover

Click to access TempTrendUSHCNbasedonSatellite.pdf

A closer look at United States and global surface temperature change

Click to access 2001_Hansen_etal.pdf

(plate shows 0.1 deg reso, max=100)

IospDMSP
http://www.unidata.ucar.edu/software/netcdf-java/tutorial/IospDmsp.html
(0.025 deg reso, ~2.7km)