DMSP: The Stars at Night, They are so Bright …
In search of a good measure of ‘urbanity’, we have looked at two previous datasets: GPW population densities and GPW rural/urban extents. The former has issues with inconsistent data resolution and irregular census updates. The latter only exists (at this time) for a single ‘time stamp.’ Both GISS and NCDC have chosen to deal with these issues by moving towards the use of satellite measured surface ‘brightness’ (aka ‘night lights’). This data comes from the Defense Meteorological Satellite Program (DMSP) Operational Linescan System (OLS). Using multiple night time images with cloud-free regions, global cloud-free aggregate images are created which display night time brightness. DMSP/OLS data is available from 1992 to the present through a series of overlapping satellite data sets (F10-F16). There are at least two distinct DMSP/OLS datasets: one with image data is scaled in Watts/cm^2, and another with information stored on a relative scale from 0-63. The scaled data has a range of 0-63, while the Radiance Calibrated (data used by GISS) has a range from 0-185. In this post, this latter data set is referred to as DMSP/RC
In order to read either the DMSP/OLS or the DMSP/RC data, I had to expand my toolkit. Both data sets are stored in GeoTiff raster data files (*.tif). GeoTiff is an extension to the standard Tiff image format – adding ‘Geo Tags’ to the image header locating the data in geographic coordinates. Unfortunately, there do not seem to be many tools available for extracting the data out of a GeoTiff file.
The toolkit I chose was the Unidata Java-NetCDF project available here. It came with a GeoTiff class that included some unfinished data reader methods. It was fairly straight-forward to take that class, create a new one (GeoTiffDataReader), and fill out those methods to extract data from the DMSP/OLS and Radiance Calibrated file
In addition, the GDAL – Geospatial Data Abstraction Library project has tools explicitly designed to manipulate GIS data files. It was here that I finally realized that the GPW matrix data files with a 6 line header that I was writing Perl scripts to parse are actually standard ArcINFO ASCII Grid files. A simple gdal_translate command converts them into GeoTiff files that can be parsed by the GeoTiffDataReader class. The things that become obvious as we scale the learning the curve!
The Defense Meteorological Satellite Program (DMSP) is a Department of Defense (DOD) program run by the Air Force Space and Missile Systems Center (SMC). The DMSP program designs, builds, launches, and maintains several near-polar orbiting, sun synchronous satellites, monitoring the meteorological, oceanographic, and solar-terrestrial physics environments. DMSP satellites are in a near-polar, sun synchronous orbit at an altitude of approximately 830 kilometers (km) above the earth. Each satellite crosses any point on the earth up to two times a day and has an orbital period of about 101 minutes, thus providing nearly complete global coverage of clouds every six hours.
Comparing DMSP/RC imagery to DMSP/OLS imagery
GISS uses the DMSP Radiance Calibrated (RC) data for their determination of satellite brightness. This data set is archived but no longer maintained by which I mean that you can download the 1996/97 data set, but it is not updated with current imagery. The DMSP/OLS data set is available for a set of satellites with overlapping coverage for the years spanning 1992-2008.
The first chart compares my parsing of station brightness from the DMSP/RC imagery with the brightness assigned to that station by GISS. While there is an obvious correlation between the brightness assigned by GISS and that assigned by my own method, the assignment is not precise. Looking into the issue in the case of 3 stations (Montreal, Cairo, and Seoul), I found that the value assigned for GISS station brightness was one that was available in a cell that was nearby the cell that my method selected for that station, but that the cell offset difference by row and column was not consistent between stations. I’ll attempt to follow-up on this issue in a later post.
This next chart compares my parsing of station brightness from the DMSP/OLC imagery with the brightness assigned to that station by GISS. The lack of correlation is disappointing.
Satellite Brightness as a Proxy for Urbanity
Below, I compare the frequency of matches between three measures of satellite brightness with three measures of urbanity. GISS maintains a separate copy of weather station inventory data (v2.inv) whose source is the GHCN v2.temperature.inv. But the GISS version includes the additional ‘satellite brightness’ column that the GHCN master copy does not (as well as a few additional stations), and that is why I am using it here.
In each set, the first row is parsed from the GISS v2.inv column 104-106 for brightness (derived from DMSP/RC)(range: 0-186). The second row is my derivation of the of the DMSP/RC (range: 0-186). The third row is my derivation of the DMSP/OLS F12 1995 imagery (range: 0-63).
In the first set, the satellite data is compared against the GHCN Rural/Smalltown/Urban field found in GISS v2.inv. In the second set, the satellite data is compared against the GHCN A/B/C satellite brightness field found in GISS v2.inv. In the third set, the satellite data is compared against the GPW Rural/Urban Extents data as derived in an earlier post.
I am disappointed in the degree of accuracy/precision between GISS and myself in the parsing of DMSP/RC data. While there is a visible rough correlation, I expected better. I’ll be exploring Imhoff 1997 and emails to GISS to see if there is something I can do to better reproduce those results.
I am even more surprised by the lack of even rough correlation between the DMSP/OLC and GISS brightness values (DMSP/RC) for GISS Brightness levels below ~50. Above 50 though, the situation seems improved. High GISS brightness values are usually high DMSP/OLS values.
The DMPS/OLS seems to provide a clearer urban/rural signal in all three binning schemes: R/S/U, A/B/C, and GPW rural-urban extents. It also has the advantage that it is maintained as a time-series, allowing comparative studies.
Hansen, et al, 2010, Current GISS Global Surface Temperature Analysis
Hansen, et al, 2001, A closer look at United States and global surface temperature
Elvidge, et al, 2003 Preliminary Results From Nighttime Lights Change Detection
See also Nighttime Lights Change Detection (Slide Presentation)