Home > dmsp, GHCN, GIStemp > DMSP: The Stars at Night, They are so Bright …

DMSP: The Stars at Night, They are so Bright …

2010 March 24

Introduction

In search of a good measure of ‘urbanity’, we have looked at two previous datasets: GPW population densities and GPW rural/urban extents. The former has issues with inconsistent data resolution and irregular census updates. The latter only exists (at this time) for a single ‘time stamp.’ Both GISS and NCDC have chosen to deal with these issues by moving towards the use of satellite measured surface ‘brightness’ (aka ‘night lights’). This data comes from the Defense Meteorological Satellite Program (DMSP) Operational Linescan System (OLS). Using multiple night time images with cloud-free regions, global cloud-free aggregate images are created which display night time brightness. DMSP/OLS data is available from 1992 to the present through a series of overlapping satellite data sets (F10-F16). There are at least two distinct DMSP/OLS datasets: one with image data is scaled in Watts/cm^2, and another with information stored on a relative scale from 0-63. The scaled data has a range of 0-63, while the Radiance Calibrated (data used by GISS) has a range from 0-185. In this post, this latter data set is referred to as DMSP/RC

Software Tools

In order to read either the DMSP/OLS or the DMSP/RC data, I had to expand my toolkit. Both data sets are stored in GeoTiff raster data files (*.tif). GeoTiff is an extension to the standard Tiff image format – adding ‘Geo Tags’ to the image header locating the data in geographic coordinates. Unfortunately, there do not seem to be many tools available for extracting the data out of a GeoTiff file.

The toolkit I chose was the Unidata Java-NetCDF project available here. It came with a GeoTiff class that included some unfinished data reader methods. It was fairly straight-forward to take that class, create a new one (GeoTiffDataReader), and fill out those methods to extract data from the DMSP/OLS and Radiance Calibrated file

In addition, the GDAL – Geospatial Data Abstraction Library project has tools explicitly designed to manipulate GIS data files. It was here that I finally realized that the GPW matrix data files with a 6 line header that I was writing Perl scripts to parse are actually standard ArcINFO ASCII Grid files. A simple gdal_translate command converts them into GeoTiff files that can be parsed by the GeoTiffDataReader class. The things that become obvious as we scale the learning the curve!

DMSP

The Defense Meteorological Satellite Program (DMSP) is a Department of Defense (DOD) program run by the Air Force Space and Missile Systems Center (SMC). The DMSP program designs, builds, launches, and maintains several near-polar orbiting, sun synchronous satellites, monitoring the meteorological, oceanographic, and solar-terrestrial physics environments. DMSP satellites are in a near-polar, sun synchronous orbit at an altitude of approximately 830 kilometers (km) above the earth. Each satellite crosses any point on the earth up to two times a day and has an orbital period of about 101 minutes, thus providing nearly complete global coverage of clouds every six hours.

http://www.ncdc.noaa.gov/oa/usgcos/documents/soc_long.pdf

Comparing DMSP/RC imagery to DMSP/OLS imagery

GISS uses the DMSP Radiance Calibrated (RC) data for their determination of satellite brightness. This data set is archived but no longer maintained by which I mean that you can download the 1996/97 data set, but it is not updated with current imagery. The DMSP/OLS data set is available for a set of satellites with overlapping coverage for the years spanning 1992-2008.

The first chart compares my parsing of station brightness from the DMSP/RC imagery with the brightness assigned to that station by GISS. While there is an obvious correlation between the brightness assigned by GISS and that assigned by my own method, the assignment is not precise. Looking into the issue in the case of 3 stations (Montreal, Cairo, and Seoul), I found that the value assigned for GISS station brightness was one that was available in a cell that was nearby the cell that my method selected for that station, but that the cell offset difference by row and column was not consistent between stations. I’ll attempt to follow-up on this issue in a later post.

This next chart compares my parsing of station brightness from the DMSP/OLC imagery with the brightness assigned to that station by GISS. The lack of correlation is disappointing.

Satellite Brightness as a Proxy for Urbanity

Below, I compare the frequency of matches between three measures of satellite brightness with three measures of urbanity. GISS maintains a separate copy of weather station inventory data (v2.inv) whose source is the GHCN v2.temperature.inv. But the GISS version includes the additional ‘satellite brightness’ column that the GHCN master copy does not (as well as a few additional stations), and that is why I am using it here.

In each set, the first row is parsed from the GISS v2.inv column 104-106 for brightness (derived from DMSP/RC)(range: 0-186). The second row is my derivation of the of the DMSP/RC (range: 0-186). The third row is my derivation of the DMSP/OLS F12 1995 imagery (range: 0-63).

In the first set, the satellite data is compared against the GHCN Rural/Smalltown/Urban field found in GISS v2.inv. In the second set, the satellite data is compared against the GHCN A/B/C satellite brightness field found in GISS v2.inv. In the third set, the satellite data is compared against the GPW Rural/Urban Extents data as derived in an earlier post.

GISS R/S/U

GISS A/B/C Brightness

GPW Rural/Urban Extents

Remarks

I am disappointed in the degree of accuracy/precision between GISS and myself in the parsing of DMSP/RC data. While there is a visible rough correlation, I expected better. I’ll be exploring Imhoff 1997 and emails to GISS to see if there is something I can do to better reproduce those results.

I am even more surprised by the lack of even rough correlation between the DMSP/OLC and GISS brightness values (DMSP/RC) for GISS Brightness levels below ~50. Above 50 though, the situation seems improved. High GISS brightness values are usually high DMSP/OLS values.

The DMPS/OLS seems to provide a clearer urban/rural signal in all three binning schemes: R/S/U, A/B/C, and GPW rural-urban extents. It also has the advantage that it is maintained as a time-series, allowing comparative studies.

References

Hansen, et al, 2010, Current GISS Global Surface Temperature Analysis

Hansen, et al, 2001, A closer look at United States and global surface temperature
change

Elvidge, et al, 2003 Preliminary Results From Nighttime Lights Change Detection

See also Nighttime Lights Change Detection (Slide Presentation)

Advertisements
  1. 2010 March 24 at 8:03 pm

    It’s been a while, I know. Sorry about that. Got distracted by the new GISTEMP paper and the DMSP/RC data set. Still have more questions than answers. But thought you guys might like to see some of what I’m poking at.

  2. 2010 March 25 at 9:09 am

    Fascinating; I didn’t realize that the RC set was so different from the OLS set. The latter appears to correspond better to other urbanity proxies, at least at first glance.

    As an aside, its worth noting that the GPW rural/urban extents data is effectively generated by combining GPW population densities and DMSP/OLS data.

  3. 2010 March 25 at 1:16 pm

    Also: here is DMSP/OLS by GRUMP rural/urban extents for the U.S.

  4. 2010 March 26 at 3:00 am

    Good work

    I’m a bit lost on the last sets of slides.. blue grey red = ?

  5. 2010 March 26 at 3:04 am

    MODIS..

    saw that the other day. very interesting.. this as well

    SAGE aims to improve our understanding of the global environment, and how it is being affected by human activities. To achieve this goal, we have been developing maps, computer models and datasets to describe the behavior of Earth’s terrestrial ecosystems, hydrological systems, and climate. We have developed a new generation of global ecosystem (IBIS) and terrestrial hydrology (THMB) models, as well as fully coupled climate-vegetation models (GENESIS-IBIS and CCM3-IBIS). In addition, we have created new datasets of global ecosystems, historical land use, and land cover change, datasets specific to the Amazon Basin.

  6. 2010 March 26 at 5:38 am

    Blue and red are just helping to see first bin and last (set of) bin(s) – which are the most obvious bins for inclusion in Rural/Urban or A/C.

    First R histograms I’ve been working with. Tricky bastards.
    Probably need to go to barplot if I want fine control.

    Basically, Hansen suggests that A=0-8%, B=8-88%, C=88-100% brightness. What you see above was initially intended to – but does NOT – represent that. Not entirely sure what %brightness on an open-ended scale is anyway. Top end is highest value on surface? In station data set?

  7. 2010 March 26 at 3:27 pm

    have a look at this

    http://oneillp.wordpress.com/2010/03/23/ghcn-metadata-v2-inv-gistemp-and-night-light-radiance/

    Kinda neat to see it all in one picture.

  8. harrywr2
    2010 March 26 at 5:03 pm

    At least they they went thru the trouble of analyzing 140 random places using the mark-8 eyeball data analyzer!!!

  9. 2010 March 26 at 10:21 pm

    I have just extended my post at http://oneillp.wordpress.com/2010/03/23/ghcn-metadata-v2-inv-gistemp-and-night-light-radiance/ to include some information regarding reconciliation with the GISS radiance values. See just after the illustrations in that post.

  1. 2010 March 26 at 9:15 pm
  2. 2010 May 10 at 12:37 am
Comments are closed.