DATSAV2: Foreign Weather Data Servicing at NCDC
Foreign Weather Data Servicing at NCDC
Plantico, M., and J.N. Lott, 1995: Foreign weather data servicing at NCDC. ASHRAE Transactions, 101, pt 1, 7 pp
DATSAV is a major source for the ISH data.
Part of the mission of the National Oceanic and Atmospheric Administration (NOAA) is to develop and maintain environmental data bases and to disseminate this information to users. The NOAA’s National Climatic Data Center (NCDC) is responsible for the archiving and dissemination of climatological information on both national and global scales. The NCDC works with the World Meteorological Organization(WMO), the International Council of Scientific Unions, and other world data centers to obtain and exchange information and data. The primary digital data source for foreign surface weather data is the U.S. Air Force’s DATSAV2 surface data base. Several publications and CD-ROM products are available that contain global data. This paper describes these sources and products and provides information on how to obtain them.
INTERNATIONAL DIGITAL DATA SOURCES
The NCDC receives foreign data through a variety of sources. As the World Data Center-A for meteorology, the NCDC exchanges foreign data with the other world data centers (Russia and China) and with individual foreign countries. The WMO has also established mechanisms for the free and open exchange of data among participating countries. The NCDC has sponsored visiting scientists from throughout the world and has worked with them on data exchange. The NCDC also actively pursues cooperative agree with individual countries for data exchange to supplement those sources already mentioned.
The WMO, under its World Weather Watch program established a communications network called the Global Telecommunications System (GTS) to transmit observations around the world. This network supplies data to the NCDC through the National Weather Service’s National Meteorological Center (NMC), the Air Force’s Global Weather Central (AFGWC), and the Navy’s Fleet Numerical Meteorology and Oceanography Center. Antarctic data from U.S. reporting sites are also received via the NASA SPAN network.
DATSAV2 Surface Data Base
The AFGWC surface data are archived and quality-controlled by the USAFETAC/OL-A as the DATSAV2 surface data base. Due to its superior quality and quantity compared to other worldwide surface data sources, the NCDC uses DATSAV2 as its primary source for foreign digital surface weather data. This data base contains hourly and/or synoptic data for about 20,000 worldwide stations, with nearly 10,000 of these stations currently active. The DATSAV2 data base includes data from such codes as synoptic, airways, METAR (Meteorological Aviation Routine Weather Report), AERO (Aviation Routine Weather Report), SMARS (Supplementary Marine Reporting Station), and drifting buoys, as well as observations from automatic weather stations.
Data are collected from the Automated Weather Network (AWN) and the Global Telecommunications System (GTS). Transmitted observations are decoded at the AFGWC at Offutt Air Force Base (AFB), Nebraska, then sent to the USAFETAC’s OL-A in Asheville for additional decoding, quality control, and archival. More than 40 million observations are added to the data base each year. The reported elements vary considerably by station, although most report the basic elements such as temperature, dew point, wind speed/direction, and pressure. Some stations also report precipitation, snow depth, sea surface temperature, wave height, and a variety of other elements. The most current data are normally available from this data base three weeks after the data month
The period of record varies by station. Many stations have data from 1973 to the present, while some stations go as far back as the 1930s. The DATSAV2 surface data base became available to the public in 1992. From January 1, 1973, until the inception of DATSAV2, observations were stored in a different format called “DATSAV.” All observations that were in the original DATSAV format have been converted to the DATSAV2 format. Figure 3 shows the breakdown of the number of surface observations in the DATSAV2 data base for selected periods. Data for the period 1930 to 1972 were keyed from manuscript forms. Data since 1972 were obtained from the telecommunications network.
The DATSAV2 data base offers several major improvements over the original DATSAV. First, the DATSAV2 format is an ANSI-standard ASCII format that conforms to federal information processing standards (FIPS). Second, it can accommodate meteorological data fields that could not be handled in the older format. Third, the DATSAV2 data has been processed through much more extensive quality control.
Quality Control (QC) of DATSAV2 Surface Data Base
The DATSAV2 surface data base, although of much better quality than previous data bases, still contains some erroneous data. Errors were introduced into the original transmission mainly through observer errors, communication problems “garbling” the data, and countries/stations not following WMO standard encoding practices. The latter problem is the most difficult to identify and correct. Many countries follow their own set of reporting practices, and some individual stations will even have local variations. Although WMO manuals are helpful, they often do not accurately reflect the current reporting practices of certain countries (e.g., a change in encoding methods or the addition of a synoptic group not previously used by the country). Careful monitoring of incoming data is required to ensure proper decoding and validation of the data.
In the late 1970s, the USAFETAC/OL-A coined the term version to assign a number to each succeeding level of data base quality control (OC). The Version 1 through 6 programs were developed from 1977 through 1983. These programs were designed to correct specifically known and identified errors. Then, in 1984, the USAFETAC/OL-A undertook a project to create a new, improved quality data base through the use of a more systematic approach to OC. This effort cumulated with the completion of Versions 7 and 8 in 1992. The current OC is an entirely automated “expert sytem” consisting of more than 400 algorithms.
Specifically, Version 7 focuses on random errors, while Version 8 focuses on systematic errors (e.g., incorrectly decoded synoptic data). Version 7 uses a series of extreme value checks, internal consistency (within observation) checks, and external continuity (between observations) checks to QC the data. All historical DATSAV2 data were processed through Version 7. Version 8 was applied to all data going back to 1982. All current, incoming data are processed through both programs. An average of more than 40 million corrections and/or enhancements have been applied to each year’s data. See Figure 4 for an illustration of version enhancements since 1973.
In addition to the QC described above, most Air Weather Service (Air Force) stations are made “serially complete” to ensure that all hourly observations are present in the data base. This is accomplished by key-entering the missing observations. Also, these same stations undergo manual QC in addition to the automated procedures described.
The DATSAV2 surface data base is currently the best source of digital global surface observational data, both in of quality and quantity. Also, with higher levels of QC planned for the future, its quality should continue to improve.