Home > Deconstructing Watts, GHCN > Watts: Dogs barking, can't fly without umbrella

Watts: Dogs barking, can't fly without umbrella

2010 April 18

Anthony Watts has posted another alarming finding in GISS & METAR – dial “M” for missing minus signs: it’s worse than we thought. METAR data recorded at Weather Underground include incorrect codes that ‘drop the minus sign’ for some stations for some times and some are coding errors (13 for 30, extraneous spaces). He includes a list of 12 records that have been corrupted by apparent miscodings. Let’s take a closer look at those 12.

Watts has been looking at the METAR data recorded at Weather Underground. But the question he raises (but doesn’t attempt to answer) is whether the miscoded METAR data is ‘infecting’ the GHCN data. For GHCN data, I will examine the GHCN-Daily data available here:
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/

Watts: The city of Yakutsk, one of the coldest cities on earth, reported a high of 79˚F on November 14th with a low of -23˚F

First, let’s find the station id for Yakustsk.
The following are run in a Bash shell on Ubuntu Linux.

>> name=YAKUTSK
>> grep -i ${name} ghcnd-stations.txt
RS000024959 62.0800 129.7500 98.0 YAKUTSK,GMO GSN 24959

Next, find the monthly record of daily maximum temps. I’m looking at TMAX because the ‘dropped minus’ should be a large enough error (and are in Watt’s examples) to show as the daily max.

>> stnid=`grep -i ${name} ghcnd-stations.txt | cut -b 1-11`
>> yrm=200911
>> grep ${stnid}${yrm}TMAX ghcnd_all/${stnid}.dly
RS000024959200911TMAX -181 S -181 S -230 S -180 S-9999 -141 S-9999 -215 S -201 S -244 S -264 S -231 S -240 S-9999 -9999 -229 S -252 S -255 S -278 S -277 S-9999 -306 S -324 S -312 S -324 S-9999 -367 S -330 S -325 S -332 S-9999

That’s a long string with daily data stored in columns. To pull out just the day of data we are interested in, try this:

>> day=14
>> i=$((14 + 8 * $day))
>> j=$(($i + 5))
>> grep ${stnid}${yrm}TMAX ghcnd_all/${stnid}.dly| cut -b $i-$j
-9999

For convenience, we can write that on fewer lines
>> day=14
>> temp=`grep ${stnid}${yrm}TMAX ghcnd_all/${stnid}.dly | cut -b $((14 + 8 * $day))-$((19 + 8 * $day))`
>> echo ${temp}
-9999

So the GHCND daily max temperature for Yakutsk on Nov 14, 2009 is -9999, the null string.

To sum the info up …
>> echo GHCND:${name}:${stnid}:${yrm}:$day:${temp}
GHCND:YAKUTSK:RS000024959:200911:14:-9999

Watts: A month later, it happened again reporting a high of 93˚F on December 14th with a low of -34˚F

Using the above method …
GHCND:YAKUTSK:RS000024959:200912:14: -9999

It looks like the bad data METAR miscode could have been recorded on for Yakutsk on Nov 14 and Dec 14 , but if that was so, it was discarded.

Watts: It turns out that if you search in Weather Underground for the station ALEKSANDROVSKOE it will point you to use the data from Nizhnevartovsk. ALEKSANDROVSKOE is a GHCN/GISS station.

METAR: Nizhnevartovsk, Russia Dec 7, 2009 M30C -v- M13C ?
GHCND:ALEKSANDROVSKOE:RS000023955:200912:07:-9999

METAR: Nizhnevartovsk, Russia Dec 25, 2009 M30C -v- M13C ?
GHCND:ALEKSANDROVSKOE:RS000023955:200912:25:-9999

METAR: Nizhnevartovsk, Russia Jan 16, 2009 M22C -v- 22C?
GHCND:ALEKSANDROVSKOE:RS000023955:200901:15:-9999

METAR: Nizhnevartovsk, Russia Feb 2, 2007 M09C -v- 09C?
(Since Feb 2 is not null, but neither is it 09C, lets look at a day +/-)
GHCND:ALEKSANDROVSKOE:RS000023955:200702:01: -88
GHCND:ALEKSANDROVSKOE:RS000023955:200702:02: -26
GHCND:ALEKSANDROVSKOE:RS000023955:200702:03: -100

(edit: This might bear further investigation, but it is definately not registering +09C)

METAR: Nizhnevartovsk, Russia Feb 13, 2007 M08C -v 08C?
GHCND:ALEKSANDROVSKOE:RS000023955:200702:12: -9999

METAR: SVALBARD, Oct 2, 2008 M03C -v- 03C?
GHCND:SVALBARD:SV000001008:200810:01: -28
GHCND:SVALBARD:SV000001008:200810:02: -29
GHCND:SVALBARD:SV000001008:200810:03: -24

Watts: Lest you think this a fault of Russia exclusively, it also happens in other northern hemisphere Arctic site and also in Antarctica.

METAR: Eureka, Northwest Territory, Canada March 3 2007 M43 -v- 43?
(use the 1st EUREKA listed in ghcnd-stations.txt)
GHCND:EUREKA:CA002401200:200703:02: -347
GHCND:EUREKA:CA002401200:200703:03: -399
GHCND:EUREKA:CA002401200:200703:04: -433

METAR?: Dome C station Dec 9, 2009
No “Dome C” in ghcnd-stations.txt
Maybe by another name?

METAR?: Nico Station, University of Wisconsin Dec 9, 2009
No “Nico Station” in ghcnd-stations.txt
Maybe by another name?

METAR: Admusen Scott Station Jan 14th, 2003 M31 -v- M3
GHCND:AMUNDSEN:AYW00090001:200301:13:
GHCND:AMUNDSEN:AYW00090001:200301:14:
GHCND:AMUNDSEN:AYW00090001:200301:15:

GHCND holds no data for Amundsen-Scott for the year 2003. It ends in the year 1996.

Discussion

Anthony Watts has identified a real issue in METAR data as recorded at Weather Underground. But he has provided no evidence that this problematic data exists in the GHCN data and a cursory check of his problem station list shows no signs that this bad data exists in GHCN-Daily. Six of the daily station METAR records identified by Watts have nulls (-9999) recorded in GHCN-Daily, which could indicate that some of these mis-codes do make it to GHCN, but are then eliminated during quality control steps.

Update

My initial ‘day extraction’ script was off by one day. It has now been corrected as has the values in the post above. The correction has made the null-values line up more strongly with the bad-data-days in the WU METAR data.

Update

In this update I look for Watt’s METAR miscoding in the GHCN-Monthly data with supplementary Antarctic Weather Station data. Once again, no bleed-through of the METAR miscodes are observed.

YAKUTSK is not in the inventory for GHCN-Monthly
but JAKUTUSK is listed at 62.02 129.72
.. Nov 2009 is listed in v2.max as -25.8C
.. Dec 2009 is listed in v2.max as -33.8C

Nizhnevartovsk is not in the inventory for GHCN-Monthly
ALEKSANDROVSK is listed twice in GHCN-Monthly (v2.temperature.inv)
.. the one that most closely matches Nizhnevartovsk is 60.95°N 76.483333°E
.. Dec 2009 is listed in v2.max as -9999
.. Jan 2009 is listed in v2.max as -17.1C
.. Feb 2007 is not listed

SVALBARD is listed in the inventory for GHCN-Monthly
..the station appears in v2.mean but not v2.max

EUREKA,N.W.T. is listed in the inventory for GHCN-Monthly
.. Mar 2007 is listed in v2.max as -9999

DOME C is not listed in the inventory for GHCN-Monthly
.. but is included in http://www.antarctica.ac.uk/met/READER/aws/Dome_C_II.All.temperature.txt
.. but there is no data listed for Dec 2009

NICO STATION is not listed in the inventory for GHCN-Monthly
.. but is included in http://www.antarctica.ac.uk/met/READER/aws/Nico.All.temperature.txt
.. but there is no data listed for Dec 2009

AMUNDSEN-SCOTT is listed in GHCN-Monthly
.. Jan 2003 is listed in v2.max as -26.2C

Advertisements
  1. 2010 April 18 at 8:02 am

    Sorry I haven’t been posting in a while. Too much overtime, not enough spring-time. But I couldn’t let this one pass.

  2. Anthony Watts
    2010 April 18 at 8:57 am

    GISS uses GHCN plus other stations that are not part of GHCN. CRU also had their own list that chose selected stations plus GHCN. If the issue is not a problem, how then do you explain this statement from GISS?

    “The data shown between 4/13 and 4/15 were based on data downloaded on 4/12 and included some station reports from Finland in which the minus sign may have been dropped.”

    My point is that since GISS has shown it to be a problem in one case, there were probably others. I demonstrated that there were. GHCN might do quality control that solves the issue, OTOH GISS admits to a quality control problem involving missing minus signs. Can you say that GISS and CRU are both unaffected by the issue? We know for certain at least GISS is.

    My whole point here is that the minus signs issue is real and observable over different locations and time frames, and that since GISS admits to exactly this type of error affecting the final climate product. The fact that it does bears further investigation. – Anthony

  3. 2010 April 18 at 9:22 am

    Don’t be coy, Anthony. “It’s worse than we thought” does not carry the same tone and alarmism as “The fact that it does bears further investigation.”

    You are spreading an alarmist message without doing the work to discover the effects of the issue you have raised. It’s a real issue. No doubt on that. But is it a real problem? As far as I can tell, you did not even look at the GHCN daily data to see if you could identify the issue where it counts before you posted your thread.

  4. Joshua zuckerberg
    2010 April 18 at 10:25 am

    Oh, Anthony, since you’re here, your moderator snipped my question I asked on your blog, so I will repeat it again for you.

    I noticed that in the original version of your compendium you claimed that

    “Around 1990, NOAA began weeding out more than three-quarters of the climate measuring stations around the world. They may have been working under the auspices of the World Meteorological Organization (WMO). It can be shown that they systematically and purposefully, country by country, removed higher-latitude, higher-altitude and rural locations, all of which had a tendency to be cooler.”

    whereas in the new version this paragraph was silently “adjusted”

    “Around 1990, NOAA/NCDC’s GHCN dataset lost more than three-quarters of the climate measuring stations around the world. It can be shown that country by country, they lost stations with a bias towards higher-latitude, higher-altitude and rural locations, all of which had a tendency to be cooler.”

    Does it mean you retract your accusations against NOAA? Don’t you think you should apologize?

  5. carrot eater
    2010 April 18 at 10:57 am

    GHCN-Daily might be interesting in some respect, but information from there does not flow to GHCN-Monthly, which is what’s relevant to GISS.

    For that the monthly and GISS, you track down the CLIMATs, which are the responsibility of the individual countries’ weather bureaus. They may or may not look at METARs while coding in the CLIMATs; it’s entirely possible that errors in METARs wouldn’t impact the CLIMATs at all. Even if the people coding at the CLIMATs were looking at METARs, they’re meant to do a bit of QC before compiling the CLIMAT.

    I’m guessing (but don’t know) that the Finland sign error was somebody mixing a ‘0’ and a ‘1’ in the CLIMAT.

  6. carrot eater
    2010 April 18 at 12:54 pm

    That was a little muddled. To Watts and Broberg: if you’re worried about GISS, don’t look at METAR and GHCN-d; look at CLIMAT and GHCN-m.

    Finding errors in METAR and SYNOP might be fun and all, but they aren’t direct sources for GHCN-monthly, and hence also not GISS. It’s unclear that Watts realises this.

    That said, there are of course also errors in CLIMATs; hence the GHCN QC check. I’m a little surprised a sign error got through the QC step, but that can be double-checked; the algorithm for the QC step is published.

    As for Watts’ “bears further investigation”: it’d take you all of a day to look into those individual stations in the GHCN-m. So the order of the day is to first blog pronouncements about “worse than we thought”, and then do the investigation.. second? never? That’s a great model for spreading understanding.

  7. 2010 April 18 at 1:04 pm

    Hmm?! You are right.

    So – in for a penny, in for a pound. And I’ve been wondering about this anyhow. As I’ve been reconstructing the v2.temperature.inv, I have been considering how best to reconstruct the v2.mean data itself.

    http://gosic.org/gcos/GSN/gsndatamatrix.htm
    http://www.ncdc.noaa.gov/oa/climate/ghcn-daily/index.php
    http://www.ncdc.noaa.gov/oa/climate/ghcn-monthly/index.php
    http://gosic.org/gcos/GSN-flow.htm

    So … are the daily and monthly GHCN v2 data sets completely independent? It would seem logical (but I am assuming) that the GHCN v2 Daily is used to create the GHCN v2 Monthly at the various national agencies.

    So I have three broad questions on the table:

    What is the relationship between METAR data such as that available on Weather Underground and the GHCN Daily collected by NCDC? Is METAR used to generate CLIMAT at the national agencies in some cases? Are they parallel products, produced by the same instruments but written into two formats. Or is one report (METAR) used to generate the other report (CLIMAT) sequentially?

    What is the relationship between the ‘national agencies daily CLIMAT’ and the ‘national agencies monthly CLIMAT’? Are monthly CLIMAT reports created directly from daily CLIMAT reports?

    What are the feedbacks between GSN Lead (Asheville) and the national agencies when it comes to QC? NCDC appears to do its own QC, as does DWD and JMA, but how are these fed back to the national agencies?

    That’s what I get for tugging on a thread.

    I’ve sent email to NOAA asking for clarification.

  8. 2010 April 18 at 1:09 pm

    No, your comment wasn’t muddled. I saw the point immediately. I’m not ready to dismiss GHCN-Monthly as completely independent of GHCN-Daily, but I did assume a closer relation between the two than I can document. Thanks for the reality check.

  9. carrot eater
    2010 April 18 at 1:55 pm

    The real question is, exactly how do the individual countries source the numbers that go into the CLIMAT, which then feeds GHCN-monthly and GISS.

    This is probably done differently in different countries and different stations. You’d probably have to contact each met service separately. But one would assume that each met service has access to the original data that was used to make the METARs and SYNOPs, so a faulty SYNOP need not mean a faulty CLIMAT at the end of the month. It could, but it need not.

    In theory, the CLIMATs get more QC attention than the METAR or SYNOP. There’s a paper from NCDC (Vose 2005) where they tried to use SYNOPs to fill in some of the gaps in the GHCN. When both SYNOP and CLIMAT were available for the station, they found that they didn’t always match. So they only used SYNOPs from stations where the two did match, when both were available. These stations were thought to produce more reliable SYNOPs.

    Since GHCN-daily is not part of the direct data flow to GISS, I’d just ignore it. But I think it’s built using SYNOPs transmitted over the GTS; check me on that. Participation is patchy; I think you’ll see it has good coverage for US and a handful of other countries, but spotty elsewhere.

    This is getting into some fairly arcane details that most won’t care about, but if Watts is going to blog about them, he may as well put a bit of effort into getting it right.

    If you want to recreate v2.mean.. fun project, I suppose. You can get CLIMATs in coded format for the recent years. For pre-1990, you’ll have to dig up the sources described in Peterson (1998). I don’t know how easy those are to access.

  10. pd
    2010 April 19 at 8:28 am

    In Poland we create CLIMAT reports automaticaly, directly from observation, not from METAR reports. Even if we have an error in SYNOP or METAR report, there is no error in CLIMAT report. I think that A. Watts is wrong. There is no link between METAR and GISS/GHCN.

  11. Ben
    2010 April 19 at 9:45 am

    Nice work…

  12. 2010 April 19 at 12:22 pm

    @pd when you say “automatically” can you expand on that?

  13. pd
    2010 April 19 at 12:59 pm

    @mrsean2k:

    There is an appliaction which is geting data from MAWS weather station. SYNOP report is made automaticaly from those data, and all calculations for CLIMAT are made from those data too. Of course there is QC. There is no such situation where a man forgets minus sign.

  14. 2010 April 19 at 1:18 pm

    @pd Thank you.

    So from the point where the instrument takes the reading, to the point where the CLIMAT data is ready for use, there is no manual intervention? Except if there’s some sort of failure – and that can happen anywhere?

  15. carrot eater
    2010 April 19 at 2:35 pm

    That might be the case in Poland, but there are surely countries where somebody goes to the instrument and writes down a reading on a piece of paper.

    And there are surely countries where the calculations for the CLIMAT and the coding of the CLIMAT are done by hand.

  16. carrot eater
    2010 April 19 at 2:37 pm

    In this case, for Poland, what errors are possible in the SYNOP?

  17. 2010 April 19 at 8:44 pm

    Thanks for the comment. Interesting information.

  18. 2010 April 20 at 1:36 am

    @carrot eater, @pd

    What’s interesting to me is that there can be such a range of approaches to gathering and processing the same data.

    I’m sure that the vast majority have historically been of sufficient quality for their original intended purposes – who could have predicted we’d be where we are, arguing the points we’re arguing even 10 years ago?

    So there’s no “case to answer” from anyone contributing this data, but we should surely now be aware of these differences and uncertainties when consuming it. And be ready to explain in as transparent a manner as possible *how* they are allowed for.

  19. carrot eater
    2010 April 20 at 5:19 am

    sean2k, could you give a bit more detail on what you mean?

    When it comes to taking monthly mean temperatures and calculating global trends, there are indeed many ways to process the data along the way. But as it turns out, you get about the same result, no matter how you go about it, so long as you choose a method that makes any sense.

    But that’s a bit downstream of what we’re talking about here, which is the mechanics of collecting and transmitting the monthly means.

  20. 2010 April 20 at 7:22 am

    @ce

    Sorry, not very clear on my part.

    I’m also referring to any steps taken between the instrument recording the temperature, and that recorded value finally being “broadcast” to other consumers of that data.

    Before consumption by 3rd parties, presumably there are a range of processes depending on the practices of the originators of data (Poland’s enviably automation for instance).

    These will have their own QC applied before releasing the data to other parties.

    Post-consumption (again presumably) whoever consumes this data, whether to produce global trends of for some other purpose, there’s another layer of consumer QC as the data is imported and employed.

    So my question is really when QC is applied to these multiple sources on import, is it one-size-fits all, or bespoke depending on the QC practices of the originator?

  21. carrot eater
    2010 April 20 at 8:24 am

    Well, let’s see. There are a few points where different practice can make a difference. One is the time of observation. Climate data traditionally uses the max and min temperature for a day, and then these are averaged over the month. But some stations only record the max and min for the previous 24 hours, and this is taken down not at midnight, but at some other time. Not taking this reading at midnight has an impact.

    Also, you can manage to calculate the monthly means incorrectly.

    For QC: before the monthly means are transmitted in, it’s up to the individual countries. I think the WMO gives some guidance, but in the end the countries do whatever they do.

    After that point, NOAA does its own QC; I think it’s described in Peterson (1998) and probably on the webpage somewhere. Basically, they look for outliers compared to the past history of that station. I don’t think the tailor this QC step to account for differences in how the countries did the initial QC.

  22. 2010 April 21 at 6:36 am

    I got a response to my email to NOAA. Thanks guys! I appreciate it.

    At this time, GHCN-Daily and GHCN-Monthly are independent products at the GSN level. But there wasn’t enough information in the reply to determine if the products are independent at the national agency level.

  23. carrot eater
    2010 April 21 at 7:50 pm

    I wouldn’t expect the guys at NOAA to know the nitty-gritty details of how each individual country writes its CLIMATs.

    As for METARs, I thought those were fairly automated for the most part. I guess some places, not.

  24. steven mosher
    2010 April 26 at 12:43 am

    Hey Ron,

    Check here for dailies on Scott

    ftp://ice.ssec.wisc.edu/pub/southpole/surface_observations/2003/

    ftp://ice.ssec.wisc.edu/pub/southpole/surface_observations/2003/0103.sfc

    if you follow the links back from BAS you end up at this ftp

  25. steven mosher
    2010 April 26 at 12:53 am

    Over on Nick site I shared some NOAA documents on the construction of V2 mean.

    Hmm lets see:

    http://www1.ncdc.noaa.gov/pub/data/documentlibrary/tddoc/td9100.pdf

    Probably wont help, maybe uve already seen it.

    Plenty of issues ( none change the final answer )

  26. 2010 April 26 at 5:59 am

    Good doc, glad to have the link.

    Worth noting:

    The reasons why the number of stations in GHCN drop off in recent years are because some of GHCN’s source data sets are retroactive data compilations (e.g., World Weather Records) and other data sources were created or exchanged years ago. Only three data sources are available in near real time. The rise in maximum and minimum temperature stations and grid boxes in 1995 and 1996 is due to the World Meteorological Organization’s initiation of international exchange of monthly CLIMAT maximum and minimum temperature data over the Global Telecommunications System in November 1994

    And this is relevant to my experience noted with SVALBARD above, that the data exists in the mean but not max db:

    Because mean monthly maximum and minimum temperature have not been regularly exchanged until recently there are large gaps in GHCN’s maximum and minimum temperature coverage.

    The QC statement which first addresses dupes is interesting because I am currently working in a data set in which the dupes are not eliminated. 10000 dupes in 30000 records!

    101 different methods of monthly means!

  27. steven mosher
    2010 April 26 at 7:55 am

    Its not clear from empirical evidence whether the dropping should increase or decrease so I’m very dubious of arguments made either way. It’s unclear what the error rate is or what the QC failure rate is. Jones and Brohan ( 06) do acknowledge the existence of these types of errors, but do not estimate them. Simply, there are a number of decisions taken in the path of calculating an average that are statistical decisions. These decisions should have an error ( however small) associated with them and carried forward. For example: in using Ghcn mean one has to decide how to process “duplicate records” which really arnt duplicates. Nobody accounts for the errors associated with that decision. Given 1 to 10 records for the same site, how do you combine them? A simple average yields one answer: other approaches, equally defensible, yield answers that can differ by .5C. So there is an error associated with this choice. It doesnt change the final answer much ( in the 1/100ths) but its an unaccounted error.

  28. steven mosher
    2010 April 26 at 7:09 pm

    “The QC statement which first addresses dupes is interesting because I am currently working in a data set in which the dupes are not eliminated. 10000 dupes in 30000 records!”

    within GHCN with 7280 stations there are something like 1.9 dups per station.

    If you look at the dups you will see cases where the are number 0,1,3 for example, which looks as if GHCN has deleted records that they consider to be dups ( per their description ). So rather than preserve the entire dup record they have culled some out.. its a small number by my count.

    WRT how one processes the “dups” there are a few methods.. Just to illustrate:

    Supoose for a Jan you have 3 dups: one reads 10, a second reads 10 and the third reads 9:

    if they are Multiple instruments at the same location the mean is 9.66.
    if you judge them as 2 dups and one different instrument the average is 9.5.
    Hansen even has his own special method for averaging these. Nothing monumental but these
    estimation errors are not reflected in the final numbers.

    CA has some posts on this.. search on scribal records.

  1. 2010 April 19 at 9:44 am
Comments are closed.