Home > Uncategorized > A Citizen Science Choice: Obfuscate or Illuminate

A Citizen Science Choice: Obfuscate or Illuminate

2010 August 20

Introduction

Welcome Real Climate readers. The bare link to this website in the post Doing it yourselves probably left a little too much unanswered. So let me lay out a short introduction.

There have been many criticisms raised regarding globally averaged anomaly records such as GISTEMP and CRUTEM. These can be broadly categorized as criticisms of “methods” and “data.”

Methods

One early criticism was the lack of the availability of the actual ‘code.’ Despite the fact that the methods were generally described in publicly accessible publications, the lack of the ability to examine the actual code lent itself to an attack on the ‘transparency’ of the methods. GISS responded with the release of GISTEMP in 2007. But CRU left itself vulnerable to this criticism which eventually led to the vexatious barrage of FOI requests sent against them.

Despite the availability of the GISTEMP code (partly due to an unfamiliar FORTRAN code base and what is frankly poor code documentation), questions and criticisms continued to be raised about the methods used to create the global anomaly charts. This began to turn around in the beginning of 2010 when a variety of technical bloggers began to write their own code to create these charts. While the similarities and differences between each of these approaches and those of the “majors” (GISS/CRU/NOAA/JMA) remains of interest, these independent and individual efforts largely corroborated the results of the institutional products. There was nothing deceptive being hidden in the code. Some of these efforts are listed below. Skeptical Science recently posted a summary: Assessing global surface temperature reconstructions

Jeff Id and Roman M
Joseph at Residual Analysis
Zeke Hausfather
Nick Stokes
Chad Herman
Steven Mosher

The “Clear Climate Code” project, which includes Nick Barnes and Dave Jones, refactored the GISTEMP code into python, uncovering some minor bugs, which were then corrected in the official GISTEMP code.

Data

With the ‘methods’ question pretty much answered, there remains questions about the data. The major source of 20th century land temperatures is the Global Historical Climate Network (GHCN). This CLIMAT (pdf) based data set was compiled from multiple data sources in the early 1990s and refreshed in 1997 as “Version 2” from over 30 different sources, but was then put on ‘automatic’ with updates occurring as various National Weather Services (NWS) reported current records via a Global Telecommunication Network (GTN). This method of updating current records has led to a decrease in the number of overall stations in GHCN with current records. The overall drop in stations has led to related claims that changes in the frequency of high latitude, high altitude, rural stations, and stations based at airports have biased the data set.

While various players spar over possible impacts of the declining number and change in distribution of the current GHCN stations, an alternate approach was suggested by Dr. Schmidt in Nov 2009 to develop a SYNOP temperature data set. With the seed of an idea set, I finally found the correct data source and proceeded to download and process the Global Summary of the Day (GSOD) data.

The GSOD data appears to be a summarized version of the Integrated Surface Hourly (ISH) (which is also called the Integrated Surface Database (ISD)). It is very sparse prior to 1950 and thus is not good for century long studies. On the other hand, after 1992, GSOD contains data from well over 10000 surface stations where GHCN drops to less than 2000. Along with Nick Stokes and Zeke Hausfather, we have begun comparing and contrasting the GSOD data with GHCN.

Some Results

My work with the GSOD is still preliminary and there remain unexamined quality issues. Nevertheless, early results seem to corroborate the results obtained with the GHCN data set. It should be noted that the GSOD data that I and others have been working with is filtered to create a GHCN-style inventory and to pass through GISTEMP quality and homogeneity checks. Here are links to some of my posts with GSOD.

GISTEMP with GSOD: Round 2 2010 June 29
GSOD: Global Surface Summary of the Day 2010 May 23
GISTEMP filtered GSOD Stations: A Pretty Chart 2010 June 27

GSOD does not ‘answer’ all the questions. It offers an alternative to those concerned about declining stations numbers or a biased decline in GHCN. Other efforts are needed to address concerns such as station siting, instrumental bias, or UHI. Some of these are more amenable to a citizen science approach than others.

Soapbox

I’m going to take a turn on the soapbox.

Some seem to see science as a method to separate “Truth” from “Falsehood.” But those who believe that have only a passing acquaintance with the history of science. Science, in my view, is a progression of approximations each better than the previous (with frequent forkings and occasional dead-ends). For some, for those who seek to obfuscate, it is enough to point to issues in a particular line of scientific inquiry and declare that approach “false.” For others, for those who seek to illuminate, it is important to quantify the net effect of those issues and to, when possible, provide improved alternatives. Whenever reading a criticism of a scientific approach ask yourself these two questions:

1. Has this critic been able to quantify the effects of the issue they raise?
2. Has this critic suggested a better approach to the problem?

If the answer to these two questions is “No”, then it might be worth taking a moment to wonder why.

Advertisements
  1. 2010 August 20 at 7:46 pm

    1. Has this critic been able to quantify the effects of the issue they raise?

    yes. Many times.

    2. Has this critic suggested a better approach to the problem?

    Same answer

  2. 2010 August 20 at 7:51 pm

    It’s a good set of questions though. There are many who criticize blindly on both sides of the debate. — there are many sides. But after two years of hard study, it’s impossibly difficult to listen to those who don’t understand the nuance.

    Maybe that’s a new blog post. Anyway, I’ve got my own opinions on just about everything. One of them is the more I study, the less I know.

  3. 2010 August 20 at 7:52 pm

    Jeff, in all sincerity, I would love to see a best effort paleo proxy reconstruction by you or McI. (Maybe you can point to something I’ve missed) And McShane and Wyner get a tip-of-the-hat for tossing a new method into the ring. I may eventually take a turn at the bat, but I bet you would do better.

  4. CM
    2010 August 21 at 8:29 am

    I remember reading Gavin Schmidt’s remark and wondering if any ‘citizen scientists’ would put their code where their mouth was. My hat off to you.

  5. J
    2010 August 24 at 7:05 am

    Thanks for this post, Ron. I enjoy reading your blog and following all your activities.

    You include a link to Skeptical Science, where I’ve been reading a lot. There has recently been some discussion over there about the GHCN data set, and the question of whether / why there are adjustments made to the v2_mean stations (NOT the adjusted ones, the “raw” ones).

    See, for example, this comment, in which somebody looks at a bunch of Canadian stations in both the GHCN non-adjusted data and the data from Environment Canada … and there seems to be a diverging trend over time between the GHCN and EnvCan versions of the same stations:

    http://www.skepticalscience.com/argument.php?p=3&t=114&&a=110#20977

    Any thoughts on that? Is there an obvious reason why GHCN “raw” data would show a warming trend that Environment Canada doesn’t show, for the same set of stations? I don’t know much about how the GHCN v2_mean file is compiled … so any words of wisdom would be much appreciated.

    There’s also a nifty explanation of the importance of gridding earlier in that comment thread.

  6. 2010 August 24 at 7:27 am

    Thanks for the comment J.

    I just bumped into the Canadian discussion a few days ago when I linked into the Skeptical Science piece. I haven’t read anymore than what shows in that thread and I know *nothing* about the “Environment Canada” network. Looks interesting from a data wrangler perspective. No comment at this time, but I’ll put it in the queue. If you have any additional links, feel free to through them up …

  7. 2010 August 24 at 7:48 am
  8. J
    2010 August 24 at 7:59 am
  9. Steven Mosher
    2010 August 25 at 12:28 pm

    1. Has this critic been able to quantify the effects of the issue they raise?
    2. Has this critic suggested a better approach to the problem?
    If the answer to these two questions is “No”, then it might be worth taking a moment to wonder why.

    Well, 1 really begs the question in an interesting way. Lets take an interesting example such as the
    “effect” of choosing a baseline period and baseline criteria for CAM. When I read Jones for the first time that was one of my questions. Now, ordinarily I would have expected to see a sensitivity study of this. Every analytical choice I made in previous work that had any subjective component was subjected to sensitivity tests. It wasnt enough for me to claim that the choice had no effect, it was up to me to SHOW that the choice had no effect. So sitting with Jones, I ask my self the question ” does that matter?” same with GISS early on. Does it matter that stations are stitched together with a 20 year period overlap as opposed to a 23 year. Hansen is silent on the matter. To a large extent that was behind by motivation to get the code. To do just the sort of tests that I was required to do. So, It’s very easy to turn the question around. Did the person making the claim, explore the sensitivity of the choices they made. If not, ask yourself why? Why is always a tough question. In the cases above, it would be very hard to estimate the effect. BUT very easy for the owner of the code to demonstrate that the choice had no effect. So, why won’t they do an easy job? and why didnt they publish the supporting evidence for their choices? Who is making the substantive claim? The person who makes the decision to treat data in a certain way. Questioning an unsupported claim, hardly carries with it a burden or counter proof. That would require more of the party EVALUATING the claim, that you require of the person making the claim.

    Question 2. has the critic suggested a better approach? This presupposes that there is a objective scale of “better” if I offer a real time calculation of something and you suggest a more precise, but slower approach, which is better? better for what?

    All that said, I DO think there is something to your question. I think we can require someone who questions a claim, to state upfront what they will take as evidence to settle their doubt. The station fallout thing is instructive. The doubters never made any precise claim. Just generalized doubt. The fair thing to do is to ask them to propose a test. Even there you will get guys like Smith who agree to a test and then forget it, or wave their arms when you do it. THAT’S when I write their doubt off.

    So, I would engage the doubters. Ask them what it takes to remove the doubt? If they cant answer that, then there is no point in engaging them. If they answer and then don’t abide by the epistemic agreement, then disengage.

    At least that is the only way I found to change the minds of the pig headed and to feret out the disengenious. its a variation on Aj Ayers question to people who believe in god. For me, the belief in “natural variation” is nothing more than blind ignorance, blind faith. Hope that’s clear.

  10. Steve McIntyre
    2010 August 27 at 9:46 am

    The so-called “barrage” of FOIs arose not because of unavailability of code, but because of untrue statements by CRU about the existence of confidentiality agreements that permitted them to send data to some people while enabling them to refuse data to people perceived as critics. The FOIs did not inconvenience them nearly as much as some people claim – a CLimategate letter by Jones says that he had to write a short webpage. Otherwise they were refused with a form email.

    In terms of replicating GISTEMP, I emulated the first few steps in R in 2007 when the ugly GISS code becaome available: http://climateaudit.org/category/surface-record/giss/gistemp-replication/

    That the CRU code was pretty simple was also discussed at CA a couple of years ago noting that gridcells here and there could be replicated.

  11. Ned
    2010 August 27 at 6:16 pm

    Hi, Ron. I’m the author of the post you link to at Skeptical Science ( http://www.skepticalscience.com/Assessing-global-surface-temperature-reconstructions.html ).

    Is there any way I can reach you by email? I’d like to get in touch about something, but don’t need to clutter up your blog comment thread with off-topic discussions.

    If you can see the email address I’m posting this under, I’d appreciate it if you’d shoot me a quick note. Thanks!

  12. PolyisTCOandbanned
    2010 August 27 at 6:18 pm

    Steven:

    I asked McIntyre for logical disaggregation and quantification of his headliner posted faults, from his posts, back in 2006. He repeatedly blew it off. The guy is not even a good analyst. He would not get it done in business, military, intelligence, etc. You should be able to see through this. It is a 5 year+ tease!

    And in case you think I’m unqualified to call him on this, Burger had the same experience of Steve playing games and blowing off real investigation of phenoma. Even being evasive and non-responsive on areas where Steve was weak.

    He’s the best the skeptics have…and he hasn’t done squat since the GRL paper. This is so rinky dink.

  13. cce
    2010 August 27 at 9:49 pm

    I think the reason that data was shared with some and witheld from others is pretty obvious. Some intended to use it in good faith, while others intended to ridicule coding practices and specific adjustments on an internet forum.

  14. PolyisTCOandbanned
    2010 August 28 at 11:12 am

    Studies have been done in many other fields (medicine, economics, psychology) that lack the pulic skeptic component and there were common issues getting code/data shared and the fundamental reason (a logical one) was that authors did not like the scrutiny. Zorita and Von Storch had issues getting the Mann algorithm in the beginning as well. And they are not blog skeptic hoi polloi.

  15. PolyisTCOandbanned
    2010 August 28 at 11:13 am

    I beleive in unilateral honesty. Unilateral sharing of code and data. Wehther with friends or foes. Qualified or unqualified. Just let the chips fall.

  1. 2010 September 3 at 5:13 am
  2. 2010 November 26 at 2:19 am
Comments are closed.