Home > climate, Deconstructing Watts, Statistics > Eschenbach: Poisson Pill

Eschenbach: Poisson Pill

2012 July 12

Recently, Dr Masters of Weather Underground publicized an estimate of the chance of having 13 months in a row each with an average temperature in the top tercile of temperatures for that month for the continental US (CONUS). The probability for the temperatures of a random month to be in the top tercile for that month is, by definition, 33.33% (one in three). He calculated the probability of 13 in a row as the probability of 13 independent events which would be 1 in (1/3)^13 . This works out to be about 1 in 1.6 million.

The error here is that monthly averages are not independent events. An above average temperature in one month will “carry over” into the next month so that the probability of that next month being above average is greater than 50%. Likewise, when one month is in the upper tercile, the probability that the next month will be in the upper tercile is greater than 33%. Monthly temperatures are “events” with “memory”, and the probabilities that a particular month will have temps that are above average, or in the top tercile, or in the top quartile, etc … is influenced by the temperature of the preceding month.

At WUWT, Willis Eschenbach took the same data, broke it into overlapping 13 month chunks, counted the number of months in the upper tercile, plotted the results, and thought he saw a Poisson distribution in it. At which point, he works a least squares fit to determine the Poisson parameters. Ironically, in light of his internet preening about looking at the data first, he makes the exact same mistake as Dr Masters and then compounds it. Poisson distributions are for data generated by independent events. Not only is the “success or failure” rate for a particular month to be in the upper tercile in each 13 month period not independent (for the reason explained above), neither are the overlapping 13 month periods independent. For example, say his first 13 month period is for “June 2011 to June 2012” and the second 13 month period is for “May 2011 to May 2012.” These two periods share 11 data points – hardly independent.

But there is something interesting that arises out of the Poisson plot. The discrete Poisson distribution is characterized by only one parameter: lambda. This parameter lambda is the mean of the data. In this case, it is the number of months expected to be found in the upper tercile (for that month) given a set of monthly temperatures over a range of 13 months. If the average monthly temperatures were independent events, the mean would be 13 months * 33.33% = 4.333 months in the upper tercile per 13 month period. Eschenbach’s fit uses a lambda of approximately 5.2. Reversing the previous calculation, 5.2/13 = 40% that any particular month will be found in the upper tercile.

That’s odd.

Let’s go back to Master’s calculation.

He calculated the probability that a string of months 13 long will exist given that each month has a 33% of being in the upper tercile. That calcuation is …
(.333)(.333)(.333) … (.333) (for a total of 13 terms), which can be expressed as
(.333)^13

But we know that if given a month is in the upper tercile, the following month is more likely than “random” to also be in the upper tercile. So, assuming that memory for monthly temperatures is only one month long, the calcuation looks something like
(.333)(.333 + a)(.333 + a)(.333 + a) … (.333 + a) for a total of 13 terms, or
(.333)(.333 + a)^12

Did Eschenbach stumble into a method of finding ‘a’?

We can discard the Poisson distribution model and go back to the data for the answer.

Using the NOAA NCDC drd964x.tmpst dataset, extract the temperatures for the lower 48 (CONUS). This data is in monthly columns and annual rows. Sorting each column, define the upper tercile for each month. Convert the matrix to year_month list and resort the data into a time series. Calculate the probability of the following month being in the upper tercile given that the previous month was in the upper tercile. That probability is 44.31%. Checking our work, we see that 33.62% of the months are flagged as upper tercile. This should be precisely 33.33% but is not due to rounding of the columns and the presense of a few NAs. Code is here.

Note that this “measured” 44.31% probability is close to, but a bit higher, than that inferred from the Poisson distribution fit of the data.

Now we can return to the original calculation in question. How likely is it that the average temperature for every month in a string of 13 months will be in the upper tercile of averages for that month … assuming that the “memory” of this data is only one month deep? By definition, the first month in a random string has a 33.33% probability of being in the upper tercile, but, per the data, each of the following months in an uninterrupted string will have a 44.44% chance of also being in the upper tercile for that month. Thus …
(.3333)(.4444)(.4444) … (.4444) for a total of thirteen terms, or …
(.3333)(.4444)^12 = one chance in 50500.

There’s my cut at the problem. I’ll note that if we calculate the string of 13 using the 40% probability inferred from the Poisson distribution model, then we get a value (1 in 179000) notably close to Lucia’s Monte Carlo modeling (1 in 167000).

===================

Update 20120712 2026: Reading Chris’ comment at Tamino’s about excluding the data that you want to ‘test’ from the model that you build for it, I recognized the error. I’ve rerun this setup using the same NOAA NCDC CONUS dataset, but excluding 2011 and 2012 from the analysis. This gives me a probability of 43.68% that a month will be in its upper tercile given that the previous month was in the upper tercile. This slight down movement means that the chance of 13 in a row is slightly less than that calculated above, but only slightly. My estimate now stands at 1 in 62,100.

Advertisements
  1. Willis Eschenbach
    2012 July 12 at 11:41 pm

    Thanks for your interest, Ron. A couple of comments.

    I started by iteratively fitting a Poisson distribution. At that point I found that the iterative fit gave me almost the identical value for lambda as in the standard Poisson distribution, for which the unbiased estimator for lambda is the mean of the dataset. This is strong evidence that the distribution is Poisson.

    Then I showed that the Kolmogorov-Smirnov test strongly fails to reject the hypothesis that it is a Poisson distribution, but it resoundingly rejects the hypothesis that it is a Gaussian or a binomial distribution.

    Then I showed that the K-S test also strongly fails to reject the hypothesis that each of the individual month’s records are also of Poisson distribution.

    Finally, I showed that the means of the monthly datasets, which are the unbiased estimator of lambda for the Poisson distribution, are all very close to each other (average = 5.14, with a standard deviation of only four hundredths).

    So, while you can fatuously claim that it can’t be a Poisson distribution because … well, because you can’t figure out how that might be happening, the fact is that it is a Poisson distribution … so if your math predicts something significantly different, you have made a mistake somewhere.

    w.

  2. FourEcks
    2012 July 13 at 3:06 am

    The fact that the Kolomgorov-Smirnov test fails to reject that the data form a Poisson distribution does not mean that the data you have *are* Poisson-distributed, merely that they are *not inconsistent* with a Possion distribution. This is subtle but important, because the data that make up your distribution are not consistent with a Poisson process ergo your distribution is not Poisson however much you might want it to be.

  3. 2012 July 13 at 5:25 am

    Hi Willis. My math could be wrong, but my logic regarding the Poisson distribution is not.

    Lets distinguish the shape of the Poisson “curve” and the use of Poisson as a Probability Distribution Function,(PDF).

    By definition, the Poisson “curve” has non-zero values over the range of discrete integers from 0 to infinity. Your curve ends at the integer 13. Thus it is not a Poisson curve. Q.E.D.

    The Poisson distribution applies when: (1) the event is something that can be counted in whole numbers; (2) occurrences are independent, so that one occurrence neither diminishes nor increases the chance of another; (3) the average frequency of occurrence for the time period in question is known; and (4) it is possible to count how many events have occurred, such as the number of times a firefly lights up in my garden in a given 5 seconds, some evening, but meaningless to ask how many such events have not occurred.

    umass

    By defintion, the use of a Poisson “curve” as a probability distribution function requires certain conditions be met. Among those conditions are the requirement that events tested are independent (ie … not autocorrelated). The probability that a particular month is in the upper tercile is not independent of the fact that the previous month was or was not in its upper tercile. Thus it is not a Poisson probability distribution. Q.E.D.

    It may well be, Willis, that there are cases where a truncated Poisson distribution is a good approximation to the probability distribution for dependent events. But until you understand the conditions that allow a truncated Poisson to be used as an approximation, you won’t understand your model.

  4. 2012 July 13 at 6:34 am

    Willis,
    That makes no sense. BY DEFINITION, the Poisson distribution applies to discrete events that are independent AND identically distributed (iid). The means no correlation, no memory, The question being asked here is actually quite complicated. A simple fit to a standard distribution ain’t gonna cut it.

  5. Willis Eschenbach
    2012 July 13 at 9:34 am

    FourEcks :
    The fact that the Kolomgorov-Smirnov test fails to reject that the data form a Poisson distribution does not mean that the data you have *are* Poisson-distributed, merely that they are *not inconsistent* with a Possion distribution. This is subtle but important, because the data that make up your distribution are not consistent with a Poisson process ergo your distribution is not Poisson however much you might want it to be.

    Thanks, FourEcks. Yes, I do I understand that it is “not inconsistent” is different than “is”, and I have understood that for decades.

    Look, I will leave it up to you good folks to explain why a Poisson distribution is statistically indistinguishable from the results from this experiment. So far, nobody has been able to do anything but tell me it is not a Poisson, and I understand that … but since there is no practical or statistical difference between the distribution and a Poisson distribution, that is of no interest to me.

    I was looking, not to understand the theory, but to understand what the real odds are of finding 13 in a row in this dataset. My answer is certainly “close enough for government work”, as they say.

    Thanks to Ron for the UMass citation … it contains a surprising statement (emphasis mine):

    [The Poisson Expectation] may be used in reverse, to test whether a given data set was generated by a random process. If the data fit the Poisson Expectation closely, then there is no strong reason to believe that something other than random occurrence is at work. On the other hand, if the data are lumpy, we look for what might be causing the lump.

    Since in this case the data “fit the Poisson Expectation closely”, in fact quite closely, what should we conclude?

    Finally, on the other thread I have given the K-S results for the Poisson distribution of the whole, along with the K-S results for the months, and the lambdas for the months, showing that in every way that I know how to test it, the data has the form of a Poisson distribution.

    I then invited people to either show test results that either show it doesn’t have the form of a Poisson distribution, or to show evidence that it has the form of some other kind of distribution. To date, no one has replied. I make the same request here—if you think it doesn’t have the form of a Poisson distribution, then what form does it have? Please, provide tests, theory is not as important as test results.

    I have tried to fit it myself with a number of other distributions, without success … but you guys are better theoreticians than I am, I await your answers.

    Thanks to all,

    w.

  6. FourEcks
    2012 July 13 at 11:40 am

    Thanks, FourEcks. Yes, I do I understand that it is “not inconsistent” is different than “is”, and I have understood that for decades.
    Look, I will leave it up to you good folks to explain why a Poisson distribution is statistically indistinguishable from the results from this experiment.

    Just because the resulting distribution resembles a Poisson distribution does not mean that the process underlying it is.

    The problem you have is that we know the underlying process is not Poisson (the monthly temperature data are not independent), so you can throw every statistical test you like at the distribution to show that it’s consistent with a Poisson distribution but they mean nothing.

    Now if I have some a process whose events are independent, and the K-S test shows that the resulting distribution is consistent with a Poisson distribution then that’s (potentially) meaningful.

  1. 2012 August 2 at 6:26 pm
Comments are closed.