Doran/Zimmerman global warming poll a tragic example of research gone wrong
It’s a “fact” asserted by political leaders, media and activists worldwide. Important public policy and corporate decisions are based on it. Researchers and public opinion survey coordinators take it as a given. School children and college and university students are assured it is true.
It is the idea that scientists agree that we are causing climate catastrophe. It is perhaps best summed up by the following statement, one heard often over the past three years:
“97% of climate experts agree that humanity is causing dangerous global warming and other problematic climate change because of our greenhouse gas emissions.”
No poll of experts has actually shown this. There has never been a reputable worldwide survey of climate scientists that has even asked the question. In fact, it has never even been demonstrated that there is any “global scientific consensus about the climate crisis”, as Al Gore continually assures us that there is.
This is simply one more of the many urban legends that permeate the climate debate, myths that are costing societies across the world billions of dollars every year.
Two pieces of evidence are most often cited to support the 97%/consensus argument:
- A 2010 paper in the Proceedings of the National Academy of Sciences of the United States (PNAS) by Anderegg et al.
- A poll conducted in April 2008 by Professor Peter Doran and then-graduate student Margaret R. K. Zimmerman at University of Illinois at Chicago. The survey results were summarized in a paper published in January 2009 in the science journal EOS.
Contrary to popular belief, the Anderegg et al study did not poll any experts at all. Instead, the paper’s authors merely evaluated the publication record of scientists they chose to represent two sides in the global warming debate. This study has been roundly condemned as worse than useless by several authors and, because I was personally involved in assembling some of the lists of experts cited by the researchers and so understand the limitations of those lists, I will explain in a future FCPP blog posting why the Anderegg et al study is not a meaningful indicator of expert opinion about this topic.
The Doran/Zimmerman study, which did poll experts, has also been thoroughly debunked by many writers and so there is little point in repeating their criticisms in this blog posting. However, there are two problems with the study that have received little or no coverage to date. Both of these problems destroy the poll’s credibility as a reliable measure of the stance of climate scientists on the supposed climate crisis.
First, to be included in the final list of 77 climatologists (75 of whom, or 97%, answered yes to Doran and Zimmerman’s ambiguous question about the causes of climate change (see below)) scientists must have published at least 50% of their peer-reviewed publications in the last five years on the subject of climate change. This means that a climatologist who has published three papers in the past five years, two of which were on the subject of climate change, would have been included in the final 77. A scientist who published 40 papers in the past five years, 19 of which were in the field of climate change would not have made the cut. How this serious flaw in the survey methodology affected the results is impossible to determine without examining the raw polling data. More information is required from Doran and Zimmerman to determine the effect of this problem.
The second issue is far more serious and undoubtedly skewed the results in favour of the view that most experts in the field agree that human-caused climate change is a problem.
Zimmerman’s Master’s thesis, the foundation of the EOS paper, apparently did not appear on line until September 2011, two years and eight months after the EOS paper was published (how media and others could trumpet the results presented in the EOS paper without first examining the survey methodology and raw data is a topic for another blog posting). Zimmerman’s thesis can now be downloaded here for $1.98. It is well worth reading, not only to note the apparent researcher bias revealed in the introduction and the obvious flaws in much of the survey methodology, but also as an interesting summary of the problems in the research on consensus among climate experts to that point. Even more illuminating are the thesis appendices which include hundreds of comments from the scientists being polled. Therein, it is revealed that many of the very serious problems with the survey questions and methodology were pointed out repeatedly by the scientist respondents while the poll was being run.
Here is just one example of the many issues respondents had with the survey.
The two principle questions in the poll, the results of which monopolized media coverage, were:
Q1. “When compared with pre-1800s levels, do you think that mean global temperatures have generally risen, fallen, or remained relatively constant?”
Q2. “Do you think human activity is a significant contributing factor in changing mean global temperatures?” The possible answers were, “Yes”, “No”, and “I’m not sure.”
In later queries, survey respondents were given the opportunity to enter text in response to, for example, question 3c:
“What makes you unsure if human activity is a significant contributing factor in changing global mean temperatures?”
Not surprisingly, many experts took the opportunity to thoroughly dismantle the obvious problems in these questions. For example, at least 34 scientists objected to the use of the word “significant” when applied in this context. Here are some samples (bolding added to most important comments. An unedited listing of the 34 scientists’ objections to the use of this word may be seen here on the ICSC Website):
- “’Significant’ is undefined, and to achieve the statistical parameters of sigificance is much of what the debates are about…The bigger question is, ‘How much [warming] does human activity add?’”
- “I assume you mean ‘substantial’ rather than statistically ‘significant’… It is possible that we have provided 5-10% of the change, but I am not sure if that is what you would define as ‘substantial.’ “I believe human activity is a contributing factor, it’s the term ‘significant’ I’m unsure about.”
- “I do not know what you mean by significant. I believe humans are affecting the climate, I am not sure how and to what level.”
- “I don’ know how to distinguish the effect of human activity from other controls, and I don’t know how you define ‘significant’.”
- “I think human activity is a significant component, but I do not know if it is 10%, 25%, 50% or more.“
- “I have no doubt that it is a factor, and part of my answer relates to the vagueness of the word ‘significantly’. Certainly natural variability is significant. I don’t think we are yet able to ease out the fraction of warming that is anthropogenic from the fraction that is natural…”
- “It depends on your definition of ‘significant. Is human activity a factor? Yes.”
- “Personally I have no doubt that human activity is a contributing factor to increased average MGT, but I cannot evaluate unquantified, qualitative statements like ‘major,’ ‘important,’ or ‘significant’ and disapprove of their use in scientific discussions/conclusions.”
- “Significant is a loaded term. Human activity has contributed to the increase in temperature, but how much has this activity impacted the global mean temperature?”
- “Significant’ is a relative term. To me, significant means that most of the changing temperature would be attributable to human activity. I’m not sure that can be demonstrated.”
- “‘Significant’ is a word that is open to multiple interpretations. Significant is the key word. it has made a difference, but I am not sure if it is a significant difference or just adding to a natural change in temperatures.”
- “That the humans are a contributing factor is clear, as to ‘significant’ is debatable.”
- “I believe human activity is likely doing something, but I hesitate to say it is ‘significant’.”
- “The key word is significant. There have been cyclic warm and cold periods since man has been on earth. The last 10 years the mean temperature has been rather flat, and we have a downward spike this winter. I’m not sure of all the factors going on…”
- “The term significant is somewhat ambiguous particularly in comparison to climate changes throughout geologic history.“
- “The use of the word significant makes me unsure. I know that climate fluctuations are normal, and I’m not convinced that humans are making current temperature changes significantly different.”
- “The way that you phrased the question implies that human activity has to be a significant contributor. I think that the data indicates we are contributors but I’m not sure that we understand the background cycles/changes well enough to know how small or how huge our impacts are.“
- “Does ‘significant’ mean perceptible or outside the ‘normal range’ of observations?…”
- “what do you mean by significant? Statistically? A player in the total rise? sure we are! How much? I am not sure.“
- “What is meant by significant? A major contribution, yes, but what is human activity compared with increased solar activity. So far, it is lost in the statistical models.“
- “Your use of the word ‘significant’. It seems clear that human activity has caused an increase in CO2 levels. That, in theory, might have caused an increase in global temperature. However, did it? If so, was it the only cause? If it was a cause, was it a significant cause?”
- “without defining what is meant by significant, you may get a wide range of responses that agree… ”
- “Q2 then asks if I think that humans are “a significant” contributor to warming temperatures, but I can only answer yes or no. I happen to think that we are one among many contributing factors, so I answered yes, but I couldn’t explain this. … I had to stop the survey at this point because it was forcing me to answer queries about why I think they are… ”
- “I have attempted to take your survey, but am dismayed at how it is constructed…”
- “I have answered some questions from your survey and some I have not answered because they are vague…”
- “Just filled out your survey and I have a suggestion. You need a question that asks to what degree we think human activity has influenced climate…”
… and so it continues throughout Zimmerman’s thesis appendices.
Imagine if, while a bridge was being constructed, 34 civil engineering experts all warned that the bridge would collapse unless there were major changes in its design. What sane chief engineer would not go back to the drawing board and make the changes necessary to avoid catastrophe?
Yet, I see no evidence that the very serious wording problems discussed by survey respondents above were properly addressed, let alone corrected, as a result of the feedback from the scientists being polled.
Of course, many Masters and PhD theses contain errors and other problems that are never corrected. However, very few of them become the topic of a major news release from the university, a scientific paper and newspaper articles across the world, let alone have their results incorporated into the speeches of many of the world’s leaders. Clearly, with so many serious issues raised by the respondents in the population being surveyed, if this part of the poll was not to be corrected, and the survey re-administered, then it never should have been publicized at all, let alone trumpeted the way it was.
Early on the morning of November 3, 2012, I asked the thesis supervisor (here is the whole e-mail):
“Professor Doran, I would have thought that so much feedback like this during the survey process would justify rewording the question and starting the poll over. Could you give me some background on why this was not the approach that was apparently followed, please?”
I have yet to hear back from Professor Doran, but will upload his response as a follow-up posting to this blog entry if he responds.
In the meantime, it is easy to understand one of the probable reasons that the poll was not withdrawn, reconstructed and then re-administered. In response to the question, “What are you going to do with the results?”, Zimmerman wrote, “My first and foremost plan for the results is to finish my M.S. thesis.”
Yes, that is always a student’s top priority—to graduate in a reasonable time frame. And perhaps that explains why Doran did not have her stop the poll and reformulate the questions into queries that actually made sense. But, considering the damning response of the scientists being polled, the completed thesis should have stayed on a shelf in the departmental office, never to be seen again.
Instead, its findings, often exaggerated by those seeking to boost human-caused climate change concerns, became the basis of talking points in support of hugely expensive public policy decisions across the world. What a tragedy.
Tom Harris is Executive Director of the International Climate Science Coalition – http://www.climatescienceinternational.org/ and an advisor to the Frontier Centre for Public Policy in Winnipeg, Manitoba, Canada.