July 08, 2005
Hans von Storch on Barton
Posted to Author: Others | Climate Change
Guest posting by Hans von Storch
My reaction to Rep. Barton's requests is split. In his five letters, he is asking for information from two different groups, namely institutions with reviewing responsibilities (IPCC, NSF) and individuals with scientific responsibilities (M, B and H). I find his inquiry of the performance of the institutions IPCC and NSF valid, but the interrogative questioning of the individual scientists is inadequate.
a) Scientists. The scientists have the task to be innovative, creative, to try new avenues of analysis and the like. They have the right to err, the right to suggest explanations and interpretations which may need to be revised at a later time. They should document what they have done, so that others can replicate.
However, this documentation often can not take the form of keeping runnable old codes of the applied algorithms, simply because the software is no longer consistent with quickly replaced hardware. For instance, most of the state-of-the-art coupled AOGCMs used in the mid 1990s are simply no longer available and running at, for instance, the German Climate Computer Center. After replacing a high performance computer with a new system, the standard model codes, including community models, need to be adapted to the requirements and possibilities of the new system, and the old code will often no longer run. This has nothing to do with the norms of the community but simply with technological progress. Also specific commercial libraries of specialized algorithms may no longer be accessible. Data and codes written on old magnetic tapes or even floppies are usually no longer readable.
Therefore the documentation must take the form of a mathematical description of the algorithms used. This is in many if not most cases sufficient for replication. Also, the intention of replicability is not to exactly redo somebody's simulation and analysis, but to find the same result with a similar code and different but statistical equivalent samples. The problem is usually not that the codes contain errors (even if many of the more complex ones likely contain minor, mostly insignificant errors), but that specific elements of implementation and specific aspects of the considered sample of evidence will lead to conclusions, which do not hold if another sample is considered or a different but equally good algorithm is employed. The reason is that we want to learn about the dynamics of the real world, and these insights should not depend on random choices in sampling and implementation. We generally do not expect scientists to manufacture results, or that unintended but significant errors will affect the eventually published conclusions.
Having this situation in mind, I consider Rep. Barton's requests to the three scientists as inadequate and out-of-scale. However, the language used by Rep. Barton makes me perceiving this request as aggressive and on the verge of threatening.
The situation is different with the second groups of recipients, the:
b) "Reviewers". Reviewers have a different role, namely they shall make sure that the standards of scientific reporting are held up. They have to ensure that the proposed explanations are considered by independent experts as to whether the presented analysis seems valid and in principle reproducible. "Independent" means that the reviewers have no vested interests for or against the case presented. In the conventional set-up these interests usually refer to academic schools of thought, but in the unfortunate, post-normal case of climate science independence from the political utility of the case should be established.
In this case, I find the inquiry of Rep. Barton to be valid. The IPCC has failed to ensure that the assessment reports, which shall review the existing published knowledge and knowledge claims, should have been prepared by scientists not significantly involved in the research themselves. Instead, the IPCC has chosen to invite scientists, who dominate the debate about the considered issues, to participate in the assessment. This was already in the Second Assessment Report a contested problem, and the IPCC would have done better in inviting other, considerably more independent scientists for this task. Instead, the IPCC has asked scientists like Professor Mann to review his own work. This does not represent an "independent" review.
The NSF seems to have failed to ensure that sufficient information is provided about work done under its auspices.
Rep. Barton should also have asked the editors of "Nature", why the original manuscript was accepted for publication even though the key aspect of replicability was obviously not met by the MBH manuscript. Actually, MBH could not meet this condition because of the strict length limitation of that journal (nowadays one would ask for extensive Supplementary Online Material). One should ask why the manuscript was accepted nevertheless - and not, as in many other cases, the manuscript was recommended to be published in a "normal" journal without the severe length limitations. I believe the reasons for Nature were the journalistic reasons - namely the expected broad interest in the subject. One should also ask why after the critique von McIntyre and McKitrik only MBH got the opportunity for a correction of his paper, whereas the short manuscript of their opponents was rejected.
To conclude - the requests to M, B and H are not fair but may unfortunately lead to a repressive atmosphere within climate science; the requests to NSF and the IPCC, however, are appropriate, as these institutions may have failed in a primary task, namely to guarantee an open scientific discourse. And, Rep. Barton should have included the editors of Nature in his analysis.Posted on July 8, 2005 12:02 AM
We are debating a paper that appeared in 1998, that's seven years ago. From the software found by McIntyre it appeared that Dr. Mann used Excel, Matlab and fortran, all three are still current. The demand is not for the libraries but for the code written by Mann himself. I have been able to succesfully recover thirty year old 9track tapes, stored in a tropical envronment. I also have been able to transcribe my own tape archive from 1987, created for my peer reviewed publication. If Dr. Mann needs any help, I'll be glad to assist. I have a proven track record of data recovery.
Posted by: Hans Erren at July 8, 2005 06:47 AM
Hans- Thanks for this post. I generally agree, though I would note that Rep. Barton has no oversight repsonsibilities of the IPCC (in fact, I'm not sure who, if anyone, does). If he wants to get a response from the IPCC he should focus his attention on scientists/administrators in NOAA, which houses the CCSP and, currently, the TSU and leadership of WG1. In 2001, President Bush asked the NRC to covene a committee to pass judgment on the IPCC finding. Rep. Barton could similalry ask the NRC to conduct a study of either the substance of the HS and/or the institutional processes involved. There are any number of avenues open to Rep. Barton. I agree that focusing on individual scientists is a bad idea.
Posted by: Roger Pielke, Jr. at July 8, 2005 06:56 AM
Echoing Hans Erren, the request is for source code - not compiled binary - mostly Fortran. The source code can be easily recompiled for any of the newer machines. The source code can be readily inspected and the methodology understood - very difficult with binary. Additionally, if the recompiled binary - most likely with a new compiler - does not produce the same results, that is a good indicator that something is seriously wrong.
Posted by: Frank H. Scammell at July 8, 2005 08:43 AM
The benefit of source code - regardless of whether it is running or not - is that it is a mathematically precise language that makes each step of a process clear. A description of an algorithm in English is invariably ambiguous. Furthermore, the oft-complained of space limitations in journals mean that a comprehensive account of the process involved can not be given in the article. Finally, steps that a researcher may deem unimportant could turn out to be critical to getting the result. Thus, even if a researcher was given unlimited space to describe their algorithm in English - it would still be inferior to source code describing exactly what was done.
For example, when inverting ill-conditioned matrices the exact computational approach used has a disproportionate effect on the result (e.g., how much precision is used to store the numbers). However, absent access to source code, an algorithm description would almost never make it clear that this was occurring (either that a step in the process involved an ill-conditioned matrix or that the inversion was affecting the results).
Posted by: John Simon at July 8, 2005 02:34 PM
"However, this documentation often can not take the form of keeping runnable old codes of the applied algorithms, simply because the software is no longer consistent with quickly replaced hardware." So we have to ask : Is this Mann's problem ? Has Mann told Von Storch this ? Or is this only a general possibility, not specific to Mann ?
-- This was a general statement, had nothing to do specifically with Prof. Mann. I was just wondering if I could comply if somebody would ask me for the code of a climate model we used in the mid 1990s. Aspects of vectorization, optimisation, parallelization of the code and the like change damatically over the years. It's not a matter of Excel and Fortran 77.
-- Description of algorithms - should be done in math, not in English, Chinese or Lower German.
Posted by: Hans von Storch at July 8, 2005 09:44 PM
On the Call for Independent Review
Quoting above: "The IPCC has failed to ensure that the assessment reports, which shall review the existing published knowledge and knowledge claims, should have been prepared by scientists not significantly involved in the research themselves."
So who shall we get to review? John Bolton, Karl Rove, Dick Cheney, their assistants? That is likely who we would end up with if we take this argument far enough. Expert review can be flawed and maybe the IPCC failed as claimed above. However I am certatin that a general policy to exclude those who have contributed to science from its summarization will, in the long run, throw the baby out with the bath water.
Posted by: Nicholas Flores at July 8, 2005 09:51 PM
One aspect of the hockey stick controversy that has been largely ignored so far is the disquiting allegation by Professor David Deming (University of Oklahoma) of a deliberate strategy "to get rid of the Medieval Warm Period."
In his paper (Journal of Scientific Exploration, v.19, no.2) he claims that, in 1995, "a major person working in the area of climate change and global warming sent me an astonishing email that said "We have to get rid of the Medieval Warm Period."
I think the time has come to reveal whether this serious allegation is credible, and if so, who this 'major' climate researcher was and who this call for action was addressed to.
Posted by: Benny Peiser at July 9, 2005 03:53 AM
The problem with Mann is that he not only acted as a scientist but also influenced public opinion in a great manner and insofar acted politically.
the following activities of Mann, acc. to his web-Side : Mann M.E., Personal Website - http://www.people.virginia.edu/~mem6u
Posted by: Bernd Stroeher at July 9, 2005 05:13 AM
The IPCC is an arm of the World Meteorological Organization and the UN Environmental Program. The lead agency would therefore be the State Department, probably through the Undersecretary for Global Affairs. NOAA/NWS also must play a strong role.
Posted by: Eli Rabett at July 9, 2005 12:26 PM
Interestingly, v. Storch et al in "Reconstructing Past Climate from Noisy Data" Science claim to have replicated the algorithm of MBH98.
There is also a further description of the algorithm at tp://holocene.evsc.virginia.edu/pub/MANNETAL98/METHODS/AlgorithmDescription.txt
Posted by: Eli Rabett at July 9, 2005 12:30 PM
I note that the description of the algorithm at the link you provide is in English rather than maths (or German, or Chinese or Lower German). The existence of the additional readme file at that same location indicates that the algorithm description is not of itself perfectly clear. Perhaps Dr von Storch would care to comment on how useful he found that description in replicating the algorithm of Mann et al.
Posted by: John S at July 9, 2005 03:54 PM
John, I take it you object to "multiply the number of apples by the number of pears". Come now, it is perfectly normal to describe an algorithm in natural language.
Posted by: Eli Rabett at July 9, 2005 07:50 PM
By way of corollary you might like to consider why there are so many lawyers in this world. If what you suggest were true there would never be any contract disputes heard in courts because the terms of the contract would be clear and unambiguous. Instead, lawyers are fully employed arguing over exactly what particular terms mean in contracts - English (or whatever) is far from clear and unambiguous even with the best will in the world.
Posted by: John S at July 9, 2005 10:21 PM
John: I think you are being more than slightly tendacious. The method of MBH98 has been replicated by several people.
The standard is (I will quote from the AGU instructions to authors, although the equivalent can be found in almost all other journals) "A paper should contain sufficient detail and references to public sources of information to permit the author's peers to repeat the work."
The standard has been met.
The standard is not that a method can be replicated by every hobbyhorsist standing at the bar in the Competitive Enterprise Institute.
The standard is not that a mathematical representation of any algorithm be presented.
In short you and many others are engaged in a moving the goalposts exercise.
Posted by: Eli Rabett at July 9, 2005 10:48 PM
Depends what you mean by moving goal posts: I have not changed what I am arguing for, my goal posts are unmoved; I will argue, however, that the goal posts of disclosure as set out in, for example, the AGU guidelines, should be moved. Standards of disclosure vary by discipline and mine are set out here: http://www.aeaweb.org/aer/data_availability_policy.html. As you will notice, these are in excess of current AGU policies. I do not argue that MBH, or any other climate scientist for that matter, have not complied with the AGU guidelines. I argue that that standard should be higher. And to tie this back into the purpose of Prometheus, I argue it should be this way because the science is used by people other than the scientists' peers - it is used for public policy.
Posted by: John S at July 10, 2005 12:25 AM
...and I just looked up tendentious and accept that. My point of view is controversial.
Posted by: John S at July 10, 2005 12:35 AM
I think that Barton is correct in his request for Mann et. al. to document their data and methods in detail. To understand why, I'd like to point out an interesting parallel to another scientific case that came before Congress 15 years ago, cold fusion. Both cold fusion and the hockey stick carry three common elements: (1) an extraordinary claim; (2) huge economic consequences, and; (3) the response of the scientific community.
(1) Pons and Fleischmann made the extraordinary claim that they had produced significant nuclear fusion by chemical means, electrolysis of heavy water. And they had done it without producing much nuclear radiation of either alpha particles or neutrons. They had produced "cold fusion" in contradiction to the known physics of nuclear reactions and quantum mechanics. (2) If this were true, energy would suddenly become very cheap and abundant [and P&F would become richer than God because they held the patent and he didn't] The social and economic implications of this discovery were almost unimaginable and would change the world. (3) The physics community immediately tried to replicate the experiments and calculate, using known physics, to understand if the phenomenon was real. A few experimental groups "replicated" cold fusion, but the overwhelming majority did not. Quantum mechanical calculations of the nuclear physics also pointed to an false experimental result. Except for a few acolytes, no one bothers with cold fusion anymore.
(1) MHB98 made the extraordinary claim that the Medieval Warm Period and the Little Ice Age never happened and that the twentieth century was the warmest in the last thousand years. This was in direct contradiction to hundreds of studies that showed the existence of both the MWP and the LIA. (2) The IPCC used MBH98 to push the Kyoto protocols which would cost the world trillions of dollars of lost economic activity because of the need to reduce human production of CO2. (3) The community of climate scientists appears to have accepted the validity of the MBH98 hockey stick despite the overthrow of years of research that shows otherwise. And, with the very public exception of McIntyre and McKitrick, it has done so without any debate. This is very unlike the case of cold fusion. Yes, there is "replication" of MBH98, but there was replication of cold fusion too.
In the case of global warming, Barton, by asking Mann et al for detailed justification of their work, is taking up the part played by the critics of cold fusion . This is a part that should have been played very loudly and vocally by climate scientists but hasn't.
Posted by: Paul at July 10, 2005 02:05 PM
Why don't some of the proponents that full disclosure has already occurred try writing up a program that replicates Mann's results and publish it (along with all source code). Use Fortran, Excel, and Matlab or whatever suits your fancy (as long as it is available to the public). I look forward to your response!
Posted by: Frank Scammell at July 11, 2005 09:14 AM
Already been done on sci.env, Frank. Thanks for asking.
Hans, if you're still checking this, we're not debating an 8-year old paper. The subject is von Storch's reaction to Barton's demands. please stay on topic.
Posted by: Dano at July 11, 2005 02:11 PM
a fascinating post. Hans argues that climate scientists, uniquely amongst all science, should be exempt from the requirement to record and keep a record of what they have done.
lab scientists keep lab notebooks for good reason. In the Pharmaceutical industry, lab records must be at the higher and statutorily-defined standard of GLP.
But when climate scientists get several million dollars to do research on computer, they can't even burn CDs or keep records on tape, because it is too difficult for the poor darlings.
What is wrong with this picture ?
Posted by: per at July 12, 2005 03:59 PM
I was arguing that it is *only* eight years old, For a publication this young, there is zero negative effect for digital archive retrieval.
Posted by: Hans Erren at July 17, 2005 05:35 AM
And another thing, the Barton letters are about MBH. So I think my comments are on-topic.
Posted by: Hans Erren at July 17, 2005 05:38 AM