Qualifying conclusions

Gavin describes a trojan horse at RealClimate in On replication. The issue is about a mis-focus to either credit or discredit an idea. It is one where basic ideas, reason, and logic get usurped by a mechanical process that can yield misleading results.

the vast majority of papers that turn out to be wrong, or non-robust are because of incorrect basic assumptions, overestimates of the power of a test, some wishful thinking, or a failure to take account of other important processes (It might be a good idea for someone to tally this in a quantitative way – any ideas for how that might be done?).

In the cases here, the issues that I thought worth exploring from a scientific point of view were not whether the arithmetic was correct, but whether the conclusions drawn from the analyses were.

the bigger point is that reproducibility of an analysis does not imply correctness of the conclusions.

Where this essay has problems is in its title. It seems that claiming the issue is about transparency and replication is a misdirection. That leads to a suspicion that it is trying to dismiss calls for a transparency of data and process that would allow others to replicate the chain of steps that lead to a conclusion. That would be an attempt to sidestep the question “how did you come to that conclusion?” which has been an issue in the climate discussion.

Comments are closed.