Tepid support from BoM
Guest author | January 9, 2011
Constrained support
The Australian Bureau of Meteorology (BoM), like the National Institute of Water and Atmospheric Research (NIWA) in New Zealand, is an advisory group of Government scientists responsible for the compilation and maintenance of official temperature records.
After NIWA scientists rewrote the official NZ temperature record — the Seven-Station Series — during 2010, their ‘Review Report’ included a letter of support from the BoM. This was seen as necessary, as NIWA’s credibility had been somewhat strained by its lengthy (and ultimately futile) defence of the old record.
Some are critical of the selection of the Bureau to review work by NIWA, as both groups have been widely criticised (especially in the blogosphere) for applying the same biases and questionable adjustment methods. See, for example, Australian Temperatures in cities adjusted up by 70%!? at Jo Nova’s blog.
As climate archivists, both agencies are extensively engaged in the work of the IPCC; and both are firmly of the school of thought led by Professor Phil Jones, head of the Climatic Research Unit at the University of East Anglia (UEA).
The NZ Climate Science Coalition wrote to Science Minister Wayne Mapp, suggesting the appointment of two genuinely independent reviewers, and putting forward names of highly regarded scientists and statisticians. The letter pointed out:
“Appointment of two additional reviewers need not create any delays or distractions. Irreconcilable differences are unlikely, once all the data and metadata are disclosed, but, if they do arise, it is far better that the differing views are transparent and defined. The reviewers’ role is only advisory, and NIWA will retain full control of both the process and outcomes, which must, however, include the publication in full of the reviewers’ report.”
This suggestion was not taken up.
Minister Mapp advised Parliament last February that the NZ Temperature Record was to be reviewed by Brett Mullan, and then peer-reviewed internally by David Wratt and James Renwick and externally by the Bureau. The work would then be submitted “as a paper to a scientific journal where it would be subject to the normal independent peer-review process.”
Terms of peer review not disclosed
A “normal independent peer-review” sought by a journal editor is reasonably well established by convention. In the best instances, the data and metadata are also made available so that the scientific or statistical work is reproducible by interested outside experts. But what are non-independent “internal and external peer-reviews”? They are not defined, and may mean whatever the user intends them to mean.
An analogous guide is provided by reviews of historical financial information, as laid down by International Standard ISAE 3000. Clearly, a peer-review does not reach the stringency of an “Audit” where the reviewer certifies that the record presents a “true and fair view”. ISAE 3000 requires the terms of any “Assurance Engagement” to be recorded in writing, defining such issues as evidence sources and materiality levels, and demands “an attitude of professional scepticism.”
The engagement letter between NIWA and the BoM remains confidential. But the role actually carried out by the BoM appears to be the ISAE 3000 equivalent of a “Limited Assurance” — the lowest standard that a member of the NZ Institute of Chartered Accountants is ethically permitted to undertake.
The BoM makes clear that its review is not a reanalysis of the record and is constrained to the “level of information supplied”. The Bureau’s formal assurance is stated in a single sentence:
“In general, the evidence provided by NIWA supports the homogeneity corrections that have been applied to the temperature record to create the ‘seven station’ series.”
Paraphrase: The material which NIWA chose to give us was supportive of their conclusions – or – NIWA offered us no evidence contradicting their own opinions.
No approval for methodology
But this needs to be read in the context of the many express caveats. NIWA’s data and methodology are assumed to be “an accurate representation of the actual analysis undertaken”. The Bureau was “not in a position to question all of the underlying analyses and data that have contributed to the final results.” Above all, there was no attempt to “independently determine the sensitivity of … the analysis methodology.”
The final sentence of the BoM letter has a less constrained assurance:
“It is also clear that a number of significant adjustments (as identified by NIWA in the reports) are clearly required for the raw/composite station series owing to inhomogeneities which would otherwise artificially bias results.”
But this is not contentious. Many inhomogeneities certainly exist, but the question is whether the NIWA adjustments have made the record better or worse? And with what degree of certainty?
BoM had limited access to data
The Bureau reveals that it did not have “full access to the raw and modified temperature data and metadata.” This is contrary both to assurances offered by Minister Mapp and the express injunction stated by the UK House of Commons in its recent report on “Climategate”:
“Climate science is a matter of great importance and the quality of the science should be irreproachable. We therefore consider that climate scientists should take steps to make available all the data that support their work (including raw data) and full methodological workings (including the computer [code]). Had both been available, many of the problems at UEA could have been avoided."
All in all, the Bureau's letter offers little or no assurance that NIWA scientists have got it right this time. At most, it offers only tepid support to NIWA’s Review Report.
I haven’t had much time to spend in the last 2 to 3 weeks but my concerns would be
a) the discrepancy between the dates to which RichardT refers in a recent post (sorry don’t have time to find the post
b) is the letter from BOM at p 13 of the NIWA report the BOM review or some preceding document from BOM
c) the letter from BOM is basically a pat on the back to the payer and the question is what data was it given
d) I see no reason why ‘the letter of engagement’ should be ‘in commercial confidence’ after all both entities are there to serve the public interest; the BOM I’m assuming was paid from NZ taxpayers funds and if I were acting on behalf of the NZCSC I would demand it
e) depending on the result to d) above then the review has or has not been done
Very quick thoughts and thanks Barry for your article and to RichardT for preceding thoughtful articles
Good summary. This is not over.
We want to see the REAL review from the BoM.
We want to see NIWA’s review submitted to a REAL peer review prior to publication.
thanks RichardT; enjoyed your ‘did the dog bark in the night’ post; I think I was the first commenter to your blog to ask that question but in respect to a previous article by you
I’m sure Barry and the NZCSC’s lawyers would agree that all data submitted by NIWA to the BOM together with what Barry calls ‘the letter of engagement’ are subpoenaeble in the current Court proceedings instituted by NZCSC together with all subsequent correspondence etc between/from NIWA and BOM
The Aussie Govt is keen on pleading commercial in confidence to prevent docs being produced but in my view this does not apply to NZCSC current case and the ‘review’; – commercial in confidence is used to prevent a competitor using material to an entities detriment and this does not apply in my view to production of the stuff referred to above as there is no competitor and in addition the review was paid for by public monies;
So let the dog start to bark and don’t let the politicians stop it
The politician’s ideas about peer review and commercial sensitivity rank right up there with ‘I got Dad to check me homework and he reckons I got all me sums right” and ‘I lost the change from the fiver Mum gave me to buy the bread and milk on me way home from the dairy’.
Children playing very transparent games, and they wonder why the grown-up electors see through them every time!
Ain’t that right, Alexander! Nicely said.
[...] go to plan, because they managed to get out of giving any positive endorsement of the new NZT7 [see “Tepid Support from BoM”]. You’d think NIWA’s friends might at least pretend to like the review! But all the BoM [...]
[...] to each other. Bearing in mind that the Australian Bureau of Meteorology (BOM) has given lukewarm support for the New Zealand temperature record, it is more than a little worrying. More to the point, when [...]
“Some are critical of the selection of the Bureau to review work by NIWA, as both groups have been widely criticised (especially in the blogosphere) for applying the same biases and questionable adjustment methods.”
And long may it continue. BOM’s ACORN-SAT is getting the 3rd degree from the likes of Ken Stewart, Warwick Hughes and Joanne Nova but I don’t think they have got to grips yet with the nature of the adjustments (that BTW, make NIWA’s NZT7 adjustments look rather ordinary)
When I asked Ken outright if BOM applied cumulative step adjustments in the case of Alice Springs as an example his response was “No”. I’ve pointed out to him that their adjustment methodology (no mention of R&S) detailed in CAWCR Technical Report No. 049 http://cawcr.gov.au/publications/technicalreports/CTR_049.pdf for 660 steps (Table 6 page 72) was justified by, for example:-
7.4 The percentile-matching (PM) algorithm ………………………………………………49
7.4.1 The overlap case…………………………………………………………………………….49
7.4.2 The non-overlap case……………………………………………………………………..50
From the ACORN Station catalogue http://www.bom.gov.au/climate/change/acorn-sat/documents/ACORN-SAT-Station-Catalogue-2012-WEB.pdf the Alice Springs site changes that might require step adjustments are (my compilation):-
***************************************************************
1910 – 1932 (22 yr span) Telegraph Station,
No overlap
1932 – 1944 (12 yr span) Post Office,
12 yr overlap
1944 – 1974 (30 yr span) Airport
Minor 550m site move at same site
1974 – 1996 (22 yr span) Airport
Automatic weather station (AWS) installed at same site, 5 yr overlap.
1996 – 2002 (6 yr span) Airport
Site moved a few metres
2002 – present (10 yr span) Airport
102 yr series.
**************************************************************
BOM don’t explicitly say that any steps were applied “cumulatively” as in NIWA’s NZT7 but I don’t know how they would apply them otherwise. I suspect that there’s a table of station step changes somewhere that Ken et al have yet to unearth. I pointed out that it is the early step changes that are the major NIWA NZT7 items in dispute by NZCSET, not individual datapoint adjustments
Bizarrely, BOM’s individual adjustments (not step changes) that, although those also occur in the NZT7, are far more extensive in ACORN. As Ken describes for the entire ACORN series:-
Ken has graphed the individual datapoint adjustments (no step changes identifiable) for Alice Springs here (Post Office and Airport) https://kenskingdom.files.wordpress.com/2012/05/alice-acorn-raw-min.jpg?w=450&h=217
I think you will all agree that if in fact those individual adjustments are IN ADDITION to step change adjustments that this would make BOM Adjuster par excellence and any “support from BOM” to any degree of warmth is a dubious recommendation at best.
ACORN – SAT posts
http://joannenova.com.au/2012/06/threat-of-anao-audit-means-australias-bom-throws-out-temperature-set-starts-again-gets-same-results/
http://kenskingdom.wordpress.com/2012/05/14/acorn-sat-a-preliminary-assessment/
http://www.warwickhughes.com/blog/?p=1584