NEW ZEALAND QUALIFICATIONS AUTHORITY AGAINST SUNDAY STAR-TIMESThe New Zealand Press Council has upheld in part a complaint against the Sunday Star-Times by the New Zealand Qualifications Authority following the newspaper’s publication of an article on March 16, 2008, headlined: “NCEA battered by secret report.”
The page three article began: “The first major analysis of schools’ NCEA marking shows many teachers are awarding better grades than examiners do – with high-profile schools among the dozens suspected of marking too high.” The second sentence said: “This means students who shouldn’t have passed NCEA may have. Others might have missed out on university entrance because their school marked tougher than others.”
The newspaper went on to say the release of what it called the “secret analysis” to the Sunday Star-Times under the Official Information Act had sent the NZQA into “damage control” to warn schools about the results, and it said many principals had said they didn’t know about the study or findings.
“NZQA is now scrambling to refine its figures, work out the size of the problem, and bring school marking into line,” the article continued, explaining in the seventh paragraph that in 2007, NZQA had analysed all schools’ marking, comparing their 2006 exam results with internal assessments grades. According to the report, one in four of the country’s schools would be under scrutiny for the marking.
The newspaper reported in the eighth paragraph that the results were tentative and should be interpreted with caution. Some 63 schools had given more generous markings than examiners while 61 schools had given lower grades.
In the 13th paragraph, it reported: “Although the gap between internal assessment grades and exam results tended to be bigger at low-decile schools, many top schools featured on the list.”
Discussion between the NZQA and the Sunday Star-Times
On March 20, the chief executive of the NZQA, Dr Karen Poutasi, wrote to the editor of the Sunday Star-Times, expressing her disappointment about the article. The research data had been released to the reporter with an explanation that the research was at its early stages and that only tentative conclusions, at best, could be drawn from it.
The nub of the complaint was the newspaper’s reference to schools statistically identified as “outliers” for certain subjects. These are schools at which results for internal assessment are either unexpectedly better or poorer than those for external assessment and greater than the majority at other schools. As part of the analysis, however, the NZQA was automatically treating 5 percent of schools in each subject as outliers.
In her letter, Dr Poutasi said the work was far from secret and had been announced at a media conference on May 29, 2007. It was also explained on the NZQA website. Contrary to the article and heading, there was no report as the process was still being refined, and raw data was still being analysed. The NZQA was mid-way through this process.
It was nonsense to say the NZQA was “scrambling” to refine its figures. A key part of the research was the inclusion of “Not Achieved” data for internal assessment which was being reported for the first time this year, and couldn’t be included until it was reported.
The NZQA was not in “damage control” over release of the information. It was keeping its customers informed that the raw data was being released halfway through a research process because it was required to release the information under the OI Act.
There was no evidence students had received incorrect grades as a result of the internal assessment. The raw data merely identified instances where the results varied from the mean by more than two standard deviations. There were a number of reasons why this might occur.
The editor of the Sunday Star-Times responded on March 26, saying the story was not misleading and that it had made clear the figures should be interpreted with caution.
“The key trend” in the article that internal assessment marks are generally higher than externals, and that gap tends to be larger at low-decile schools, was confirmed by the NZQA itself.
The headline had referred to a “report” and while she accepted it was not strictly a report, the distinction was not significant from a reader perception.
The editor said she believed the use of the word “secret” was justified. While the intention to compare results was mentioned in May last year, references NZQA had sent were fleeting and vague.
While a Principals’ meeting in August 2007 had discussed the figures, principals of schools the newspaper had spoken to did not know they were considered “outliers” until the NZQA contacted them as a result of the newspaper’s inquiries.
A version of Dr Poutasi’s letter was published on March 30. At the bottom was an unsigned note, presumably authorised by the editor, saying the newspaper had responded privately to a not-for-publication version of the letter published, “strongly rebutting each of the points raised.” The note defended the article.
Not satisfied with the outcome, on April 10, Dr Poutasi formally complained to the Press Council. With her letter, she included correspondence between the newspaper and the NZQA in the days prior to publication.
Dr Poutasi reiterated that the main heading – “NCEA battered by secret report” – was inaccurate. There was no such report and NCEA was not battered by the raw data supplied.
The first paragraph stating “analysis of schools’ NCEA marking” showed many teachers awarding better grades than examiners was the newspaper’s own analysis. NZQA had repeatedly stated any conclusions were at best tentative and there could be a range of reasons why a school might appear in an “outlier” category, most of which were not due to any inappropriateness in marking. No schools were “suspected of marking too high.”
There was no “secret analysis.” The work had been discussed at a May 2007 media conference, written about in other newspapers and referred to in background on the NZQA website.
The article referred to NZQA Deputy Chief Executive Bali Haque discussing the issue at a Principals’ meeting and said he “made it clear that he was deeply concerned about them (the results).” This suggestion was not raised with Mr Haque prior to publication and his concern when discussing the issue with principals was not about the data but the likely tone of coverage by the Sunday Star-Times.
The editor had said the article had said the “results” were tentative and should be interpreted with caution. But that was in the eighth paragraph and, as stated repeatedly to the reporter, the data provided was in raw form and did not constitute results. Any conclusions were at best tentative and that was by no means made clear in the tone or content of the article or headline.
The release of the information to the Sunday Star-Times had not sent NZQA into “damage control.” It was carrying out its normal practice of informing schools involved if they were likely to be the subject of media interest and the NZQA’s response.
It was baseless to say NZQA was “scrambling to refine” the figures because the data supplied remained the only data and identical data had been supplied to other media.
In her letter, the editor had misrepresented the article when she said the key trend outlined was that internal assessment marks are generally higher than externals. Its theme was that a serious issue existing in that internal assessment led to students receiving incorrect grades. There was no evidence of that. It had long been recognised that achievement is higher in internal assessment than in external.
Dr Poutasi said it was unacceptable that in spite of being referred to research at Auckland University indicating the validity of NCEA assessment, the Sunday Star-Times printed “a sensationalist, negative article” and then attempted to “hide behind” qualifiers such as “might” and “may” as representing balance.
The Newspaper’s Response
The newspaper’s deputy editor wrote at considerable length of the efforts the reporter had gone to so that a balanced and comprehensive report could be written about the data. The reporter had repeatedly asked for access to the methodology used and to speak to an NZQA statistician. The requests were declined.
Because of the “lack of assistance” offered by NZQA, the newspaper had sought the advice of two independent statisticians who had supported trends about internal and external examinations and low-decile schools. These trends had been confirmed by the NZQA.
Because of the NZQA’s concern over figures, the newspaper took great care in choosing which figures to quote.
The deputy editor defended the heading. The raw data showed clear trends, and the anomalies clearly damaged the credibility of NCEA marking.
Similarly, she justified the word “secret” because references to the work had been fleeting and made no mention of their findings or potential import.
Dealing with the first paragraph, she said it could not be disputed that there was a suspicion that some schools were making too high.
The deputy editor reiterated her editor’s remarks that the newspaper had reported the results were tentative and should be treated with caution, that the newspaper had good cause to believe the proposed article sent NZQA into “damage control.”
Justifying the newspaper’s statement that NZQA had been “scrambling to refine its figures”, the deputy editor said the tone of its communication to principals “to us implied a hasty response to a damaging revelation in the wake of confirmation that the Star-Times was doing a story it clearly would have preferred to remain unreported.”
The reporter had two sources for the information that Mr Haque had told principals he was deeply concerned about the results and that their variability revealed a huge problem for NCEA. The paper did not think it necessary to take those concerns to Mr Haque because the NZQA’s significant efforts to follow up the data confirmed the level of the authority’s concern.
The deputy editor also said that NZQA’s concerted efforts to investigate internal assessment issues confirms “the scale of the marking variability is more than the authority believes is appropriate . . . The story served simply to highlight that the new data had revealed the disparities and work was underway to monitor and resolve them.”
The analysis of raw data and statistics requires the greatest of care, and it is clear from the correspondence and additional material supplied that both parties to this complaint understood that. The additional material, in particular emails and written responses, gave the Press Council insight into the complexity of the subject and how the parties negotiated with each other.
This material proved helpful to the Council when it came to deliberate on the complaint, even though the anchor points for those deliberations have to be the details of the complaint and what was published in the newspaper article.
Of the headline, Dr Poutasi complained there was no actual report and certainly not a “secret report”. The word “report” usually indicates some formal documentation and the newspaper seemed prepared to concede there was no such document but that readers had not been badly misled.
As to its use of the word “secret” in the heading and in the story when referring to analysis, the newspaper said any public references to the issue were fleeting and vague. The New Zealand Oxford Dictionary definition of “secret” is “kept or meant to be kept private, unknown, or hidden from all or all but a few.” Some details of the NZQA’s analysing of its data were known and had been made public– they were not a secret.
The issue of whether Mr Haque’s comments to principals should have been put to him before publication is a troubling one. The deputy editor saw no reason to do so, yet the newspaper’s reporter had said in an email on March 4 to the NZQA’s senior media advisor: “Just to fill you in, I had a call from Bali this afternoon . . . I repeatedly reassured him that we will not be running this without putting everything to NZQA first . . .” In view of such assurances about “everything,” the newspaper was unwise to overlook the commitment of its reporter.
The newspaper article made at least two references to the tentative nature of the data but Dr Poutasi complained they were not sufficiently prominent, especially when the emphatic heading and introduction are taken into account. An ordinary reader of the entire article would have received that message but at first glance, there was a strong indication that NCEA was in trouble.
The bold first heading sits oddly with the tentative nature of the research and analysis which the newspaper says it accepts. It follows therefore that any conclusions had to be tentative – reported as such - and could not bear the weight of unconditional conclusions.
The newspaper rightly says it did advise readers the results were tentative. The issue for the Press Council in considering this aspect is whether the overall impression from the heading and article supports that contention.
The discussion over whether the newspaper’s inquiries sent the NZQA into “damage control” seems based on semantics. The NZQA says advising principals of likely media interest was normal procedure, the newspaper saw something more sinister and feared other media were being alerted. The newspaper’s description of what happened, while more than the NZQA believes justified, is not unreasonable.
The claim that NZQA was “scrambling to refine its figures” could also be considered reasonable in that the NZQA is going to analyse the data further and take up issues raised with relevant schools.
Dr Poutasi said the editor had misrepresented the thrust of the article when she said the “key trend” of the story was that internal assessment marks were generally higher than external. But as this criticism does not apply directly to the article itself, the Press Council has no need to make judgment.
There is huge public interest in NCEA. The Sunday Star-Times was justified in making its inquiries. But it had a duty to ensure its article was accurate, fair and balanced. Articles based on raw, incomplete data have their risks.
Documents presented to the Press Council help explain some of the processes both parties went through to ensure a balanced report was published. Both parties appeared to begin the process in good faith but it is clear that towards the end, some tensions arose. This was regrettable.
The heading pushed beyond the boundaries of accurate reporting or correct analysis of the data. Given that everyone agreed that the raw data had to be treated with caution, there was no place for absolutes or firm conclusions.
It is unfortunate in the Press Council’s view that it was presented unconditionally because the article would have suffered little from a less emphatic initial tone and without dramatic references to a “secret report” or “secret analysis.” Nevertheless, the Sunday Star-Times was entitled to make its own analysis of the raw data received.
There was, however, a semblance of balance about the incomplete nature of the data within the body of the story.
The complaint that the headline is inaccurate is upheld. There was no report as such, and the error is quite significant. Given that the data is inconclusive – something the newspaper concedes – it is a step too far to say that the NCEA has been “battered.”
The complaint that the use of the word “secret” in the heading and text is inaccurate is upheld. Secrecy implies a deliberate withholding of or attempt to conceal information and this is clearly not the case.
The complaint alleging in effect a lack of balance in not giving Mr Haque an opportunity to respond to statements he was reported to have made at a meeting of principals, which appeared to be at variance with later statements, is upheld. The newspaper’s reporter had said she assured Mr Haque “everything” would be put to the NZQA for a response. Having given that assurance, the newspaper should have kept to its word.
The complaint that insufficient prominence was given to the tentative nature of the data is difficult. Its tentative nature was acknowledged but offset by the strong first heading. Such an important aspect should have been accommodated. The emphatic nature of the heading contrasts confusingly with the later statements about the tentative nature of the data. On balance, however, the shortcomings of the data are acknowledged and the complaint is not upheld.
The complaint about the statement of “damage control” is not upheld. The Sunday Star-Times was entitled to draw that conclusion.
The complaint about the statement of “scrambling to refine its figures” is not upheld. The newspaper had reason to believe this statement was true and, given that the figures will be refined, it is not inaccurate.
Press Council members considering this complaint were Barry Paterson (Chairman), Aroha Beck, Ruth Buddicom, Kate Coughlan, John Gardner, Penny Harding, Keith Lees, Clive Lind and Denis McLean.