ACT says NAPLAN comparisons are unfair. A buried report suggests otherwise

The Education Directorate is pushing to change the way socio-educational advantage is calculated for ACT students but experts say it does not explain widespread underperformance in schools.

Subscribe now for unlimited access.

or signup to continue reading

ACT schools' literacy and numeracy NAPLAN scores are behind similar students under the current model. Picture by Karleen Minney

They say there are "anomalies" that mean students in the highly advantaged territory are shown as being behind similar cohorts from other states and territories when adjusted for socio-educational advantage.

A five-year-old report has surfaced through freedom of information that shows the widespread underperformance does not change when compared with six other methods of calculating socio-educational advantage (SEA).

The Centre for International Research on Education Systems (CIRES) at Victoria University was commissioned by the ACT Education Directorate to explore how socio-educational advantage worked in the ACT.

The 2021 report said that when looking at the original NAPLAN data for year 9 numeracy, 35.1 per cent of ACT schools were top performers.

But when compared with similar schools, the proportion of schools in the top 25 per cent fell to 2.7 per cent.

"The results for ACT schools are unchanged by using alternative constructions of SEA, or by how SEA is applied in the similar-schools model," the report said.

"That does not mean that there are no anomalies in SEA and its construction. Rather, further work on how SEA is constructed, and analysis of the way SEA affects the ACT, may be needed."

The report also said: "There is no evidence to suggest there is a systemic performance issue in the ACT, given outcomes on other valued student outcome measures."

The report did not elaborate on what the other "valued student outcome measures" were and suggested further work may be needed.

Comparison 'valid'

Centre for Independent Studies education fellow Trisha Jha said the report showed ACARA's method for comparing similar schools, known as the Index of Community Socio-educational Advantage (ICSEA), was valid.

"Fundamentally, it shows that the calculation of the socio-economic advantage used by ACARA that underpins its similar schools model of comparison between jurisdictions is valid, and ... there is no underlying issue with the data in terms of the quality of the input. That's very clear," Ms Jha said.

"In order for this model to be changed, the ACT would have to convince all the other states and territories that there's something wrong with it, and I don't think that there's enough evidence to suggest that there is something wrong with it."

School education policy expert Dr Peter Goss said it could be worth examining how schools at the high end of the socio-educational index were affected by the similar schools comparison, which would affect the ACT more than other jurisdictions.

"At the very top, it's hard for ICSEA to discriminate among schools. Personally, I think it might be worth looking at that in more detail," Dr Goss said.

"I'm not convinced that it's the major explanation why the ACT's results tend to look lower than other jurisdictions once you adjust for ICSEA, and I'm not sure why whether it would explain why it looks lower, as has been reported in a bunch of places, including work I've done in my 2018 report measuring student progress."

NAPLAN 'a national treasure'

Dr Goss co-authored a 2018 Grattan Institute report that showed that students in the ACT made two to three months less progress than the national average in primary school and secondary school.

He said the ACT would have to meet a high bar to persuade the other states and territories to make any changes.

"NAPLAN, I think, is a national treasure as a resource. We are incredibly fortunate to have it, including the comparisons with ICSEA, including the similar student comparisons, because that means we can actually understand and look for educational improvements at scale," Dr Goss said.

"Because we have this really well solidified data set, the threshold for any change should be pretty high. Other states and territories may well be nervous about it. The ACT would have to make a pretty compelling argument as to why it was worth changing, in my view, to get that through."

An ACARA spokesperson said it was still reviewing how the ICSEA model worked in the ACT in collaboration with the Education Directorate.

"As the ICSEA model is a national one, any proposed changes will require agreement from all states and territories to be implemented."

An ACT Education Directorate spokesperson said ACT school system representatives had met ACARA to discuss the issue.

"ACARA agrees there are anomalies and that data from all ACT schools consistently and disproportionately shows all ACT schools as underperforming against the 'similar students' measure, which contributes to how comparative NAPLAN scores are calculated."

The spokesperson said ACARA briefed its expert measurement advisory group and the national assessment, data, analysis and reporting reference group in 2025.

The My School website has a caveat for all ACT schools that states: "A review of how SEA calculations are used in 'similar student' comparisons has identified anomalies for ACT schools. Investigations are continuing."

The Education Directorate spokesperson said the matter would not be settled in time for the next My School update but it hoped to reach a resolution for subsequent updates.

More from Education