Winter 2005

The Assessment agenda

Unpacking PISA

Every three years the Programme for International Student Assessment (PISA) collects statistical data from 30 OECD countries about the performance of 15-year-olds in a range of competencies. In many countries, including Australia, the statistics show significant variations between schools and between students. BARRY MCGAW discusses some of the reasons why.

COUNTRIES COMPARE THEMSELVES WITH OTHERS all the time. We are used to it in economic and sporting performance but now we see it increasingly in educational performance. There are international rankings of universities; data on proportions of the national populations completing secondary and tertiary education; data on the actual performances of students; and so on.

All 30 Organisation for Economic Co-operation and Development (OECD) member countries assess their 15-year-olds every three years in reading, mathematics, science and some more general competencies in OECD’s Programme for International Student Assessment (PISA). Other countries are joining in the comparisons—in PISA 2006 there will be 27 of them.

The simplest way to compare the performances of students across countries is with their average results. That produces the rankings that make easy headlines, but it is also important to look at variation in performance.

In the PISA 2003 mathematics results, the variation is more than a third greater in some countries than others. On average, across the countries, the variation between schools is about half the variation within schools, but that average also hides remarkable differences. The results for all OECD countries in the 2003 survey are shown in the figures (see p 6). (The UK’s results are not included because of inadequacies in their sample. France’s results are not included because France withheld data on schools.)

In the Nordic countries, almost all of the variation among students occurs within schools; there is very little difference between schools— Iceland (4%), Finland (5%), Norway (7%), Sweden (11%) and Denmark (13%). In the US, where schools are largely funded by local communities, 26% of the variation is between schools. One of the strong influences on house prices in the US is perceptions of school quality and, more and more, data are being added to inform the perceptions. The ratings agency, Standard and Poor’s, has a website (www.schoolmatters.com) that offers data on schools and districts for people shifting locations and deciding where to live.

Australia is not far behind the US, with 21% of the variation among students being variation among schools. In Australia, choice of community is only one factor, since there are also public and private options. There are some strong perceptions about differences but little unambiguous evidence. In the Australian data in PISA, for example, information on differences between public and private schools is not available because the information on school type is suppressed.

Looking across the OECD as a whole, private schools have better average performances than public schools in most countries, though not in Luxembourg, Japan, Italy and Switzerland. Once differences in the social backgrounds of students are taken into account, however, all the superiority of private schools disappears. It is the homes, not the schools, which make the difference. We don’t know if that is true in Australia because, by and large, we don’t ask.

In some countries large differences among schools exist by design. In some, more than half the variation among students is due to variation between schools; for example, Netherlands (58%), Hungary (58%), Japan (53%) and Germany (52%). These do not play out in the same way in average performance levels—15-year-olds in the Netherlands and Japan have high PISA average scores while those in Germany and Hungary have low averages.

In Germany, the large differences between schools are deliberately created by streaming students into schools of different types—academic and vocational— from the age of 12. There is a strong educational rationale for this practice. The intention is to provide students with an education that best fits their needs and capacities, determined by their educational performance up to age 12. It turns out, however, that almost 80% of the large variation between schools that results from this practice is attributable to differences in the social backgrounds of students. Students from socially advantaged backgrounds are most likely to be tracked into high status, academic schools where high academic performance follows. Students from less socially advantaged backgrounds are much more likely to be tracked into low status schools which achieve low academic results. The school system is, in effect, reproducing the existing social arrangements and, in the process, producing low average results.

Japan and the Netherlands have a similar pattern in relation to social background, though both achieve high average results. In the Netherlands, where there is also strong streaming into academic and vocational schools from an early age, 75% of the variation between schools is attributable to differences in the social background of students. In Japan, the figure is almost 70%.

Since 1999, Poland has moved away from the Germanic system of differentiated schools and early tracking of students in academic and vocational tracks, replacing it with an integrated system of comprehensive schools. Poland is the only country that improved its average performance on all PISA scales measured in both 2000 and 2003. It did so by raising the performance of its lowest performing students. No longer assigning these students to schools that label them as low performers and, in the process, setting higher expectations for them has made a difference.

Labels and expectations matter. We can do better without the former and with the latter set high for all students.

Barry McGaw is director for education at the Organisation for Economic Co-operation and Development in Paris, France.

The author owns the copyright in this article. For information related to the reuse of this work in any form please contact the publisher denise.quinn@curriculum.edu.au