Winter 2005

The Assessment agenda

Going to bat for standardised testing

JULIETTE MENDELOVITS takes a cool look at a hot topic. PISA results show that while Australia rates pretty well by international standards, our strengths are in pragmatism and information rather than criticism and reflection. Could this be a starting point for debate on State and national assessment practices?

AUSTRALIAN EDUCATORS GENERALLY ARE HOSTILE to standardised tests. This was brought home to me afresh when, preparing to write this article, I read through the autumn 2004 issue of EQA , ‘Talking English’. Two of the articles took swipes at standardised tests. Implicit in the manner of both was the assumption that they would find easy assent among their readers to comments like Paul Sommer’s, ‘we seem intent on following trends in the UK and USA, which narrow and restrict the curriculum through standardised testing’; or to Brenton Doecke’s throw-away assertion that governments are ‘making a fetish of standardised literacy testing’ while ‘no one interrogates the assumptions about literacy behind such tests’.

The idea that standardised tests dominate the teaching of English or any other subject in Australia is laughable. In most jurisdictions, tests are administered only at three year levels (years 3, 5 and 7). Far from making a fetish of testing, the federal and State ministers and directors of education have recently decided not to introduce standardised tests in literacy and numeracy at middle-secondary level, but rather to use the three-yearly results from the Programme for International Student Assessment (PISA) for benchmark-setting at that level. As far as the State-based population tests of literacy and numeracy are concerned, it seems that their reports are barely registered at the classroom level. Where they do have some impact, it is most likely to be at whole school or policy level, in the allocation of resources and personnel to address areas of weakness that the assessments have exposed.

The idea that standardised tests necessarily narrow and restrict the curriculum is also a furphy. No one would argue that the format and specific content of tests should dictate exclusively what is learnt in schools and how it is learnt. But good tests will encapsulate some of the essential understandings and skills that students are expected to have developed, and their results are therefore real indicators of learning. Similarly, while ‘teaching to the test’ is a phrase guaranteed to generate an automatic hiss of disapproval, it is a useful and proper activity when something essential is being tested. Good standardised assessments can model good assessment practice. Further, they provide pointers to how curriculum content and pedagogical style might be changed for the better.

I want to use the PISA reading literacy study to elaborate the argument that good standardised tests can be both illuminating and a force for good.

In some of the 41 countries that have participated in PISA to date, it has already had a profound impact. In Japan, PISA’s findings about the relationship between subject time in schools, amount of homework and cognitive performance has led to serious discussion about the length of the school day and the hours of homework demanded of Japanese students. In Germany, which has been shocked by its just-average performance in all the cognitive areas assessed by PISA, and especially by what PISA has revealed about German students’ sense of alienation from teachers and other adults, there has been a good deal of soul-searching and the beginnings of some educational reforms in school structure and educational ethos.

Every taxi driver and teenager in Germany has an opinion about PISA. By contrast most Australians—even teachers—have never heard of PISA, except the leaning tower variety. This is despite the fact that Australia has participated in PISA from its inception, and that it is managed for the Organisation for Economic Cooperation and Development (OECD) by a consortium led by the Australian Council for Educational Research. There are probably two reasons for this ignorance. First, as I’ve argued, Australian educators are generally profoundly uninterested in, if not hostile to, standardised external assessments. Secondly, Australia has done well in PISA. In 2003, the year in which PISA was last administered, only Finland significantly outperformed Australia in reading literacy. Australia ranked third in mathematical literacy and fourth in scientific literacy. In this context, good news is no news. There is no press mileage in announcing that Australia is doing well in international educational stakes— no opportunity to beat anyone about the head for our failings.

There is, nevertheless, one negative feature of the Australian results for reading literacy that is highlighted in the Australian report on the 2003 cycle of testing—Australian boys consistently perform worse in reading literacy than girls in every State and Territory; and although boys perform worse than girls in all participating countries, Australian boys perform, relative to girls, even worse on average than boys in other OECD countries. Why is this so?

The PISA reading literacy assessment allows us to look in detail at the areas in which Australian students show strengths and weaknesses, and in particular those in which boys and girls are differentiated in reading performance.

ILLUSTRATIVE TASKS FROM PISA READING LITERACY

Illustration 1

This task is based on a 400-word article from a student magazine about the importance of wearing good sports shoes. The text is classified as expository, and the task as one that requires retrieving information.

FEEL GOOD IN YOUR RUNNERS

For 14 years the Sports Medicine Centre of Lyon (France) has been studying the injuries of young sports players and sports professionals. The study has established that the best course is prevention … and good shoes. According to the article, why should sports shoes not be too rigid?

Illustration 2

This task is based on a 1700-word complete short story, ‘The Gift’. The story is about a woman trapped in a house on a flooded river, and how she deals with a panther on the verandah. The text is classified as a narrative, and the task as one that requires reflecting on the content of a text. Here is part of a conversation between two people who read ‘The Gift’:

Give evidence from the story to show how each of these speakers could justify their point of view.

Illustration 3

This task is based on a display from an archaeological atlas. The display gives information about rock art located near Lake Chad in Saharan Africa. The text is classified as graphical, and the task as one that requires forming a broad understanding.

This bar graph is based on the assumption that: A The animals in the rock art were present in the area at the time they were drawn. B The artists who drew the animals were highly skilled. C The artists who drew the animals were able to travel widely. D There was no attempt to domesticate the animals which were depicted in the rock art.

Contrary to Brenton Doecke’s claim that ‘no one interrogates the assumptions about literacy behind such tests’, the assumptions behind the PISA assessments have been rigorously interrogated and articulated in frameworks developed by international groups of experts. Like the other PISA domains, PISA reading literacy explicitly sets out to examine how well young people nearing the end of compulsory schooling are equipped for the literacy demands of their ensuing lives, whether those lives involve continuing study or entering the world of work; and, in either case, encompassing the educational, public, private and occupational demands of life in the 21st century.

To that end, the reading literacy framework covers not just functional reading (for example, reading a flight schedule or advice on preparing a job application) but also reading for personal growth and pleasure (for example, the reading of literature), and reading to inform effective social and political participation (for example, the reading of public argument and exposition). So much for the text types. The PISA reading framework also specifies a number of ‘aspects’ or processes of reading that describe how readers approach a text. Five aspects are defined: forming a broad understanding; retrieving information; developing an interpretation; reflecting on and evaluating the content of a text; and reflecting on and evaluating the form of a text. While the first three of these aspects focus on the reader’s ability to make meaning within the text, the last two deal with the reader’s capacity to bring to their reading their external experience, values and standards—their whole selves, more or less. High reading achievement as measured in PISA is contingent on the capacity to engage with all of these types of texts, and from all of these perspectives.

Australian boys performed relatively poorly on narrative and argumentative texts, and on tasks involving the aspects related to reflecting and evaluating. More pervasively, Australian students of both sexes performed poorly, relative to 15-year-olds in other English-speaking countries like Canada, England and Ireland, on reflecting and evaluating tasks, and relatively well on retrieving information tasks.

What kinds of conclusions might we draw about our education system and perhaps about our culture from these results? On the whole we’re doing pretty well, by international standards, but our strengths nationally are pragmatic and information-oriented; we are relatively unreflective and uncritical. And these descriptions apply particularly to Australian males—or at least, to young males. Might this characteristic also explain something about the national attitude to standardised testing?

Large hypotheses perhaps, but they are at least based on some empirical data—data that would not be available without a standardised assessment.

Only a fool would argue that all national and State assessments are beautifully conceived and constructed, and that every use made of them is wholesome. But standardised assessments should not be the butt of unreflective attack. They can and should provoke critical review of our education systems and our larger educational and national culture.

References

Doecke, B (2004). ‘Is anybody listening?’ in EQ Australia , issue 1.

Kirsch, I, et al (2002). Reading for Change: Performance & Engagement across Countries. Results from PISA 2000 , OECD, Paris.

Organisation for Economic Co-operation and Development (2004). Learning for Tomorrow’s World: First Results from PISA 2003 , OECD, Paris.

Sommer, P (2004). ‘Critical literacy: problem or opportunity?’ in EQ Australia , issue 1.

Thomson, S, Cresswell, J, and de Bortoli, L (2004).

Facing the Future: A Focus on Mathematical Literacy Among Australian 15-year-old Students in PISA 2003, ACER, Camberwell.

Juliette Mendelovits is a senior research fellow at the Australian Council for Educational Research and is the domain leader for PISA reading literacy.

The author owns the copyright in this article. For information related to the reuse of this work in any form please contact the publisher denise.quinn@curriculum.edu.au