Make font smaller  Make font larger

Winter 2005

The Assessment agenda

What do Australian students know about Asia?

Using a combined set of sub tests tailored to each State and Territory curriculum, PATRICK GRIFFIN and KERRY WOODS conducted a national survey of more than 7000 years 5 and 8 students from over 300 schools about their understanding of Asia. They reported in terms of a standards referenced framework.

SINCE 1993, SCHOOLS IN EACH AUSTRALIAN STATE AND TERRITORY have been implementing studies of Asia programs. In 1999, the Asia Education Foundation (AEF) and National Asian Languages and Studies in Australian Schools (NALSAS) Taskforce asked the University of Melbourne’s Assessment Research Centre to develop assessment instruments that would assist the Commonwealth and States to establish baseline measures of students’ knowledge and understanding in this study area.

The goal was to measure student proficiency at two formative stages of development—years 5 and 8. This involved development of two tests consistent with a generic content of studies of Asia. A major challenge for the researchers was development of measures that would not discriminate against students from any particular State or Territory, regardless of the curriculum implemented in that State or Territory. This article explains the process of developing two assessment instruments that could be used nationally and then statistically equated. This meant that students were assessed only against items relevant to their local curriculum, but their test results could be meaningfully compared.

Project establishment

An analysis of curriculum materials in national profiles, State curriculum guidelines and syllabus statements, and the NALSAS and AEF materials guided which areas were included in tests. A mapping exercise identified elements of studies of Asia outcomes that were considered nationally important; these formed blueprints for test design and determined selection of prompt materials. The task design specifications are presented in the table below, and provide a broad indication of the scope of the tests.

Task design specifications
GoalSOSE/HSIEEnglishArtsItems
Developing concepts43411
Challenging stereotypes54312
Contemporary issues54312
Contributions by Asia54312
Links with Asia54413
Total24181860

Subject area specialists from each State and Territory worked in collaboration to write test items. It was agreed each test should have about 60 questions because of the amount of time available for testing in schools. A set of 15 items was specified as an overlap between the year 5 and year 8 tests, to enable the tests to be linked between year levels. Consultation with State and Territory representatives then identified a sub set of test items most related to that State or Territory’s curriculum. These were the ‘state essential items’. Students were only assessed using the items identified as essential to their State or Territory.

Achievement measures

In total, 3290 year 5 students and 3889 year 8 students took part in the study. A weighting procedure was used to ensure the relative contribution to national estimates was in proportion to the relative numbers of students within each State or Territory.

Using item response modelling, items and students were simultaneously mapped onto the same underlying scale. Test items were ordered according to difficulty, and then audited to describe achievement levels. Seven clusters or knowledge levels were interpreted. A panel of subject specialists joined the project to offer substantive interpretation of these levels. Figure 1 shows the proportion of year 5 and year 8 students at each knowledge level.

figure1

The scale is cumulative and probabilistic, so that students who could correctly answer questions at higher levels of complexity were also more than likely to correctly answer questions at lower levels of complexity. Analysis of knowledge levels, combined with information about item characteristics, indicated that more than half of the year 5 students showed a basic knowledge of Asian languages, art and lifestyles and contextualised this knowledge within regional or national boundaries, but there was a good deal of variation between students. For example, almost all of the year 5 students successfully answered items related to Asian food and martial arts (level 1). By contrast, students found questions related to geographic location, such as recognising specific countries of Asia on a map, rather more difficult, although a majority (60%) answered most of these questions correctly. Similarly, 62% knew that Indonesia was a near neighbour of Australia (a level 4 question) but only 46% could identify Indonesia on a map (level 5). Many students had difficulty answering questions about East Timor (level 6); slightly less than half correctly answered questions about samurai, Buddhism, Tet, Islam and the Chinese invention of fireworks (levels 5 and 6). Figure 2 shows an item that year 5 students at level 3 or more on the knowledge scale were likely to answer correctly.

figure2

In year 8, more than half of the students had an understanding of historic and contemporary events and their influence on Asia and Australia. Many of the year 8 students showed an understanding of the national and regional significance of festivals, celebrations and traditions in art, text, language and theatre. However, as with the year 5 students, there was a good deal of variation between year 8 students in knowledge and understanding.

Students at the lowest achievement levels (level 1) recognised the meaning of the Yin Yang symbol (see figure 3) and also knew the name of the symbol (level 2). These students also successfully answered questions about the significance of traditional dress (level 2), but could not extend beyond this basic level of knowledge.

figure3

Year 8 students at level 3 on the knowledge scale were likely to correctly answer questions about the Indian god Ganesh, the White Australia Policy and immigration, and Mahatma Gandhi. As with year 5 students, however, many year 8 students had difficulty identifying Asian countries on a map. Indeed, less than half recognised Indonesia (level 5), although approximately 60% were able to correctly name China and Thailand (level 4). In addition, almost half of the year 8 students correctly answered questions about political leaders such as Ho Chi Minh and Mao Zedong, the British involvement in Hong Kong, and could supply the name of Islamic mosques (levels 5 and 6). Relatively few knew the purpose for which the Taj Mahal was constructed, although a majority could name the structure. Less than 25% of students correctly answered questions about the Indian film industry (Bollywood) or the most popular religion in Indonesia (level 7).

In summary, item response modelling was used to map test items onto a single underlying scale so that years 5 and 8 students from different States and Territories could be meaningfully compared in terms of their knowledge of Asia. The project provided a range of benefits to schools, including an opportunity to assist education systems in monitoring student outcomes in areas related to studies of Asia. Standardised instruments, developed as a result of the project, have the potential to improve quality of assessment in these learning areas, and to assist in the planning and resourcing of studies of Asia programs.

author picture Patrick Griffin holds the Chair of Education (Assessment) at the University of Melbourne.
author picture Kerry Woods is a research fellow at the Assessment Research Centre.

top