by Andreas Schleicher
Director, OECD Directorate for Education and Skills
Claudia Costin
Senior Director, Education Global Practice, World Bank
How do large-scale student assessments, like PISA, actually work? What are the key ingredients that are necessary to produce a reliable, policy relevant assessment of what children and young people know and can do with what they know? A new report commissioned by the OECD and the World Bank offers a behind-the-scenes look at how some of the largest of these assessments are developed and implemented, particularly in developing countries.
A Review of International Large-Scale Assessments in Education: Assessing Component Skills and Collecting Contextual Data provides an overview of the main international, regional, national and household-based large-scale assessments of learning. The report shows how the major large-scale assessments have several things in common that contribute to their reliability and relevance. For example, they each produce clear frameworks to describe the philosophy, content, test design and response styles of their tests. These frameworks not only guide the creation of items (questions or tasks in a test paper) for the test, but also act as a way of communicating information about the assessment to the broader community.
The mode of delivery for most of the large-scale assessments is paper and pencil, but there is a shift towards computer-based assessment and this will undoubtedly be the main mode of delivery in the future as it increases efficiency and reduces data error. All of the assessments covered by the report collect contextual information that can be related to the test scores and help to inform policy choices. The reviewed surveys devote considerable time and resources to coder training and coding itself –this is the process of marking students’ responses with codes once tests are complete, including the steps taken to confirm that coding is being undertaken with acceptable reliability. In one or two cases, methods and approaches have been developed to include out-of-school children in learning assessments.
The report gives particular emphasis to learning from large-scale assessments in developing countries and makes recommendations in the following areas for the benefit of the OECD’s PISA for Development project: assessment frameworks; scoring; modes of delivery; collection of contextual information; methods and approaches to include out-of-school children in learning assessments; and analysis, reporting and use of the data collected.
The report also reveals some little known facts about the major large-scale assessments. Did you know that an assessment used in French-speaking mainly African countries uses questionnaires to collect information on whether students are working outside of school? Analysis of the results helps countries to determine whether working hinders students’ learning. And did you know that a large-scale assessment in Latin America routinely collects information on food, transportation, medical and clothing programmes and relates the data to student test scores? Or that a regional assessment in southern and eastern Africa finds that the active involvement of relevant government staff in research is one of the most important factors in converting analysis of the results of assessments into policies and changed practice?
Initially commissioned to provide recommendations for designing the PISA for Development project, the report is a valuable reference for policymakers, development organisations and other stakeholders with an interest in developing or participating in large-scale learning assessments.
Links:
A Review of International Large-Scale Assessments in Education: Assessing Component Skills and Collecting Contextual Data
The Experience of Middle-Income Countries Participating in PISA 2000-2015
Towards the development of contextual questionnaires for the PISA for development study
PISA for Development Technical Strand C: Incorporating out-of-school 15-year-olds in the assessment
PISA in Low and Middle Income Countries
For more on PISA for Development, visit: www.oecd.org/pisa/aboutpisa/pisafordevelopment.htm
Photo credit: © epicurean / iStockphoto
