Beyond borders: What really matters when comparing student performance across countries

Whenever an American or European wins an Olympic gold medal, we cheer him or her as a hero. When a Chinese does, the first reflex seems to be suspicion, perhaps because we’ve become accustomed to always winning. There seem to be parallels to this in international education assessments. Very soon after results from PISA 2018 showed that the four Chinese provinces/municipalities that took part in PISA outperformed every education system in the Western world, those results were put into question.

Some have criticised the principle that, for China and India, PISA allows individual provinces, rather than the whole country, to participate, and that the participating Chinese provinces and municipalities (Beijing, Shanghai, Jiangsu and Zhejiang) are more economically developed than most other areas in China. That is certainly true, and was taken into account in the interpretation of the results (it’s also why the OECD points out in its reports that the results should not be interpreted as representing the whole of China).

But it is far from unusual. Indeed, Tom Loveless, an education policy expert and the first who made this point, was a member of the Steering Committee of the International Association of Educational Achievement (IEA) when OECD countries like Belgium and the United Kingdom participated mainly with their most economically advanced regions in the IEA’s international assessments, like the Trends in International Mathematics and Science Study (TIMSS). The partial participation of countries is longstanding practice in OECD international surveys too. For example, only high performer Alberta, in Canada, participates in the OECD Teaching and Learning International Survey (TALIS) and only high performers Flanders in Belgium, Jakarta in Indonesia, and England and Northern Ireland, both in the United Kingdom, participate in the Programme for the International Assessment of Adult Competencies (PIAAC).

What is striking is that those who now cry foul with China had never raised any concern about the partial participation of countries in the Western world. The reason for that is simple. While there can be technical, administrative or political considerations behind the limited geographic coverage of countries in PISA, each entity that takes part offers additional points of comparisons, with new insights and inspiration for peer learning and collaboration. Certainly, I have learned much from the education systems of England in the United Kingdom and Flanders in Belgium. In fact, averaging out the many interesting differences between the regions in Belgium to some national figure would blur some of the most relevant policy insights.

In another article, Mark Schneider, Director of the Institute of Education Sciences at the US Department of Education, claims that the highest-performing state in the United States, Massachusetts, would score among the world’s leaders. But when Massachusetts took part in PISA 2012 and 2015 (and, by the way, the state of Alabama did not) it performed more like Germany or Viet Nam in the 2012 PISA mathematics assessments – quite well, but still around 100 score points below Shanghai (a gap as wide as that between Massachusetts and Mexico).

The argument that I find hardest to comprehend goes like this: We can learn from high performer Estonia, because Estonia is a country, but we can’t learn from high-performing provinces/municipalities like Beijing, Shanghai, Jiangsu and Zhejiang, because they do not represent national entities.

I just came back from Estonia this week. I have always been impressed by education in that country; but the four Chinese provinces/municipalities that participated in PISA 2018 have a combined population of 180 million – that is over 130 times larger than the population of Estonia – and they have been able to deliver high-quality education at a much larger scale. Ignoring their results because of political borders or ideologies seems such a big loss. I have learned a lot from education in these provinces; their capacity to invest into high-quality teachers, to treat their teachers as true professionals, and to attract talented teachers into challenging classrooms is certainly something the world can learn from.

When Shanghai participated in PISA 2012 as the first Chinese municipality to do so, the strong results in mathematics generated a lot of interest. England’s former Parliamentary Under-Secretary of State at the Department for Education, Liz Truss, a former mathematics teacher, went to Shanghai and was impressed by the mathematics teaching that she observed, and by the teacher-to-teacher and school-to-school programmes in the municipality. She worked with the Chinese to create an exchange programme for teachers between China and England. Some 50 English-speaking mathematics teachers from China were deployed to more than 30 maths hubs in England. They showed the teaching methods they use, including teaching to the top and helping struggling students one-on-one. They gave daily mathematics lessons, homework and feedback. The Chinese teachers were also running masterclasses for local schools and provided subject-specific, on-the-job teacher education. In turn, leading English mathematics teachers from each of the maths hubs went to work in schools in China. The programme attracted considerable attention in both countries, showing how much teachers can and want to learn from other cultures if they are given the opportunities to do so – and if we dare to pull down ideological walls.

What I find most interesting in all of this is that, when I reviewed what was published in China about PISA since the results were announced last week, most of the articles focused on the weaknesses in Chinese education that the PISA results had made apparent, such as the comparatively poor social and emotional outcomes among Chinese students. The articles also noted China’s desire to improve these outcomes by, among other things, learning from other countries.

A consistent effort to look outward and incorporate the results of that learning into policy and practice seems to be a common denominator among high-performing education systems. This may be a key distinction between the countries that will make progress in education and those that will not. The difference may be between those education systems that feel threatened by alternative ways of thinking and those that are open to the world, and ready to learn from and with the world’s education leaders. The world is indifferent to tradition and past reputations, unforgiving of frailty and ignorant of custom or practice. Success will go to those individuals and nations that are swift to adapt, slow to complain and open to change. The OECD stands ready to help citizens meet the challenges of building high-quality and responsive education systems.

For a conversation with Andreas Schleicher about PISA’s role in global education, listen to our recent TopClass podcast.

Photo credit: @Shutterstock