The Sour Grapes of Pisa

November 28, 2013
Still standing.

Still standing.


The new Pisa 2012 will be released on Tuesday, which for those who are unfamiliar with it is a recurrent survey on the performance of schoolchildren from all over the world. The winners in this survey tend to be the same over the years: various Chinese populations (Shanghai, Hong Kong and Singapore), Finland, Canada, Australia, Japan and South Korea.

A high rank is generally interpreted as the result of a good policy and a low rank will usually create headlines demanding reforms.

The Pisa Hall of Shame

At the bottom of the order we find poor and often Muslim countries like Kyrgyzstan, Azerbaijan, Peru, Panama, Qatar and Albania. But besides rich countries at the top and poor at the bottom, there is also the phenomenon of over- and underachievers, poor countries whose children perform well and vice versa. This measure is more interesting since it indicates a failed education policy or other factors that may have been overlooked.

Some of the worst underachievers (excluding tax havens and small oil countries) are USA, UK, Austria, Germany, Denmark and Sweden. How do people in these countries respond to the results of the survey?

America: Self-Criticism and Fear of China

When commenting on the results American media have mainly been comparing themselves with China and been surprisingly self-critical, as for instance from Stacie Nevadomski in the Huff Post shows,

“The truth, the real news, is that there is no news here. These results should be no surprise. The long slide in American student performance relative to global peers has been a constant drumbeat, paralleling the domestic failures of our schools shown in Waiting for ‘Superman’.”

Or education secretary Arne Duncan,

“The findings, I have to admit, show that the United States needs to urgently accelerate student learning to try to remain competitive in the knowledge economy of the 21st century.”

James Fallows in The Atlantic agrees but adds that that Shanghai, the winner of the Pisa 2009, isn’t representative of the whole of China – which is correct; neither is Hong Kong or Singapore who also rank at the very top. These are all elite populations. America scored better against the other Chinese regions of Macao and Taiwan and it would probably do even better compared to all of China. Although those who are familiar with unpublished results from other parts of China claim they are very respectable.

Regardless of how well America compares to China, it’s still a fact that 13 countries score better than America and all have significantly lower GDP per capita. Maybe that would be a more constructive focus.

European Skepticism

A more disturbing reaction has come from some of the European underachievers. Recently, the largest newspaper in Sweden, Dagens Nyheter (Today’s News), has featured an article about the upcoming Pisa 2012, with the headline “Several Countries Cheated with School Results”, suggesting that countries like Italy, Slovenia and the United Arab Emirates has falsified their results. The article is based on an unpublished study by German and Canadian sociology professors Jörg Blasius and Victor Thiessen. “The result means that the credibility of the Pisa survey can be questioned,” says Blasius.

This story is also getting attention in Denmark, another underachiever, where one of the major papers, Berlingske Tidende has an article about it. The article includes other criticism as well, mainly that of Svend Kreiner, a statistics professor at the University of Copenhagen. Kreiner has analysed earlier results. He is critical of how a lot of questions are omitted for some countries but included for others. He claims the methods of scoring are so arbitrary Denmark could be ranked second or 42th depending on arbitrary tweaks in the evaluation. In the article, president of the Danish Teachers Association, Anders Bondo Christensen, says it’s time to scrap the survey altogether.

In the UK (also an underachiever), there is a similar discussion on the TES educational community. In an article, TES’s William Stewart writes,

“Politicians worldwide, such as England’s education secretary Michael Gove, have based their case for sweeping, controversial reforms on the fact that their countries’ Pisa rankings have “plummeted”. Meanwhile, top-ranked success stories such as Finland have become international bywords for educational excellence, with other ambitious countries queuing up to see how they have managed it.”


“But what if there are “serious problems” with the Pisa data? What if the statistical techniques used to compile it are “utterly wrong” and based on a ‘profound conceptual error’? Suppose the whole idea of being able to accurately rank such diverse education systems is ‘meaningless’, ‘madness’?”

Sour Grapes?

However, fact is the alleged cheating is only concerned with follow-up questions to principals that have been found to be largely identical in many cases. It doesn’t concern the performance of the schoolchildren. It hasn’t even been established if it is actual fraud designed to make the countries in question look better or if it’s just a matter of laziness or even the fact that some principals are heads of more than one school.

Also for Kreiner’s analysis, Pisa’s own statistician, Andreas Schleicher, questions it on grounds that Kreiner is using a very small part of the data in spite of having access to all of it. He also questions the methods Kreiner used and suggests that they our outdated. As a response to alleged cherry picking, Kreiner replies by accusing Pisa/Schleicher of doing similar things. To me, that sort of rhetoric doesn’t exactly increase his credibility.

It’s not easy for a non-expert to make any sense of this, but I have to say that there is something disconcerting with the fact that Svend Kreiner is being awarded a prize for his critique while no one in Danish press is asking the questions that Schleicher’s comment raises. Is everyone in Denmark so familiar with statistics that it’s a non-issue? And big headlines about cheating even though it hasn’t been established?

Alternative Explanations

Rather than blaming the statistics, there could be other things behind why some countries underachieve. The most obvious thing would be changes in national IQs.

The Pisa survey (and similar tests) correlates strongly to intelligence tests; so much in fact that it actually is an intelligence test although it’s rarely referred to as such. This explains a lot of the rank order, because we know that intelligence is highly heritable and resistant to external forces – like education policies. Smart people like the Chinese are going to rank at the top and less smart people like Ugandans are going to be somewhere at the bottom. This is also a reason to be skeptical of the European sour grapes skepticism I mentioned earlier. If there was something seriously wrong with the Pisa it wouldn’t correlate so much with similar tests.

But intelligence alone can’t explain under- and overachievers. If we look at the latest national IQ estimates, the underachievers score like this,

Austria 99, UK 99.1, USA 97.5, Germany 98.8, Denmark 97.2, Sweden 98.6,

and, the three overachievers score like this,

Finland 100.9, Estonia 99.7 and Poland 96.1.

There is not much difference; the averages for these groups are 98.4 and 98.9. But maybe this snapshot disguises a trend in which underachievers are on the way down and vice versa?


I would suggest that this is the case, and that the reason for this is immigration. East Asian countries don’t have much immigration to speak of, but in Europe there has been a varying influx of people in recent years, especially from Muslim countries. The national IQs in these countries are usually around 85 so Western countries that receive a lot of these immigrants should see a larger decline in national IQ averages than other countries. If we look at PEW’s survey of Muslims in Europe, we can make a comparison between over- and underachievers. The most striking overachievers are Estonia, Poland and Finland, countries that all have extremely small Muslim minorities making up 0.1, 0.1 and 0.8 percent of the population respectively. Compare that with the figures for the underachievers Austria 5.7, UK 4.6, Denmark 4.1, Germany 5.0 and Sweden 4.9. Many immigrants are very young children who will take the Pisa survey in years to come or are taking it now but have yet to become adults and have an effect on the economy. Since the Pisa survey is just an intelligence test for children they simply reflect the influx of young and low IQ people. Underachievers have a larger influx so they score worse than you’d expect from the current national IQs and wealth because the effects on these metrics will kick in some years in the future. And overachievers are just maintaining their national IQs and consequently rising in rank since the rank order is relative.

So the way to improve the scores is not to reform the education system but to change the immigration policy.

So, Any Bets for Tuesday?

If I was to guess I would base it solely on national IQs, immigration and introversion scores, although that last one is a bit speculative. This would lead me to the safe bet that East Asians will stay at the top and no real low IQ countries will surprise anyone with a high rank. Judging by the immigration projections from PEW, Eastern Europe looks like it could be on the rise, or at least maintaining positions, although Russia and Bulgaria look problematic. The real winners here are probably small to medium sized countries that are relatively stable, like Estonia, Poland, Slovenia, Croatia and Hungary. Western Europe will show a downward trend, especially for countries that are increasing their share of the Muslim population from an already high level, like the UK, Austria, Sweden, Belgium and France.

But whatever happens, you can be certain that many people in the underachieving countries will keep blaming the test. Because changing your view on human nature and society is hard work and shooting the messenger is easy.

For more details about the Pisa survey, check out Steve Sailers blog which features several interesting posts on this subject.


%d bloggers like this: