How hard is the maths GCSE?

This week, there's been a lot of chatter about a tough question in a GCSE exam issued by Edexcel. The algebra of Hannah's orange and yellow sweets flummoxed a decent portion of the country's 16-year-olds. This question opens up a hoary old argument that is guaranteed to annoy teenagers: just how hard is the current GCSE? This is a surprisingly tough question to answer for a number of reasons. But we can certainly say that it is harder than its caricature in the popular press has it.

To answer it, it's worth a look at a fascinating new piece of work by Ofqual, the qualifications regulator, which got a group of mathematicians to grade GCSE maths questions on how easy they were from a mathematical perspective: 35 judges compared pairs of exam questions thousands of times to build a nuanced picture of which qualifications were more complex. It was commissioned because England is changing how it runs its GCSEs, so was designed to benchmark new exams and the old ones against one another and against foreign comparator exams for similarly aged young people.

This rather busy diagram shows the results:

In brief: each bar represents one maths exam. The higher up the black line in the middle is, the harder the median question. The top edge of the box shows where the hardest quarter of questions starts and the bottom edge shows where the easiest quarter ends. So a bigger box means that the exam covers a wide range of aptitude and a higher line means the middle question is harder.

The green bars are for foreign exams. The dark blue bars are current "higher" GCSE maths papers, which contain the questions designed to tell higher aptitude children apart. The light blue bars are the current "foundation" papers, which have questions designed to tell apart students at the lower-aptitude end. The red and orange bars show the same for the proposed post-reform GCSE papers.

To take a worked example, the right hand bar is green, meaning it is foreign (in that case, New Zealand). It has the hardest median question of all, since the black line in that column is highest of all. But since it is a squat column that does not spread far up or down, the exam does not cover much of the ability range.

What sort of exam is the maths GCSE?

This diagram clearly shows up the GCSE's strength as an exam: it distinguishes the ability of children at the very bottom of the attainment spectrum and children at the top. Just look at how wide a range of aptitude they cover. To do that, you have to look at two bars together - say, the Pearson Foundation paper which is the furthest to the left, and the Pearson Higher paper, which is about halfway up.

You can also see that the GCSE is particularly good at something that is important, but often gets little credit: distinguishing achievement at the bottom end. Indeed, Ofqual found that the testing found that GCSE maths contained the easiest questions of any test for 15-16 year-olds from a range of countries. These are questions like this one:

These sorts of questions often get cited in attacks on the GCSE as dumbing down. But these are questions that are telling apart students on course for a G from students who might get an F. In lots of other countries, these pupils would be doing another qualification - or none.

This matters: the Korean paper examined by Ofqual is tougher than ours, but is taken by only around 1% of students. The Japanese paper by about 30% of pupils. That makes it like the old English O-level, which surprisingly few students took. You can also see that the New Zealand exams, which are taken by everyone, only measure a narrow range of ability.

But having a framework that includes and measures most students well is a good thing, since schools have an incentive to pay attention to help everyone to make some progress. And remember that the GCSE papers do also contains tougher questions, like the notorious one from yesterday, which are designed to help us tell apart students competing for the higher grades.

Here it is:

The algebra here isn't too tough. The difficulty comes in seeing how to connect the probability to the rest of the problem. If it is any consolation to students who struggled with it, this requires a similar conceptual jump to a question that Ofqual's panel of mathematicians thought was among the hardest - also set by Edexcel. Again, it's a question that merges probability together with another concept.

Still, the Ofqual research does make an important criticism of the GCSE. The current GCSE is weaker than some other countries' exams at stretching our high-attaining students. Looking back to the diagram at the top, you can also see that the new GCSEs (in red and orange) will be a bit more demanding - but they will still not match the pure mathematical complexity of questions elsewhere.

So we can say from this that there is less room to test differences in conceptual grip among top-end students in the core GCSE, but is the exam easier? Not so fast.

First, it is worth noting that the toughest questions on this measure from around the world are not conceptually a lot harder than the top English questions. This, for example, was the toughest set of questions in the whole set. The maths involved in the second part of the Hannah's sweets question would be enough to solve the first, supposedly the hardest of them all. The margins are fine. (The first two are from New Zealand. The third is from the Netherlands.)

Second, just because a paper has more complex questions, it does not mean it is harder. On Ofqual's scoring system, a paper that covered very few topics but did so in great depth would score very highly. Having a wider range of content in the paper is not recognised. So the true picture is more nuanced. The new GCSE is bulking up on content, so Ofqual's measure may understate the scale of the change that is coming.

Third, remember that variation in a paper's contents from year to year make it harder. When I was an undergraduate studying history, I always did history of art questions because no-one else ever did them and so the exam questions never changed. And an objectively complex question becomes easy if it's asked every year. (The old O-level had that problem in a big way.)

So what can we say about the maths GCSE from all this? It is a bit less complex at the top end than some other exams - probably a consequence of its great strength at the bottom end. But there's a lot more to whether an exam is hard than complexity. And there's a lot more to how good an exam is than how hard it is.