Exams 2022 - a call to action

I remember, vividly, my English teacher. His name was Mr Shone, and he taught me from age 11 to GCE (yes, GCE – I show my age!). Not only did he teach me English, he taught me how to think, and I shall never forget his introducing the class to Straight and Crooked Thinking by Robert Thouless. Nor shall I forget, during the ensuing discussions, his emphasising that “You must always remember that much writing is neither neutral nor dispassionate. Rather, much writing – and much good writing too – has the objective of persuading you to a particular point of view. And in doing so, the author is very likely to be highly selective, deliberately choosing the ‘right’ evidence to cite – and deliberately omitting material that might direct your attention the other way.”

Wise words indeed. And ever since, I’ve always tried to identify what’s missing, and whether what’s missing might be important.

So, with that in mind, let me be quite explicit that, yes, this blog very much has the objective of seeking to persuade you to a particular point of view. But I trust I am not misleadingly omitting important information; rather, I am drawing on what others have omitted to make my point.

My theme, unsurprisingly, is the return of formal exams, and my initial sources are Ofqual’s July consultation, the transcript of the Education Select Committee hearing of 7^th September, some announcements by the now-departed Nick Gibb and Gavin Williamson, and the recent writings of ASCL’s General Secretary Geoff Barton, former UCAS Chief Executive Mary Curnock Cook, and Sam Freedman, a former senior policy adviser to Michael Gove at the DfE. 

Yes, dealing fairly with so-called ‘learning loss’ is critical, and having a wise contingency plan, sensible. These are – we all hope – transitory, and, with the passage of time, will recede into history. But exams are likely to be with us for many years, and so their structure, content and administration is a matter of much importance. 

All the sources I’ve mentioned agree that the current BIG QUESTION is the ‘standard’, as measured in terms of the distribution of grades over the mark range, and as determined by policies defining what percentage of students are allowed each grade. This in turn splits into two sub-questions: should the standard be that of 2019, 2020 or 2021? And, if 2019, should that standard be applied at once in 2022, or more gradually over the next two or perhaps three years? Unsurprisingly, the sources disagree, and one source – Geoff Barton – has changed his mind, and is now leading ASCL in advocating that 2022 exams should follow the 2019 distribution. I’ll return to this shortly…

…but before I do so, and to avoid falling into the trap of ‘deliberate omission’, I must mention some other sources, notably EDSK, Rethinking Assessment, and the National Baccalaureate Trust, who are advocating a variety of different educational concepts, with the common ground of reforming the assessment system in its entirety, including, for example, scrapping GCSEs altogether. Their answers to the grading questions “2021, 2020, or 2019? And if 2019, how quickly?” are “none of the above” – with which I personally have much sympathy. The reformers, however, have the problem that those who, until very recently, held power showed no signs of listening, and in this context, the views of Nadhim Zahawi and Robin Walker are as yet unknown. Whether or not there is any governmental appetite for reform therefore remains to be seen…

…and so let me revert to the standard issue – 2021, 2020, or 2019? At which point, I will draw on Mr Shone’s memorable teaching, and ask “what is missing from all these sources?” And in answering that question, I rapidly come to the conclusion that whether the standard should return to 2021, 2020, 2019 – or indeed 1999 (when, at A level, 17.5% of students were awarded grade A, compared to 19.1% A*, 44.8% A*/A, in 2021) – is a red herring, a distraction.

For what’s missing from all the sources is any mention of the fact that exam grades as ‘awarded’ during the 2010s (certainly), and the 2000s (possibly), have all been hugely unreliable. And if grades are to be determined in the same way from 2022 onwards (or rather, from October 2021 onwards) as they were in the 2010s, then future grades will be hugely unreliable too, regardless of whether the grade boundaries reflect the distributions of 2021, 2020, 2019 or 1999. What’s the use of hugely unreliable grades, whatever the standard?

I’d better elaborate on “hugely unreliable”.

Here are two statements from Ofqual, both made at the Select Committee hearing held, after last year’s fiasco, on 2^nd September 2020. The first was given by Ofqual’s then Acting Chief Regulator, Dame Glenys Stacey, who acknowledged that exam grades “are reliable to one grade either way”. That sounds quite reassuring; but it doesn’t take too long for the penny to drop that this statement is the same as “an A level certificate showing ABB actually means ‘any set of grades from A*AA to BCC, but no one knows which’”. Perhaps Dame Glenys would be so kind as to explain that to any of your students who failed to win the university-place-of-their-dreams by missing by just one grade.

The second admission was made by Dr Michelle Meadows – at that time, and still, Ofqual’s Director for Strategy, Risk and Research – who “took solace that 98% of A level grades, and 96% of GCSE grades, are accurate plus or minus one grade”. This is both more startling (“took solace”? really?) and more informative than the statement made by Dame Glenys, for it implies that 2% of A level grades are at least two grades adrift, and 4% of GCSE grades. The actual numbers are worth bearing in mind: about 15,000 A level, and 200,000 GCSE, grades, all at least two grades adrift without right of appeal. Might those have been grades for some of your students?

Both these statements are oblique, and – once again I return to Mr Shone – have something important missing. “98% of A level grades are accurate plus or minus one grade” leaves many possibilities, of which two are (1) that 97% of grades are right, and only 1% one grade adrift, or (2) that 1% of grades are right and 97% are one grade adrift. Officially, Ofqual have never resolved that. But they have published data enabling that to be unscrambled, albeit only after some leg-work. The resolution is here, with the take-home message that, on average across all subjects, about 1 grade in every 4 is wrong – that’s about 1.5 million wrong grades every year – with (large) variations by subject (about 4 Maths grades in every 100 are wrong, and about 46 in every 100 History grades), and by mark within subject (so, for example, in all subjects, any script marked at, or very close to, any grade boundary has about a 50/50 chance of being awarded the right grade: you might as well toss a coin…).

So there it is. Amongst all the talk about grade inflation, grade distributions, grade boundaries and all the rest, no one, but no one, has said “STOP!!!” No one has said “The most fundamental issue to address must be to ensure that all assessments are fully reliable and trustworthy!”

For if we examine how to make assessments reliable and trustworthy, something remarkable happens. To achieve reliability, the policy to determine assessments has to change. And in so doing, the grade inflation problem goes away. In fact, avid readers of Blog 6 may be thinking that this rings a distant bell. As indeed it might, for on 7^th December 2020, the SFCA kindly published a blog describing how to deliver assessments that are fully reliable and trustworthy. Very briefly, instead of a certificate showing a grade (with all the implications as regards boundaries, distributions and the rest), the certificate shows, for each subject, the candidate’s mark (say, 65, on a mark scale from 0 to 100) and also a measure of that subject’s ‘fuzziness’ (say, 4 marks), a number that recognises the reality that a different examiner might, legitimately, have given the same script a different mark (4 marks indicating that, for this subject, it is very unlikely that any other mark would have been higher than 65 + 4 = 69, or less than 65 – 4 = 61).

Grades are no longer used, and so there is need to determine grade boundaries. Which solves two problems simultaneously: how to deliver reliable assessments, and how to ‘reset’ after the pandemic: since this assessment is so different, comparisons to any previous year have no meaning. And – to return to the reformers – this could also be the first step along the reform path, a pragmatic and highly valuable step that can be taken at once.

As I have mentioned several times, any discussion of the importance of making assessments reliable and trustworthy is currently ‘missing’ – no one is talking about it. And if that continues, then, when exams return, we’ll be back to 1 grade in every 4 being wrong, without recourse to appeal.

So perhaps ‘someone’ should be talking about it, stressing its importance, lobbying for a solution to be implemented.

Who is that ‘someone’? Is it you? If everyone who reads this says “well, that’s interesting, but…”, then nothing will happen. And if, for example, you’re a History teacher, about one half of all your students will end up with the wrong grade, year after year after year. 

But if everyone who reads this actually does something, then perhaps the new Ministers, Nadhim Zahawi and Robin Walker, might listen. And maybe something beneficial might actually happen…

Dennis Sherwood runs Silver Bullet Machine, a boutique consultancy firm, and is an education campaigner.