Everyday mechanisms

This piece was initially published on the Matthew's EduContrarian blog here. 

People feel they understand complex phenomena with far greater precision, coherence, and depth than they really do; they are subject to an illusion – an illusion of explanatory depth.

Rozenblit and Keil, 2002

How does a toilet work?

I am confident that I have a fairly good idea. If asked, I would rate my confidence at about 5 on a 7 point scale. I know that you push a lever, or pull a chain, and water floods the bowl. I know that if I look in the cistern I will see the water level falling. I will see a ballcock attached to an arm. As the water level falls, the ballcock sinks and lifts the arm, causing…

And at about that point I start to get a little fuzzy. What exactly is the mechanism by which the water is released and the cistern is refilled? If this system went wrong and I had to fix it, would I be able to easily do so? Or would I scroll through YouTube help videos, or give up and call a plumber?

Having tried to describe the mechanism, my confidence falls. Perhaps I don’t understand the mechanism with the level of precision that I assumed I did.

Psychologists Leonid Rozenblit and Frank Keil asked a number of people to explain how a toilet works. They asked the same about a speedometer, a zipper, a piano key, a cylinder lock, a helicopter, a quartz watch, and a sewing machine. The participants in their study were asked to write a detailed explanation of each, with diagrams. Next, they were asked diagnostic questions. Lastly, they were then given an expert’s description of the causal mechanisms for each item. They rated their level of confidence on a 7 point scale after each stage of the experiment.

The results of this study show the fall in confidence of participants as they are forced to confront the gaps in the weaknesses of their folk theories about how everyday items work, then the increasing confidence when presented with expert descriptions.

Rozenblit and Keil called the effect revealed by these studies the illusion of explanatory depth.

Most people feel they understand the world with far greater detail, coherence, and depth than they really do… knowledge of complex causal relations is particularly susceptible to illusions of understanding.

In a previous post, I wrote about a big red button on a wall. Despite not knowing what this button does – what mechanism sits behind it – the temptation to push it is great. This button, I argued, is analogous to the urge we feel to take action. To do something!

But what if we believe that the mechanism triggered by the button is known to us? By pressing it, we have an expectation of what will happen. Might we be prone to our own illusions of explanatory depth?

Imagine we set up our own experiment in schools in which we ask participants to describe the everyday mechanisms upon which the functioning of the school relies. What do we know of the causal mechanisms which underpin the settling of a class, the modelling of a concept, the comprehension of a text, providing constructive feedback on a pupil’s work, or scaffolding a task such that it is accessible to the struggling pupil. The novice will observe the trigger and the result – the teacher’s raised hand that quickly silences the class, or the instruction which leads to fluid completion of a task. They know that if you pull the handle, the toilet flushes; but little beyond this. They know the what, not the why.

I spent a fascinating hour with a primary teacher after a researchED conference recently learning about the intricacies of phonics and number bonds. The mechanisms of the mind and of the instructional process were set out in all of their complexity. My ignorance was laid bare. I had what I believed to be a basic understanding of both, but quickly discovered how shallow and vague my insights were.

I would suggest that our illusion of explanatory depth affects not only our knowledge of the regularities of our schools but also our attempts to improve them. We may adopt a reading program which has pedigree in improving reading fluency, but do we know precisely how this effect is achieved?

Dr Sam Sims and Harry Fletcher-Wood make this point in the recently published systematic review of professional development conducted for the EEF. They seek to understand the causal mechanisms which underpin the most impactful professional development programs – the active ingredients – and caution against blind adoption of such programs without acting to ensure that the causal mechanisms required to ‘make the cake rise’ are protected as the program is necessarily adapted to the school’s context and needs.

But school improvement efforts are so often characterised by the import of ‘good ideas’, of proven interventions, or programs with a known ‘effect size’, without proper consideration of the causal process by which these fixes have their supposed effects. ‘It worked’ is so often assumed to mean ‘it will work’. We might criticise the simplistic notion of ‘effect sizes’ which are used as a signal of desirability by organisations such as the EEF, or we might blame the school leaders who naively grasp at ‘evidence-based approaches’, or copy superficially successful practices they see in other schools without questioning what it is that actually caused impact, or whether this is replicable in their own school. However, these behaviours are symptomatic of a more general ill, that is the custom and practice around school improvement. Like the souped-up engine of a boy-racer, we look for add-ons to supercharge our schools rather than confronting the sometimes dull and thankless task of servicing the existing machinery with care and attention, fine tuning it to achieve efficiency and performance.

None of this is to say that we shouldn’t import ideas or adopt new processes, just that we should pay careful attention to the mechanisms which are claimed to bring the desired effect. The art of school improvement is the art of implementation. I wrote about this here. As with a malfunctioning toilet, sometimes I need the nitty-gritty expertise of the plumber, not the grand designs of the engineer.

Education policy often suffers from the same lack of attention to mechanism. In our upcoming book, Becky, Ben and I term such policies ‘mandated miracles’: policies which do not contain the instructions for how to achieve the desired effects. Policy makers are as prone to an illusion that they understand the workings of the education system as the rest of us, perhaps more so as they view it from a distance.

What can we do to shatter our illusions? That is a question I will return to, as it may be key to securing better school improvement. In the meantime, I will talk to as many knowledgeable people as I can about the intricacies and complexity of schools. I am under no illusion that I have much still to learn.

Matthew Evans writes and blogs about education and is a secondary school headteacher.