In 1988 I was drafted onto a project to build a statistical moderation system for the Joint Matriculation Board, one of the UK's national examination boards. I was there because I knew JSP. I came from IBM mainframes, PL/I, IMS and DB2, so the platform was new to me: JSP-COBOL on VAX/VMS, running on MicroVAX machines1 that hummed like they were thinking.
Moderation2 itself was simple enough. Examiners are human. Some mark hard and some mark soft, and two students of equal ability can walk away with different marks depending on who happened to pick up their scripts. The system's job was to adjust those marks afterwards, smoothing out the strictness of examiners across the country without pretending that any one examiner was right and the others wrong. It was going to be applied to every GCSE subject, regardless.
Regardless was the word nobody examined too closely. The method assumed there would be enough marks to show a pattern, and for most subjects that was true. English Literature, Maths, Biology: the numbers behaved themselves. But the furthest outlier was Coastal Navigation, with four candidates in the whole of the UK. Four candidates don't give you a pattern to level against. Whatever adjustment the system produced would have been mostly noise.
As the project grew urgent it pulled us into overnight work, myself and Vince3 — another northerner, dry‑humoured, and given to asking the question nobody expects. We were trying to decide whether the method made sense for an exam with almost no data behind it. In the middle of the discussion he looked up from his terminal and said, half as a joke, "Where would we be if there were no such thing as a hypothetical question?"
It was the most useful thing said that night. Whether to apply the system to Coastal Navigation was above our pay grade, and I don't think anyone above us had thought about it. I still don't know how it was settled.
For four people in the country, it was never hypothetical.