A study lands and the headline writes itself: school phone bans do not improve test scores or attendance. Depending on which side you were on, this is either vindication or a reason to distrust the researchers.
It is neither, and the reason is worth more than the finding.
What was the apparatus built to settle?
Ibn al-Haytham, working in Cairo around a thousand years ago, made one methodological demand that still does most of the work in any argument about evidence. A dispute about how something behaves is settled by building an arrangement that makes the disagreement visible — a dark chamber, an aperture, a vessel of water — and the first question about any such arrangement is what it would have to show before one of the rival accounts became untenable.
That question is doing something more specific than asking whether a study is good. It asks whether the thing built is capable of deciding the dispute that people are actually having.
So: what was the dispute?
Almost nobody argued for these restrictions on the grounds that they would raise test scores. Read the case as it was made — by teachers, by legislators, by the parents surveyed — and the claims were about attention in the room, about students talking to each other at lunch, about the anxiety of being filmed by a classmate, about a teacher spending the first four minutes of a lesson on confiscation. Those are the propositions on the table.
A test-score regression is a well-built instrument that answers a different question. Aggregate attainment is affected by a very long list of things, moves slowly, and is measured with instruments coarse enough that an effect on a single classroom behaviour would have to be enormous to appear. A null result there is close to what you would predict even if every claim the supporters made were true.
That is not a criticism of the study. It is an observation that a well-executed measurement of one quantity is not evidence about another, and that a great deal of public argument consists of exactly that substitution.
What would settle it
The honest version of his demand is uncomfortable for both sides, which is how you know it is working.
If your claim is about attention in class, the apparatus is observation in classrooms — time on task, interruptions per lesson, before and after, with observers who do not know which condition they are in. If your claim is about social contact, it is structured observation at break, not a survey asking students whether they feel more connected. If your claim is about being filmed, it is incident reports.
Each of those is buildable. None is as cheap as a regression on data somebody already holds, which is precisely why the cheap instrument keeps standing in for the expensive one.
And the demand cuts the other way just as hard. If a supporter of the policy would not have abandoned it on any test-score result, then they should stop citing learning benefits in the argument for it, because they have just revealed that no measurement of learning was ever going to move them.
Watch out for
The failure of this method is that stating the demand is much easier than meeting it, and its author is the evidence.
He described apparatus far more often than he reported actual readings. A. Mark Smith's assessment of the Optics is that several of the set-ups as described could not have produced the results claimed for them, and that some function as demonstrations of a conclusion already held rather than as tests of it — his refraction figures track values already tabulated in Ptolemy rather than any fresh measurement. The man who insisted disputes be settled by construction repeatedly published the construction and not the numbers.
His doubt also points outward and never at its own warrant. He had already written a work restating Ptolemy's models without a murmur before writing the one that dismantles them.
So treat the demand as a discipline rather than as a weapon. Anyone who invokes it to dismiss a study they dislike, without saying what result would have persuaded them, is doing the thing he did.
Answer this next
Whatever you believe about phones in schools, write down the observation that would have changed your mind — before you read the next study.
If nothing would have, the question you have been arguing about is not an empirical one, and it should stop being conducted as though it were.
