The Case of the Margin of Error

COLD OPEN

TRACY (V.O.)Nobody lied to me. Nobody held a thing back.

It still took me eighteen months to hear what they said.

— music sting —

ANNOUNCERWelcome to Adverse Events. Transcribed stories from the case files of Dr. Tracy Grant, who writes the protocol, and Connie Chase, who makes it all happen.

ANNOUNCERThe story you are about to hear is fiction. Any resemblance to your last study is purely coincidental and entirely predictable.

ANNOUNCERThe Case of the Margin of Error.

SCENE ONE — THE OFFICE

a lamp chain pulled · a ream of paper set down heavy · a chair taking weight

TRACY (V.O.)The MERIDIAN trial. Ninety-four people, twelve weeks apiece, eighteen months to get them all through the door, and every one of them signed up because their knees had quit letting them do things.

One primary outcome. A pain score, zero to ten, once a week. That number was the whole case, because a trial that names one outcome has already promised what it intends to argue about, and you do not get to change your mind in month eighteen.

pages turned, unhurried

Ninety-four people, and a difference between the two arms you could hide behind a decimal point.

Null.

Now, understand, null is honest. Null is most of the literature, and anybody who tells you different is selling something. Null simply means not finished. Null means the thing you asked about didn't move the needle. It doesn't mean that you didn't discover something valuable.

And yet, I don't prefer null results. I prefer to move needles.

a drawer, and a folder out of it

So I sat down to write the limitations section. It is the one part of a paper where you are required to argue against yourself. I got two sentences in before I stopped.

Eighteen monthly reports on that trial. Every one of them a clean little stack. And every one of them answering a question somebody thought of in the first four weeks, back when we still believed we knew what we were looking for.

the folder set down

Eighteen months, and I had never once asked that study anything it wasn't already expecting.

— music bridge, four bars, then under —

SCENE TWO — THE INTERROGATION

the lamp again · a box lifted onto the desk · a lid off · the building quiet around it

TRACY (V.O.)A filing cabinet. How do you force answers from a thing that cannot speak?

You can put it under a lamp. You can take away everything it can look at except you. And then you start with the questions you already know the answers to, because you're trying to see if it's going to walk the straight and narrow or if it's going to try tell you tales.

a rubber band off · pages

Pain score by arm. It told me. Pain score by week. It told me. By site, by age, by which of the two clinics did the intake. I ran it down every road the monthly report had ever paved, and it came back with the same face every time.

This cabinet was cooperative. Polite. And useless. It had made its statement in month four and it was sticking to it, word for word, the way only an honest witness and a very good liar ever do.

a chair back · a handset lifted · a number dialed

Somewhere around one in the morning I did the thing you do when you have run out of your own ideas.

ring · click

RECORDED VOICEYou have reached the Biostatistics Consultation Service. Consultations are available to investigators at the design stage. If your study has already been conducted, there's a vanishingly small chance that we can help, but leave your name and number anyway.

the handset down, slowly

TRACY (V.O.)I'll say this much for that recording. It has never once been wrong.

the box again

So back to the box. And here is the part I would take out of the transcript if I were the kind of person who edited transcripts.

I sat there and I asked it the pain score by arm a second time. Same question. Same road. Like leaning on a man who has already told you twice, hoping the third time he gets tired.

He does not get tired. Paper never gets tired. The only one in that room getting tired was me.

— music bridge, two bars —

SCENE THREE — THE SAME ROOM

a door · two cups set down · a chair dragged over

CONNIEYou know what I like about this floor at one in the morning? Nobody's reheating fish.

TRACYConnie.

CONNIEI'm just saying it's an improvement. Did you hear they're repainting the second floor the same color it already is?

CONNIEAt least it's better than when we had that freezer disaster on four. They still won't give me a key to this day.

TRACYOkay. What do you need.

CONNIESignature. Bottom of the second page, and I need it tonight, because the fiscal year ends Friday and after Friday that money turns into a pumpkin.

a pen, and a signature

CONNIEThank you.

TRACYWhat did I just sign?

CONNIEChairs.

TRACYWe're buying chairs.

CONNIEWe are buying four chairs, because in June there was money for four chairs, and there is a rule somewhere that says a chair in June is a chair and a chair in July is a federal matter.

CONNIEYou're still on MERIDIAN.

TRACYIt came back null.

CONNIEEverybody finished, though. I'd have heard.

TRACYEverybody finished.

CONNIESame two clinics as the last one?

TRACYIt was, and it isn't the clinics. I looked.

a cup set down

CONNIECan I say a thing that's probably nothing?

TRACYPlease.

CONNIEThose weekly diaries come in on paper, and people don't write neat. Some of those sevens are ones. Some of those ones are sevens. Somebody sat and typed ninety-four people's handwriting into your columns. After Mrs. Okonkwo I don't let that one go anymore.

TRACYDouble entry would have caught it.

CONNIEAnd you think I didn't do double entry?

a beat

CONNIEGo home, Tracy.

a door

TRACY (V.O.)She went home. I did not.

— music bridge, two bars —

SCENE FOUR — THE ORIGINALS

a cabinet drawer, deeper than the last one · hanging folders · one file out

TRACY (V.O.)We did do double entry. Two typists, working blind to each other, and every place they disagreed is in a log I keep because I am the kind of person who keeps them. Eighteen months of that log, and the error rate on those diaries is under a tenth of one percent.

Connie was wrong.

And then I sat with why she was wrong, which took longer.

Double entry catches a seven that got read as a one. Two people key the same page, blind to each other, and the log tells you where they disagreed. What it cannot catch is the two of them agreeing perfectly about a form that had nowhere to put the thing.

Because everything I had been leaning on all night was the extract. The clean copy. Somebody went through those weekly diaries with a codebook and put the numbers in the columns we built, which is correct, and which is also a person deciding, once, in month one, what is going to count as an answer.

Somewhere two floors down was the paper those columns were made from, in the handwriting of ninety-four people.

a stairwell door · two flights, unhurried

Three in the morning and two floors down, I put my hands on the originals.

a file opened · a single sheet

Participant thirty-one. Week five. Question four, the pain score, zero to ten.

She had written a six. And then, in the white space beside it, where there is no field and no column and nowhere for it to go, she had written:

Still hurts. But I got to the post office and back on Tuesday.

nothing · then a chair, very slowly

I read that maybe nine times.

She answered the question I asked. A six. Then she answered the question I did not ask, and she put it in the margin, because we had not left her anywhere else to put it.

— music bridge, two bars —

SCENE FIVE — THE COUNT

the cabinet again · files pulled in armfuls · paper across a desk, and more paper

TRACY (V.O.)I brought all ninety-four upstairs. It took four trips and I did not care who saw me.

pages, steadily, for a long time

I read all ninety-four.

Twelve of them had written in the margin.

Not the same words. Nobody says a thing the same way twice. But the same shape, over and over, in twelve different hands.

Made it up the stairs without the rail.

Did my own shopping Thursday.

Went to my granddaughter's thing and stayed for the whole of it.

the sorting stops

Not one of them said the pain was better. Several of them said plainly that it wasn't.

The trial was not null. The trial was deaf.

— music bridge, two bars —

SCENE SIX — THE WAKE UP

a window · early traffic, thin · a handset lifted · a number dialed · ring, and ring, and a pickup

CONNIESomebody had better be dead.

TRACYConnie. It's Tracy.

CONNIEIt's a quarter to six.

TRACYI know what time it is.

CONNIEIs this about the chairs?

TRACYNo.

CONNIEBecause June was your idea.

TRACYIt isn't the chairs. Connie, I've been here all night.

CONNIEOh.

Was it the sevens?

TRACYIt wasn't the sevens. Error rate's a tenth of a percent. Your typists are fine.

CONNIEWell. That's the worst good news I've had all week.

TRACYI went and got the originals.

CONNIEThe paper ones. Downstairs.

TRACYNinety-four of them. Participant thirty-one wrote a six in the box, and then she wrote a sentence out beside it where there is no box.

CONNIEShe wrote a sentence where there is no box. Am I still dreaming?

TRACYListen to me. She wrote something outside the box.

CONNIEWhat did she write?

TRACYThat she got to the post office and back.

a beat

CONNIEThat's a good week for her.

TRACYThere are twelve of them, Connie. Twelve people who told me their knees still hurt and their lives were getting better, and I built a study that could hear exactly one of those two things.

CONNIEAnd you'd never have gone down there if I hadn't sent you after the wrong thing.

TRACYNo. I wouldn't have.

CONNIEYou're welcome.

a beat

TRACYGo back to sleep, Connie.

CONNIEI'm going back to sleep. But I want to see the other eleven when I get in.

TRACYAll right.

CONNIEAnd have the coffee ready. Just not from the machine on four.

the handset down

— music bridge, two bars —

SCENE SEVEN — THE CHARGE

a window · early traffic, thin · paper being squared into a stack

TRACY (V.O.)So I sat with it until it got light out, and then I did the hardest arithmetic in this business.

Twelve margins is not a finding.

It is twelve people, chosen by nobody, counted by me, after I already knew what I was hoping to count. I went looking for it and then I found it.

So it goes in the discussion, in the back, in the paragraph nobody reads, phrased so carefully it almost apologizes for existing. And then it goes in the next protocol as a second outcome, measured on purpose, in a study designed to be able to hear it.

a lamp chain pulled, and the room a little brighter without it

That is the whole trade, and nobody warns you about it going in. You do the eighteen months, you sweat the box all night, you finally get somebody to talk.

And what you have at the end of it is not a conviction.

a pen · one line written

But it's a lead.

— music —

ANNOUNCERThis has been The Case of the Margin of Error.

SIGN-OFF

— music up —

ANNOUNCERDr. —— appeared as the Voice of the Biostatistics Consultation Service.

ANNOUNCERAdverse Events is brought to you by Software for Research. For those who love discovery more than data entry.

ANNOUNCERAnd even though the reproducibility working group would prefer this not be repeated whenever they hear us say it: this is Adverse Events.

— music out —