Why we publish negative results

Most published work in decision science is positive: a method beat a baseline on a benchmark. The file drawer, the methods that did not beat the baseline, is where a great deal of the actual knowledge lives, and almost none of it is written down. We think that is a mistake, and we run our research to correct it.

The discipline

Each serious experiment is pre-registered: the question, the dataset or environment, the baseline, the metric and the success threshold are fixed before the result is known. A frozen claim is then graded on a strict scale, where a narrow, honestly-scoped result is the common outcome. Most candidates do not survive. That is the point.

One part of this is a goal rather than a practice, and it would be poor form to blur that on this page of all pages. Independent replay: a result running again from frozen inputs, for someone who did not build it, is the standard we are building toward, not one we can claim today. Verification here is currently our own, done adversarially and against a frozen falsifier. That is weaker than an outside pass and should be read that way.

What it looks like in the library

The test of an evidence scale is whether its lowest grades ever get used. Of the published research, lab and client work, most entries carry idea, no result yet, and exactly one is a negative result: a screening certificate that costs more than the solve it saves. It stays up so the next person considering that approach finds out cheaply.

The discipline also shows up as claims held back rather than dressed up. Two entries on the de-censoring frontier state in their own text that a field claim would need a live baseline with the falsifier frozen before the data is opened, and that until that run exists what they have is a backtest and not more.

Why it is worth the discomfort

• It stops us, and, when published, others, from re-spending on a dead end.
• It calibrates our own confidence, so a claim we do make carries weight.
• It is the honest half of a research reputation. A vendor with only successes is either extraordinarily lucky or not telling you everything.

What this is

This is a statement of practice, labelled as an idea rather than a measured result, precisely because it makes a general claim rather than a specific one. The specific negative results: a robustness case that gained nothing over a greedy baseline, an unbiasedness claim that failed an exact counterexample, are published individually, each with its own pre-registration and scope, wherever disclosure is safe and adds information.