Validation
How we find out whether the idea works before anyone builds it
Most of the money a company loses on software is lost on building things that were never going to pay. We have watched it happen, and early in our history we were paid to help it happen. So we sell two short, fixed-fee steps whose only purpose is to answer the question first: would a better method beat how you decide today, and by enough to be worth it? A large commitment follows the evidence, or it does not follow at all.
The two steps
What each step costs and what it answers
The audit puts a number on the opportunity, and can conclude that there is not one. The proof of value tests the method against your own data. Both are fixed price, agreed in writing before they start, and either can be the last thing we do together.
Decision Opportunity Audit
Maps the decision, its economics, the constraints, the data and the operating path, and ends with a clear recommendation: go, reshape or stop.
- Frames the decision and puts an economic baseline on it
- Assesses the data, constraints and route to operation
- Reviews prior art so you are not paying to reinvent
- Ends with an explicit go / reshape / stop recommendation
- Useful even when nothing has been built yet
Sample deliverables
Proof of Value
Runs a candidate method on your own data and compares it against an agreed baseline, with a route to production designed in from the start.
- Runs on your own data, with the mess left in
- Compares against an agreed baseline and success threshold
- Requires a sponsor and a fixed decision date
- Includes the route to production as well as a model
- May conclude, legitimately, that the system should not be built
Sample deliverables
A valid result
A proof that ends in "do not build" has done its job.
If the evidence says the method does not beat what you do today, or that the value does not justify a build, then you have spent four to eight weeks and a fixed fee to avoid spending a year and a team. We count that as a success. So should you, although in our experience it takes a few weeks to feel like one.
The objection
"You show no client savings figures. Why should I trust you?"
A fair challenge, and here is the position. We do not publish client margin numbers, because they belong to the client and most clients prefer not to. We could estimate them, and some firms do. But the first CFO to check an estimate against the client would be the last to read this site, and we would lose the clients who kept quiet as well. So we ask you to judge us three other ways, and none of them needs anyone's permission:
Read the code. The OR library our proofs run against and our MLOps platform are open source and readable today. The theorem prover is written and licensed but its repository is not published yet. We would rather you inspected our competence than took our word for it, and the code is where you can.
Judge the deliverable. A worked sample of the findings is published on a synthetic company, three of them in full, at the depth a paid audit is written to, so you can weigh the quality of the thinking before you pay.
Buy the smallest step. The Audit is fixed-price and can end in "don't build." You are not betting a large sum on our word. You are buying a bounded piece of evidence, and the second step waits on what it says.
What you can check today
- Scale, in production
- Nine internal systems behind a platform serving 100M+ people (Livesport), and Gitrevio, 31 connected sources, running.
- Correct, bounded science
- Machine-checked proofs, published negative results, and limitations stated on every item in the research library and the Lab.
- A method you can run yourself
- The margin-at-risk calculator is the toy version of the forecaster we build.
Our own evidence labels, counted
How we grade our own research, and how often we use the low grades
Every piece of research, lab work and client work we publish carries an evidence status, and the status is chosen before the writing. Here is how the 27 published items fall. The counts come from the collections when this page builds, so nobody typed them in.
- formally proved
- 3
- in production
- 4
- backtested
- 5
- reproduced
- 5
- idea
- 9
- negative result
- 1
The label we use most is the weakest one
More of the library is marked idea than anything else. That is what a frontier looks like from the inside, where most of what you are working on has not worked yet. And a taxonomy that never reaches for its lowest grade is a marketing device, whatever it calls itself.
One entry is a published dead end
Decision-stability certificates is labelled a negative result: a screening step that turned out to cost more than the solve it was meant to save. It stays published so that the next person with the same idea can find out for the price of a page, and that is less than it cost us.
The same discipline holds a claim back where dressing it up would be easier. One backtested lab entry states outright that a field claim would require running against a live baseline with the falsifier frozen before the data is opened, and that until that run exists it is "a strong, honest backtest and not more". Writing down in advance what would prove you wrong makes the rest worth reading.
Qualification
Is there anything to validate yet?
If you can answer yes to these, an audit will probably pay for itself. If you cannot, a conversation costs nothing and will get you further than a proposal would.
- A repeated decision that carries money, risk or time
- A named sponsor accountable for the decision
- A realistic route to the data the method depends on
- A willingness to act on the answer, including a decision not to build
FAQ
Questions about validation
What are the two validation steps?
A Decision Opportunity Audit, which finds and frames the opportunity and puts an economic baseline on it, and a Proof of Value, which runs a candidate method on your own data against an agreed baseline. Each is a fixed, self-contained step, and the second only follows if the first says it should.
How long do they take and what do they cost?
The Audit runs two to three weeks, from €15,000. The Proof of Value runs four to eight weeks, from €40,000. Both are starting prices. The final figure depends on scope, and we agree it in writing before you commit.
Can validation conclude that we should not build?
Yes. A proof of value that ends in "do not build" has done the job you paid it for. Ten years in, we have come to think that stopping a weak investment at €40,000 is worth as much as backing a strong one, because the alternative is finding out the same thing at €400,000, eight months later, with a team already hired.
What does a Proof of Value require from us?
It runs on your own data, with the mess left in, and needs a named sponsor and a fixed decision date. It compares the method against an agreed baseline and success threshold, and includes the route to production as well as a model.
Why should we trust you without published client savings figures?
We do not publish client margin numbers, because they are the client’s to disclose and most prefer not to. What you can do instead is read the code we have published, judge a full worked sample of the audit findings written for a synthetic company, and buy the smallest step first. The audit is a fixed fee, and it can end with a recommendation not to build.
How do we know you can operate at scale?
We run nine internal systems behind a platform serving more than 100 million people for Livesport, and Gitrevio, with 31 sources, in production. Our proofs are machine-checked and we publish negative results, with limitations stated on every research item.
Start with the decision
Start with the audit
Two to three weeks, fixed fee, run by a partner. It maps the opportunity, puts a number on it, and ends with one of three words: go, reshape, or stop.
Who stands behind this
A partner is accountable for every claim on this page, and for telling you when a result does not hold. Client work stays confidential and unattributed unless the client decides otherwise. So the numbers here come from our own research, and the engagement numbers stay with the clients who own them.
