Hypothesis Stress-Test — by an AI council
Attack a hypothesis to find where it would break before you test it. Four rival AIs debate it and hand back: A list of the assumptions, confounds and the experiment that would falsify it.
A list of the assumptions, confounds and the experiment that would falsify it.
Signed off by an independent Final QA audit — with a “verify before you rely on this” list, never a black box.
Why this team
Hypotheses die expensively when they are tested before they are attacked, so the attack comes first: Stephen Hawking’s seat asks whether the idea is precise enough to be wrong at all, the Statistician designs the confounds and power problems out before they design themselves in, and the Deep Researcher checks whether the answer already exists in prior work. The Data Skeptic hunts the alternative explanations for the result you are hoping for, the Devil’s Advocate tries to kill the idea outright, and the Steelman rebuilds the strongest surviving version — because the goal is a better hypothesis, not a dead one.
Each seat runs on a different frontier model — GPT, Claude, Gemini, Grok — so it’s a genuine cross-model review, not one AI in costumes. How a run works →
What the debate sounds like
An illustrative excerpt — A behavioural-science graduate student hypothesises that daily gratitude journalling improves sleep quality, and plans a four-week study with volunteer undergraduates self-reporting both.
First make it falsifiable in its current words: “improves sleep quality” — by what measure, how much, for whom, against what comparison? As stated, any positive drift confirms it and nothing can refute it, which makes it a hope, not a hypothesis. Commit now: a defined sleep measure, a minimum effect worth caring about, a comparison arm. A hypothesis that cannot specify the result that would kill it is not ready for data.
The design as described manufactures its own result: volunteers who sign up for a gratitude study expect gratitude to work, and both variables are self-reported by the same person who knows which arm they are in. Expectancy plus demand characteristics will produce a positive finding with no real effect. The fix is structural — an active control doing neutral journalling, so both arms carry equal expectation and equal pen-and-paper time.
Power arithmetic before a single participant: effects in this literature are small-to-moderate, and small effects with noisy self-report measures at a plausible volunteer sample size mean the study is more likely to produce a false negative or an inflated fluke than a trustworthy answer. Options: a bigger sample, a within-subject crossover design, or an objective sleep measure with less noise. Choose one deliberately — or the four weeks answer nothing either way.
Questions
Is this only for academic research, or for product and business hypotheses too?
The machinery is identical for “feature X will reduce churn” and for lab science: precision, falsifiability, confounds, the cheapest decisive test. Product hypotheses actually benefit more, because business experiments are routinely run with the design flaws this council is built to catch — self-selected samples, no control, metrics chosen after the fact.
What does “what evidence would falsify it” give me in practice?
Pre-committing to the disconfirming result is the single strongest protection against fooling yourself: after data arrives, every ambiguous outcome gets reinterpreted as support. The deliverable states, before you run anything, exactly which result kills the hypothesis — so the future argument with yourself is already settled.
My hypothesis survived — what does the output actually contain?
The assumptions ranked by fragility, the most likely alternative explanation you must rule out, the strongest rebuilt version of the hypothesis, and the single most efficient experiment with its falsifying result named. Surviving the council does not mean the idea is right — it means the test you run next will be worth running.
Your material is used only to run your review — never to train public models. Encrypted in transit and at rest. Security & privacy →
Want full control — pick your own minds, set the depth? Open the full council →