SYS/DEF
CRO hypothesis
Because [evidence], if we [change], then [metric] will move by [size], measured by [setup]. Anything else is opinion.
Definition
An experiment hypothesis ties an empirical observation, a change, a primary metric, an expected size, and a measurement setup. “We should make the button bigger” is not a hypothesis. “The CTA below the fold is clicked by only 4% of sessions; moving it above the fold should lift CVR by 0.3 to 0.4 points, measured over 2 weeks, unique users” is one.
ICE / PXL scores help prioritise a backlog. They do not replace expected-value maths or MDE: an “impact 8/10” hypothesis that traffic cannot detect is fiction.
When it matters
Before every test, in writing, shared. It is the contract that stops you changing the primary metric on day 4. It is also the entry to the tracking plan: if the event does not exist, the hypothesis is not testable.
In results review, you reread the hypothesis, not the dashboard. A significant effect on a secondary metric does not “confirm” a hypothesis written on another.
Pitfalls
Stacking three changes in one arm (“while we’re at it”). Forecasting a +30% lift because the sprint must “be worth it”. Rewriting the hypothesis after the fact to match the winner. Using ICE as an excuse not to estimate n.
Related terms
Related tool
Traffic, expected lift, conversion value, implementation cost: prioritise in money: Open expected value of a test.