Shamalo

SYS/COCKPIT

CRO tools

Pre-test, post-test reading and sequential peeking — the experimentation lab in one workspace.

Pre-testMDE

MDE for a given n

Reverse power: with the traffic you have — as visitors, or as a rate × a duration (days, weeks, months) — what is the smallest detectable effect? Several MDEs are compared, and each one says whether it fits the window.

SYS/IN

5MDE (minimum detectable effect): the smallest gap this traffic can still see. Absolute = percentage points; relative = a fraction of the baseline. Glossary definition.
How to set the traffic
The window multiplies the rate by the duration, then splits the total equally across arms.
Site-wide traffic, not per arm. A week is 7 days, a month is 365.25 / 12.
Rate unit
Read durations in
Arms
The statistical n stays that of a two-arm comparison. 3 or 4 arms stretch the calendar (total traffic = n × arms). Multiple-testing correction lives in its own tool.
Test direction
One-sided when only a gain matters. Two-sided when a loss must be visible too.
MDE shown large
5
80
MDE scenariosTick several effects. Each gets its own n and duration. The detectable MDE is computed on the traffic you actually have.

SYS/OUT

Detectable MDE

1 pp

  • in percentage points: 1 pp
  • relative to the baseline rate: 20%

Available n per arm: 8,155 — Window covered by this n: 5.9 weeks

Duration by chosen MDE
MDEn / armDuration (weeks)With this traffic
0.5 pp31,23122.4Out of window
1 pp8,1555.9Covered
2 pp2,2101.7Covered
10 % relative31,23122.4Out of window
20 % relative8,1555.9Covered
Duration by chosen MDE

Bars: duration of each ticked scenario, in the unit you picked. The table is readable with JavaScript off.

How this is calculated

Method

Reverse power: given a budget of (n) visitors per arm and a baseline rate (p_1), what is the smallest effect still detectable at the chosen (\alpha) and power?

The calculation inverts the two-proportion z-test sample-size formula (bisection on the absolute MDE, monotone in (\delta)). The result is returned in the requested mode:

  • Absolute: (\delta) in percentage points.
  • Relative: (\delta / p_1), a fraction of the baseline.

Both readings are always shown together, so a “20%” cannot be mistaken for points. This is the honest question before launching a test: can this traffic even see the effect we hope for?

Sources

Window and scenarios

n per arm can be typed in, or derived from a rate (visitors per day, week or month) times a duration in the unit you pick, then split equally across 2, 3 or 4 arms. A week is 7 days. A month is 365.25 / 12 days — an average, not a named month.

Each ticked scenario is a candidate MDE, absolute (points) or relative (a fraction of the baseline). For each one the tool computes the required n and the duration in the display unit, then says whether the available traffic covers it. The detectable MDE stays the power inversion on that traffic, not an average of the scenarios.

A one-sided test replaces (z_{\alpha/2}) with (z_{\alpha}). It is only justified when a loss does not have to be detected.

Limits

  • Same model as binary sample size: independent visitors, fixed horizon, two arms, one metric.
  • An (n) observed after the fact is not a pre-registered (n). If the test has already been peeked, this MDE is no longer the protocol’s MDE.
  • This is not a “B wins” decision. It is a detectability floor under H0, not a verdict on an arm.

SYS/VIEW

SYS/NAV