SYS/DEF
Novelty effect
Users react to newness, not to the treatment. Monday’s lift is gone by Thursday.
Definition
The novelty effect is a time bias: variant B draws clicks and conversions because it is new, not because it is better at equilibrium. The inverse exists too (habit change, primacy): returning users snub B while they relearn, then converge.
You see it on daily lift curves that fall after 3–7 days, or on a gap concentrated among returning visitors vs new ones. It is not a stats bug; it is a product phenomenon.
When it matters
Whenever the treatment is visible (new header, new price, new nav). A test that is too short captures mostly novelty. Hence the weekly-cycle reminder in test duration: at least one full week, often two, not “as soon as p < 0.05”.
It also matters at readout: a significant lift piled into the first 48 hours, flat afterwards, is not a win to ship as-is.
Pitfalls
Stopping at the first spike. Excluding returning users “to keep it clean” — you then measure only the naïve. Rerunning the same test after a partial ship: the cohort has already seen B. Confusing seasonality (weekend, payday) with novelty.
Related terms
Related tool
Turn an n into calendar weeks, with the reminder that a short test is one you will peek and whose lift is mostly novelty: Open test duration.