SYS/CASE
Growth & Performance · CRO · Analytics · 2025
Airline — analyse an A/B test and recommend not to ship
By Mathieu Milon, AI engineer growth & CRO · Published
Case study — analysis of an A/B test on the booking journey of a major airline (real extract, 135,298 visitors). Hypothesis tested: replace the “Flight details” pop-up with a side panel without breaking progress to the recap.
Anonymised client. Figures from the working file, not a company press release.
−2.25% · do not ship
Context
Two brands in the same airline group share a booking funnel (only the colours change). At the flight-selection step, “Details” opens flight information. The test changes the container of that information: pop-up on one side, side panel on the other.
Problem
Two questions before even reading the numbers: which KPI decides (purchase is too underpowered to decide alone), and how to read the effect by device — a side panel does not behave the same on mobile and on desktop.
Approach
Primary KPI = recap-page reach rate (not purchase, too rare to be significant). Secondaries: payment, revenue per visitor. The analysis then runs in three times.
Extract cleanup
214,473 extract rows reduced to 135,298 unique visitors: that is the base, not the raw log, that is then read. SQL and Python rebuild the analysis from the extract.
Statistical tests
SRM check (p = 0.18) before any reading: without a healthy split of visitors between variants, no figure holds. Then z-test on the primary KPI — the recap rate — and on the secondaries.
Segments
An HTML explorer to walk the 29 crossed segments — device, brand, haul. That is where the effect localises: mobile degraded, desktop roughly neutral.
Tools
- SQL
- Python
- HTML explorer — 29 segments
Where AI comes in
Rebuild the analysis in SQL + Python in days rather than weeks, generate the 29-segment explorer, review the results — the decision stays human: do not ship.
Results
Recap (primary)
−2.25%
49.36% → 48.25% · sig. > 99%
Purchase
−2.54%
not significant (underpowered)
Where it breaks
Mobile
desktop roughly neutral
| Group | Recap rate |
|---|---|
| Control | 49.36% |
| Variant | 48.25% |
−2.25%, significant at over 99% · purchase −2.54%, not significant · SRM p = 0.18 · 214,473 extract rows → 135,298 unique visitors · 29 segments analysed. Recommendation: do not ship the side panel.
The axis starts at zero and stops at 55%, deliberately: a truncated axis would turn a 2.25% gap into a cliff, whereas the whole case is about measuring a small gap — small, negative and yet significant.
Full protocol
| Measure | Value |
|---|---|
| Extract rows processed | 214,473 |
| Unique visitors analysed | 135,298 |
| Crossed segments | 29 |
| Recap rate, control | 49.36% |
| Recap rate, variant | 48.25% |
| Effect on purchase | −2.54% (not significant) |
| Split check (SRM) | p = 0.18 |
Takeaway
- 1
The primary KPI first: choosing the right decision measure before looking at the numbers avoids being led by a signal that is too weak.
- 2
A clean SRM before reading: without a healthy split of visitors between variants, no conclusion holds.
- 3
Knowing how to say no: the real value of an analysis is sometimes to recommend not to ship.
Visuals
Control: flight details open in a pop-up Variant: flight details slide in as a side panel
Related expertise: CRO, Analytics, AI engineering.
A similar project?
Let's talk about your next engagement
A funnel, a product or an extract to analyse: book a 30-minute call, written scoping within 48h.