The Holdout Test: How to Know If Your Popup Actually Makes Money

A spin-to-win popup can hit every vanity number you’d want — thousands of spins, a healthy signup rate, a wall of “10% OFF” codes handed out — and still lose you money. Discounts cost margin. Free shipping costs your shipping budget. If the shoppers who would have bought anyway are the ones claiming the code, you paid for a sale you already had.

The only way to know the difference is to compare against shoppers who never saw the popup at all.

What a holdout group actually is

A holdout is a slice of your traffic — say, 10% — that the app deliberately excludes from the wheel. They browse and buy exactly as they would if you’d never installed a popup. Everyone else sees the wheel as normal.

Because the split happens per visitor and stays consistent, you get two comparable groups over the same traffic, the same marketing spend, and the same time period. The only difference between them is whether they saw the wheel. Profiter makes this split deterministic by hashing the visitor’s ID with the experiment ID using the FNV-1a algorithm. This means the same visitor always lands in the same group across sessions, with no server-side state needed.

Whatever profit gap shows up between the two groups is the wheel’s effect — not seasonality, not a traffic spike, not a sale you were already running. Compare that to before/after: “spins went up in March” tells you nothing, because March might have had a promotion, more ad spend, or just more visitors. A holdout removes that ambiguity by running both conditions at the same time.

Spins and signups aren’t the metric

A spin is an impression. A signup is a lead. Neither one is money, and both are easy to inflate without helping your business — generous odds and a low-friction form will pump both numbers while your margin quietly erodes underneath.

The number that matters is profit per visitor, computed after the real costs: the discount given, any free shipping absorbed, and your margin on what was actually sold. Profit per visitor in the wheel group, compared to profit per visitor in the holdout group, is the only comparison that tells you whether the popup is worth running.

If margin per visitor is higher in the wheel group than the holdout, the popup is adding profit. If it’s lower — even while spins and signups look great — the discounts are outweighing whatever extra purchases the wheel is generating.

Reading the result without fooling yourself

A few things worth knowing before you trust a number. Profiter’s measurement stack is built to prevent common statistical traps. Here’s how it works:

Test	Method	Profiter’s Implementation
Group Integrity	Sample Ratio Mismatch (SRM)	A Chi-square goodness-of-fit test runs automatically. It uses a strict significance level (alpha = 0.001) to check if the traffic split matches the percentage you set.
Statistical Significance	Two-proportion z-test	A two-proportion z-test calculates the probability that the observed difference in profit per visitor is real, not random chance. Results are shown with a 95% Wald confidence interval.
Result Reliability	Minimum sample size	A result is only called “reliable” after the test has seen at least 1,000 exposed visitors. A gap measured over a few hundred can be noise.

The strict alpha level for the SRM test is a deliberate choice. A false alarm that your test is broken is more costly than a missed one, because it erodes trust in future results. If the holdout and wheel groups don’t look like they came from the same traffic, the test flags it before you draw a conclusion from a corrupted comparison.

Finally, give the test time. A holdout run for a day catches a day’s worth of traffic patterns. A full week (ideally several) averages out weekday/weekend and any one-off spikes.

What to do with the answer

If the wheel is adding profit, that’s your signal to leave it running and maybe test a second variable — a different prize mix, a different trigger — against the same holdout. If it isn’t, the fix usually isn’t “remove the popup,” it’s the odds. Profiter’s recommendation engine flags common issues based on your store’s data, such as:

• A “no win” share below 20%: You’re discounting nearly every visitor who plays.
• A jackpot probability over 1.5% without a redemption limit set, creating margin risk.
• Ignoring the Rule of 100: If your average order value is over $100, fixed-amount discounts are usually safer than percentages. If it’s under $100, percentages are often better. This is the fastest lever to check, and it’s covered in detail in a simple way to set fair wheel odds.

Either way, you find out from a number grounded in your actual margins — not from a spin count that looked good in a screenshot.