How to measure an AI chatbot's real revenue: the holdout test

Why does "revenue influenced by chat" overstate what a chatbot earns?

Because the shoppers who open a chat are not a random slice of your traffic. They are more engaged, often already on a product page with an item in mind, and some would have bought without the bot. Counting all their orders as chatbot revenue credits the bot for intent it did not create.

This is a self-selection problem. The act of chatting is a signal of buying intent, so any report that compares chatters with non-chatters, or sums the orders of everyone who chatted, mixes two things: the effect of the conversation and the pre-existing difference between the people who chose to have one. A dashboard line such as "chat-assisted revenue" is a form of attribution, not a measurement of cause.

Advertising research shows how large that gap can be. In a study of 15 large advertising experiments at Facebook, Gordon, Zettelmeyer, Chapsky and Bhargava (Marketing Science, 2019) found that common observational methods often failed to reproduce the effects measured by randomized experiments, even after controlling for extensive demographic and behavioral data. Chat is a different channel, but the selection mechanism is the same: the people exposed are not like the people who are not.

What is a holdout test for an AI chatbot?

A holdout test randomly assigns each visitor to one of two groups before they see anything: most get the chatbot, a smaller share never sees it. Because only chance decides the group, the two groups are alike on average, so the revenue difference between them is the chatbot's causal effect.

This is a controlled experiment, the same design behind A/B testing. Kohavi, Henne and Sommerfield describe it in their practical guide to controlled experiments on the web (KDD 2007) as the design for establishing a causal relationship between a change and user behavior: with random assignment, any systematic difference in the outcome is the result of the assignment.

In product teams, "holdout" usually means a control group kept away from a feature over a longer period to measure cumulative impact. Spotify's experimentation glossary describes holdouts as typically 1 to 5% of users. For a chatbot, the holdout is the group of visitors for whom the chat bubble simply does not load. The resulting difference is the incremental lift, and a holdout test is the most direct way to get it.

How do you set up a chatbot holdout test on a Shopify store?

Decide the metric and the minimum lift you care about, randomize at the visitor level, hide the chatbot entirely from the holdout, and compare revenue per visitor across all visitors in each group, not just those who chatted. Write the plan down before the test starts so the result cannot be reinterpreted afterwards.

The most important design choice is what you compare. Compare every visitor assigned to the holdout with every visitor assigned to the chatbot group, whether or not they opened the chat. This is called an intent-to-treat comparison. It looks diluted, because most visitors never chat, but it is the honest number: it is what the chatbot adds to the store as a whole.

1. Pick one primary metric. Revenue per visitor captures both conversion rate and average order value. Track conversion rate as a secondary metric, because it needs far fewer visitors to read clearly.
2. Choose the minimum lift worth detecting. If a 3% lift would not change your decision, do not size the test for it.
3. Randomize by visitor, not by page view. A shopper should stay in the same group on every visit, otherwise she sees the bot on Monday and not on Tuesday and both groups are contaminated.
4. Make the holdout clean. The chat bubble, proactive prompts and any chat-driven emails must not reach holdout visitors.
5. Fix the sample size and duration in advance, in whole weeks (see below).
6. Run an A/A check first if you can. Split traffic with both groups seeing the same experience and confirm the split and the revenue numbers come out even.
7. Analyze once, at the planned end, and report the lift with its confidence interval.

How many visitors do you need for a holdout test?

It depends on your baseline conversion rate, the smallest lift you want to detect and how you split traffic. Small lifts on low conversion rates need hundreds of thousands of visitors. Halving the lift you want to detect roughly quadruples the visitors you need, and a small holdout share stretches the test further.

The arithmetic is worth seeing once, because it explains why many stores cannot measure a small chatbot effect in a month. The worked example below uses invented numbers for a hypothetical store and the standard two-proportion approximation with a 5% significance level and 80% power, where the combined z-value factor is (1.96 + 0.84)² ≈ 7.84.

How long should a chatbot holdout test run?

Run it until the planned sample size is reached, and always in whole weeks, with a minimum of one to two weeks. Weekday and weekend shoppers behave differently, and stopping mid-week tilts the result. Do not stop early because the lift looks significant on day five.

Kohavi and colleagues recommend running experiments for at least a week or two and then in multiples of a week, so that day-of-week effects even out. They also show the cost of a lopsided split: a test needing five days at 50/50 takes about 25 days at 95/5, and they advise against very small splits that would run for months, because effects such as cookie churn start to contaminate the data (KDD 2007 guide).

Stopping early is a common way to fool yourself. Evan Miller's How Not To Run an A/B Test shows that repeatedly checking for significance and stopping when it appears inflates the false-positive rate far above the stated level. Fix the sample size in advance, or use a sequential method designed for early stopping.

What can go wrong with a holdout test?

The common failures are novelty effects that fade, seasonal events that swamp the signal, holdout visitors who still reach the chatbot, and analyzing only the shoppers who chatted. Each one produces a confident number that is wrong, so check for all of them before you trust the result.

What should a chatbot ROI report include?

Report the lift in revenue per visitor with a confidence interval, the conversion and order-value changes behind it, the dates, the split, the number of visitors in each group and any known contamination. Then convert the lift into incremental revenue and subtract the chatbot's cost to get ROI.

A single lift percentage with no interval invites overconfidence. An interval such as "+1% to +9%" tells the owner both that the effect is probably positive and how uncertain its size is. Report the result even when the lift is flat or negative; a test that can only return good news is not a test.

• Primary result: lift in revenue per visitor, with a 95% confidence interval.
• Drivers: change in conversion rate and in average order value, each with its interval.
• Design: start and end dates, split, visitors and orders per group, randomization unit.
• Caveats: promotions or holidays during the test, contamination rate, anything that changed.
• Money: incremental revenue = lift in revenue per visitor × visitors exposed; ROI = (incremental gross profit − chatbot cost) ÷ chatbot cost. Use gross profit, not revenue, if you want true return.

How does Tycoon approach revenue measurement?

Tycoon is measured with a holdout: a small, randomly chosen share of visitors does not see Tycoon, and the dashboard compares their revenue with everyone else's. The Growth plan includes one 30-day lift test, and the Tycoon plan adds continuous holdout measurement.

We think a sales associate should be judged on incremental revenue rather than on chat-assisted revenue, for the reasons above. On Growth and the Tycoon plan, the holdout is built into the product rather than left to the merchant to wire up, and the plan details are on the pricing page. For how the agent decides what to recommend and when to make an offer, see how it works and our guide to upsell and cross-sell with AI. If you are comparing vendors, ask each one how they measure lift; our guide to choosing an AI shopping assistant has a list of questions.

Frequently asked questions

Sources

1. Kohavi, Henne, Sommerfield: Practical Guide to Controlled Experiments on the Web (KDD 2007) · checked October 20, 2026
2. Gordon, Zettelmeyer, Chapsky, Bhargava: A Comparison of Approaches to Advertising Measurement (Marketing Science, 2019), Kellogg School of Management · checked October 20, 2026
3. Evan Miller: How Not To Run an A/B Test · checked October 20, 2026
4. Evan Miller: Sample Size Calculator · checked October 20, 2026
5. Spotify Confidence: What is a holdout group? · checked October 20, 2026