Before/After Is Not a Test: How to Measure a Shopify App's Real Lift

In short: If you installed an app in September and October's numbers look good, the season explains most of it. In the first two days of India's 2026 festive sales, fashion ran at three to five times its normal daily GMV. App dashboards have the opposite problem: they count orders the app touched, and studies that measured both find that most of those orders would have happened anyway. When recommendation systems are put through randomized tests, the lifts that hold up are usually in the low single digits. The only way to know what an app adds is to hide it from a random share of shoppers and compare. If your vendor can't do that, the five questions below tell you how much to trust what they report instead.

It is the end of October. Plenty of fashion brands installed a new app in September, ahead of the festive sales, and are now looking at an October that beat September by a wide margin. The app's dashboard agrees and shows a large attributed-revenue number.

Both numbers are real, and neither tells you whether the app worked. This post covers why, what a real measurement looks like, and how to question a vendor about theirs.

Why September against October tells you nothing

The season dwarfs any app. Redseer measured the first 11 days of India's 2025 festive sales at nearly 3.5 times business-as-usual GMV. For the first two days of 2026 it put e-commerce GMV at around six times a normal day, and fashion at three to five times its usual daily level. A recommendation app that adds a few percent is invisible next to that. In the US the same thing happens in late November: Adobe recorded $11.8 billion of online spending on Black Friday 2025 and $44.2 billion across Cyber Week.

The festive bump is different every year. You can't take last year's festive growth and subtract it. Redseer reported fashion growing only in the low single digits through the first leg of the 2025 sales, then 22 to 25% year on year in the first two days of 2026. The calendar moves as well. Unicommerce's comparison of the two years' first festive weeks set 22 to 28 September 2025 against 7 to 13 October 2026, so even a year-on-year window lines up different weeks of the season.

The shoppers change. In the first festive week of 2026, Unicommerce reported that Tier 3 towns placed 42% of e-commerce orders, and Tier 2 and Tier 3 together nearly two-thirds. When a different mix of people arrives, conversion rate and AOV move even if nothing on the store changed.

New things get clicked because they are new. Ron Kohavi, who ran experimentation at Microsoft, describes the novelty effect: a visible new feature draws clicks at first, and the initial effect fades over time. The first weeks after an install are when an app looks its best.

Bad months recover anyway. If you switched an app on because a collection had a weak month, that collection was likely to bounce back regardless. This is regression to the mean, and it flatters any fix applied at a low point.

Each of these alone can produce a bigger swing than the app. Together they make a before/after comparison unreadable.

The app's dashboard has the opposite problem

An app's attributed revenue counts orders the app touched. What you want to know is how many orders would not have happened without it. Those are different numbers, and the research that has measured both finds a wide gap.

eBay, randomized: Blake, Nosko and Tadelis switched off paid search ads for parts of the US and found the returns were a fraction of what the non-experimental estimates suggested. For brand-keyword ads they found no measurable short-term benefit. Regular shoppers came anyway.
Amazon, quasi-experimental: Sharma, Hofman and Watts studied 2.1 million users' browsing over nine months and estimated that at least 75% of recommendation click-throughs would likely have happened without the recommendations. A follow-up put it at 50 to 80% depending on the product category.
Facebook, randomized: Gordon, Zettelmeyer, Bhargava and Chapsky compared 15 large ad experiments with the observational methods advertisers usually use. The observational methods often failed to recover the effect the experiments measured.

The most useful statement on this comes from a vendor. Stylitics, which sells outfitting software to large retailers, wrote in September 2026 that "engaged-versus-non-engaged comparisons overstate impact," because shoppers who interact with outfit content already intend to buy more. It recommends treating those figures as directional. Comparing shoppers who clicked a widget with shoppers who didn't is exactly that comparison, and it is the headline of most app case studies.

The figure you will see quoted most often, that 35% of what people buy on Amazon comes from recommendations, comes from one sentence in a 2013 McKinsey article. The article gives no method or data source for it. At best it is a share of attributed purchases. It is not a measured lift.

We covered how attribution windows inflate these numbers in how to read a Shopify app's attributed revenue.

What a real lift looks like: small

When recommendation systems are tested properly, the results are modest.

Zalando ran a randomized A/B test of a new complementary-item carousel. Users were split into two groups, the test ran for several weeks in several countries, and it ran in full-week cycles so weekday and weekend shopping balanced out. The result was +6.23% on click-through rate and +3.24% on products ordered from the carousel. Two caveats matter. The control group saw Zalando's previous recommender, not nothing, and the 3.24% counts carousel orders only, not the whole site.
Alibaba tested its outfit generator, POG, for seven days and reported more than 70% higher click-through than the model it replaced. That is clicks only, against another model, with no purchase figure.
Lee and Hosanagar ran one of the few randomized tests against no recommender at all, at a large North American retailer. "Customers who bought this also bought" lifted views by 25% and items purchased by 35% among shoppers who bought. That measures buyers only, and the category was not fashion.

Kohavi's figures from Microsoft give the base rate. About a third of the ideas they tested improved the metric significantly, a third made no difference and a third made it worse. His rule, which he takes from Twyman's law, is that any figure that looks interesting or different is usually wrong. If a before/after comparison shows an app adding 40%, look for the mistake before you believe it.

Why a holdout works when before/after doesn't

A holdout hides the app from a random share of shoppers and compares them with everyone else over the same weeks. Both groups shop in the same season, see the same prices, get the same ads and come from the same mix of cities. The only systematic difference between them is the app, so the gap between them is what the app caused.

That also means a holdout can run through Diwali. The festive surge lifts both groups equally and drops out of the comparison.

The cost is that it needs volume. A Shopify Partners guide gives the arithmetic for a 50/50 test at a 2% conversion rate:

To detect a lift of	Visitors needed per group	Total
20%	19,784	39,568
5%	309,928	619,856

The same guide says 250 purchases per group is sometimes enough and 500 or more is more common. A store with 20,000 sessions a month would need about two months to detect a 20% lift and more than two and a half years to detect a 5% one. A holdout of 10 or 20% leaves a smaller control group, so it takes longer still. Small stores can only see large effects, and most real effects are small. That is a reason to run the test for longer, not a reason to go back to before/after.

Who offers a real holdout

Few vendors document a control group that sees no app at all. This is what each one's own documentation says, as of October 2026.

Vendor	Control group that sees no app?	What their docs say
Dynamic Yield	Yes	5% of users go into a Global Control group that gets, as far as possible, a non-Dynamic Yield experience
Bloomreach	Yes, in Engagement	The control group is "customers who didn't see anything"
Rebuy	Yes	A/B tests can include a no-widget variant. Revenue is scoped to Rebuy-tracked orders and won't match Shopify totals
Nosto	Through an outside tool	Built-in tests compare variations. Its guide describes a test where the original shows no recommendations
Kimonix	No	Splits traffic between two ordering strategies for the same collection
Sort'd	No	Tests two sort strategies head to head
Shopify Search & Discovery	No A/B mode found	Reports click rate and purchase rate on recommendations
Stylitics	No published randomized test	Case studies compare engaged and non-engaged shoppers, which the vendor says overstates impact
LimeSpot	Not documented	Runs A/B tests of "experiences" on higher plans; a no-app control isn't described

A test of one strategy against another is still a valid randomized test. It tells you which configuration of the app does better. It doesn't tell you whether the app does better than no app.

Five questions to ask before you believe a lift

Is there a randomized group that sees no app at all? If not, every lift figure is a comparison of something else.
What is the attribution window, and does it count clicks or views? Nosto, for example, uses a same-session "View & Buy" model. Windows differ by vendor, so two apps' attributed revenue can't be compared with each other, or with your Shopify totals.
Does the reported revenue reconcile to your Shopify totals? Add up what every app on your store claims and compare it with what Shopify says you made.
How many orders were in each group, and what is the range on the result? Fewer than a few hundred purchases per group is too few to read.
Did the test run in full weeks, and outside a sale? If not, ask how the season was kept out of the result.

If your app has no holdout

Ask the vendor for one, and treat a refusal as information. If you can't get one:

Don't judge the app on a festive or sale window. Wait until traffic is back to normal.
Compare covered products against uncovered ones over the same weeks. Pick, before you look at results, a set of products or collections that show the app and a similar set that don't, and compare how each changes. This is the approach Stylitics suggests alongside randomized tests. It removes the season, but it isn't randomized, so the two sets may differ in ways that matter.
Read attributed revenue as a ceiling. It is the most the app could have added. What it did add is somewhere below that.

Where Angadi sits

Full disclosure: Angadi is our product.

Angadi credits a sale only when the shopper clicked a look and bought within 24 hours, and we treat that figure as a ceiling ourselves. Since October 2026 the app also has a setting called "Measure what the widget really adds." It hides the widget from a random 10% or 20% of shoppers and compares them with the rest. It is off by default. Before you start it, it shows what it could cost, worked out from your last 30 days of orders, because held-out shoppers miss whatever the widget would have sold them. It also shows roughly how long a first result will take at your current pace. When there is enough data, the Analytics page shows the revenue the widget caused, with a range.

Two things it can't do. It can't make a small store's result arrive faster than the arithmetic above allows, so at low volume expect to wait months. And we don't have merchant results from it to share yet. When we do, we will publish them with the ranges included.

If you want the background on why outfit recommendations lift basket size at all, does complete the look raise AOV goes through the evidence. For what our 24-hour window leaves out, see the revenue we deliberately don't claim.

Frequently asked questions

Why can't I compare my sales before and after installing an app? Because too much else changes between the two windows. In India's 2026 festive sales, Redseer put fashion's first two days at three to five times its usual daily GMV. The shoppers change too: in the first festive week of 2026, Unicommerce reported 42% of orders came from Tier 3 towns. Any app installed in September will look like it worked in October, whether it did or not.

What is a holdout test? A holdout hides the app from a random share of shoppers, often 5 to 20%, and compares them with everyone else over the same weeks. Because both groups shop in the same season, see the same prices and get the same ads, the difference between them is what the app caused. It is the only way to separate the app's effect from everything else that happened that month.

How many orders do I need for an A/B test to mean anything? More than most small stores expect. A Shopify Partners guide puts it at 250 purchases per variation at minimum, and more often 500 or more. At a 2% conversion rate it estimates 19,784 visitors per variation to detect a 20% lift, and 309,928 to detect a 5% lift. Most real effects are small, so a small store can usually only detect a large one.

Is an A/B test between two app settings good enough? It answers a different question. Testing one sort order or recommendation strategy against another tells you which setting of the app works better. It does not tell you whether the app beats having no app at all. Kimonix and Sort'd, for example, test one strategy against another. For the second question you need a group that sees nothing.

My app has no holdout. What can I do instead? Ask the vendor for one first. If that fails, avoid judging the app on a festive or sale window, compare products or collections that show the app against ones that don't over the same weeks, and treat the app's attributed-revenue figure as the most it could have added, not what it did add. None of these is as reliable as a randomized holdout.

Sources

Redseer Strategy Consultants, E-commerce festive sale trends: 1st leg (11 day) closes at INR 60K to 62K crore, Nikhil Dalal, 6 October 2025.
Business Standard (PTI), E-comm sell goods worth ₹24,000 crore in first two days of festive sales: Redseer, 11 October 2026. Fashion figures: Storyboard18.
Unicommerce first festive week 2026, via Free Press Journal, e-commerce order volumes jump 31% year on year, 14 October 2026. Unicommerce press page.
Adobe Analytics US holiday 2025, via Chain Store Age and Payments Dive.
Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020 (Twyman's law and the novelty effect). Kohavi et al., Trustworthy online controlled experiments: five puzzling outcomes explained, KDD 2012.
Kohavi, Online controlled experiments: lessons from running A/B/n tests for 12 years, KDD 2015 keynote.
Blake, Nosko and Tadelis, Consumer heterogeneity and paid search effectiveness: a large-scale field experiment, Econometrica 83(1), 2015. NBER working paper 20171.
Sharma, Hofman and Watts, Estimating the causal impact of recommendation systems from observational data, ACM EC 2015. Follow-up: Split-door criterion, Annals of Applied Statistics 12(4), 2018.
Gordon, Zettelmeyer, Bhargava and Chapsky, A comparison of approaches to advertising measurement: evidence from big field experiments at Facebook, Marketing Science 38(2), 2019.
Stylitics Marketing Team, Why your PDPs should own "what to wear with", modified 16 September 2026.
MacKenzie, Meyer and Noble, How retailers can keep up with consumers, McKinsey & Company, 1 October 2013.
Wu, Sánchez Rodríguez and Corona Pampín, Session-based complementary fashion recommendations, recsysXfashion workshop, ACM RecSys 2019.
Chen, Huang, Xu et al., POG: personalized outfit generation for fashion recommendation at Alibaba iFashion, KDD 2019.
Lee and Hosanagar, How do recommender systems affect sales diversity? A cross-category investigation via randomized field experiment, Information Systems Research 30(1), 2019. Lift figures from the earlier version, Impact of recommender systems on sales volume and diversity.
Shopify Partners blog, Thinking about A/B testing for your client? Read this first (guest article, undated).
Vendor documentation, read October 2026: Dynamic Yield Personalization Impact Report; Bloomreach A/B test evaluation; Rebuy A/B testing dashboards; Nosto campaign testing attribution and recommendations campaign testing; Kimonix A/B testing; Sort'd listing summary; Shopify Search & Discovery analytics; LimeSpot A/B test feature. Stylitics case study: Suzanne's.