At Shero, we have run CRO programs across many of our clients' Shopify stores. Until now, A/B testing on Shopify has always meant a third-party app. These tools brought real experimentation capability to the platform. They also came with tradeoffs: external scripts, additional page weight, and $100 to $500 per month added to the tech stack
Shopify Rollouts changes part of that equation, but not all of it.
Rollouts is Shopify’s native A/B testing and staged deployment feature, announced in the Winter ’26 Edition and built directly into the admin.
Before you cancel your testing app subscription, read this. Rollouts cover more than most merchants expect and less than the announcement suggests. Where it lands relative to your current setup depends entirely on what you are testing.
What does Shopify Rollouts do?
Image taken from the Shopify Winter '26 editions
Until now, A/B testing required tools like Shoplift, Shogun, or Convert Experiences. These apps worked, but they also injected external scripts into your storefront, which often slow down your pages, increase your monthly costs, and run outside the core platform.
Rollouts lets you roll out multiple versions of your storefront, control which percentage of real visitors see that version, and measure the impact directly in the Shopify Admin.
No app install. No third-party scripts. No extra monthly cost.
How do you access rollouts?
Go to Markets > Rollouts in your Shopify admin. You can also access them from Online Store > Themes, where a Rollouts button appears next to your published theme.
Rollouts requires the new version of Markets to be enabled on your store. It is currently in early access and only available to certain stores. If you do not see it in your admin, you do not have access yet.
For example, I tested accessing multiple stores from our clients, and none of them had it available yet at the time of this post's latest update. (March 2026).
What can you split-test with Rollouts?
Rollouts work inside the theme editor, which means you can test any visual or structural changes to your storefront, including:
- Homepage layouts and hero sections
- Product page designs and information hierarchy
- Collection page structure and filtering
- Navigation menus and header/footer layouts
- Content, copy, and calls-to-action within theme sections
Each Rollout has its own dedicated analytics page in the Shopify admin, designed to help you measure the performance of your changes.
| You can test | You can't test |
|---|---|
| Homepage and hero sections | Checkout process |
| Product page layout and copy | Pricing and discount logic |
| Collection page design | Audience targeting |
| Navigation and header/footer | Custom event goals |
| Section level CTAs | Pop ups and widgets |
You can use Rollouts at the Market level
Rollouts lets you run separate tests per market. A different homepage for UK visitors versus US visitors, with analytics for each. If you sell internationally, this matters more than the A/B testing itself.
You could already customize the customer journey using Markets. Rollouts takes it further by showing you exactly what works in each region, using split-traffic data.
What you can’t test with Rollouts (yet)
Shopify has been straightforward about the current limitations. Rollouts cannot test:
- Theme settings, app embeds, or Liquid templates
- Product pricing or checkout flows
- Audience segmented variants (showing Version A only to returning customers)
- Custom conversion goals beyond Shopify’s standard analytics
Shopify has said they plan to expand Rollouts to more surfaces. No timeline has been confirmed.
Shopify also did not ship a public API with Rollouts. Third-party testing tools still have to run on the client side. If you need complex CRO with multiple variants, audience rules, and custom goals, you still need a dedicated experimentation platform.
Should you keep your testing app?
If your app’s main job is changing how your theme looks for a percentage of visitors, Rollouts replaces it. If it does anything involving pricing, checkout, segmentation, or custom analytics, it does not.
Here is the decision matrix:
| What you test | Verdict | Why |
|---|---|---|
| Pricing, shipping thresholds, checkout, or post-purchase flows | Keep your app | Rollouts have no pricing layer and no checkout access. |
| Audience segmentation, custom goals, or statistical reporting | Keep your app | Rollouts splits traffic randomly with no targeting. Analytics are limited to Shopify standard reporting. |
| Basic theme tests: layouts, hero sections, button placement, copy | Rollouts covers it | If splitting theme traffic is all you pay for, Rollouts does it natively at no extra cost. |
| You have never used a testing tool and make changes blind | Start with Rollouts | No setup cost, no scripts, no extra app. A meaningful step forward from publishing and hoping. |
| You are on a standard Shopify plan and Rollouts availability is a deciding factor | Wait | Shopify has not confirmed which standard plans include Plus. Do not cancel anything until the tier breakdown is published. |
Note: Shopify could update Rollouts to include more features. Third parties could build better solutions on top of it. Treat this as a snapshot, not a permanent verdict.
Where do you still need dedicated A/B testing tools
A/B testing tools on Shopify are still needed for:
- Pricing and shipping rate testing. Rollouts has no pricing layer. Testing whether a $5 price increase hurts conversion enough to lower revenue, or whether a free shipping threshold at $50 vs $75 drives higher AOV, still needs an app like Intelligems.
- Checkout testing. Testing cart page layouts, trust signals, upsells at checkout, or post-purchase flows. Rollouts does not touch checkout.
- Audience segmentation testing. Showing Version A to returning customers and Version B to new visitors. Segmenting by device, location, or traffic source. Rollouts splits traffic randomly with no targeting.
- Custom conversion event tracking. If your success metric goes beyond Shopify’s standard analytics, like “clicked the size guide” or “watched the product video,” Rollouts will not capture it.
- Pop-up, email capture, and widget testing. Exit intent offers, sign-up forms, and promotional overlays fall entirely outside the scope of Rollouts.
Last, rollouts also lets you schedule theme changes in advance. Set a Black Friday homepage to go live at midnight, run a flash sale banner for 48 hours and auto revert, or deploy to one market first and go global later. No more midnight deploys.
How many sessions do you need for reliable results?
This is where most merchants get Rollouts wrong. They run a test for three days, see a 5% lift, and push it to 100%. That is not a valid result.
A/B tests need enough traffic to reach statistical significance. Without it, you are making decisions on noise.
Here are rough minimums based on the size of the effect you are trying to detect:
| Detectable lift | Sessions per variant (approx.) | What this means |
|---|---|---|
| 20%+ lift | ~1,000 per variant | Large changes: full page redesigns, major layout shifts. Most small stores can reach this in 2 to 4 weeks. |
| 10% lift | ~3,800 per variant | Moderate changes: hero section swaps, CTA placement, content reordering. |
| 5% lift | ~15,000 per variant | Small changes: button color, copy tweaks. Most stores under $1M/year will struggle to reach this. |
These numbers assume a baseline conversion rate of around 2 to 3% and a 95% confidence level. Your actual requirements will vary based on your conversion rate and traffic distribution.
If you get under 5,000 monthly sessions, treat Rollouts results as directional. Run tests longer. Focus on large, high-impact changes where you are more likely to see a measurable difference.
Rollouts does not currently show confidence intervals or statistical significance indicators in its analytics. That is one of its biggest limitations for stores that want to run disciplined experiments. Until Shopify adds this, you will need to export your data and run the math yourself, or use a free significance calculator.
What a real testing workflow looks like (SimGym + Rollouts)
Before you run a Rollout with real traffic, there is a faster way to catch obvious problems. SimGym is a Shopify app that uses AI agents to simulate shopper behavior on your storefront. The agents are built from aggregated data across billions of transactions on the Shopify platform. It is still in research preview with a waitlist.
A word of caution. Research from Nielsen Norman Group and other UX practitioners has found that synthetic users exhibit idealized behavior and miss the messiness of real human decision-making. SimGym is useful for catching obvious friction. It is not a substitute for testing with real visitors.
That said, SimGym and Rollouts work well together as a two-stage process. Here is what that looks like in practice:
Step 1. You notice your product page conversion rate is lower than it should be. Set up a SimGym run against a revised layout. It flags two issues: the size guide is buried below the fold, and the add to cart button is low visibility on mobile.
Step 2. Fix both. Run SimGym again. Nothing new flags.
Step 3. Create a Rollout. Set traffic to 20%. Let it run for two to four weeks to gather enough sessions for a meaningful read.
Step 4. The revised version outperforms. Push it to 100%.
The order matters. SimGym handles the obvious issues cheaply. Rollouts handles the confirmation with real visitors.
Does your tech stack still make sense?
Rollouts does not replace a mature A/B testing program. But it does replace the basic layer most stores were paying for.
We see this across our client base at Shero. A store installed a testing app two years ago, uses it to split test homepage hero images, and never touches the advanced features. They are paying $200 to $400 per month for something Shopify now handles natively.
If your current app is doing nothing beyond splitting theme traffic, you are likely overpaying. Audit your testing stack the same way you audit your app stack: quarterly, with a clear ROI case for each tool.
If your testing app handles pricing experiments, checkout optimization, audience segmentation, or custom analytics, keep it. Rollouts is not a replacement for those capabilities.
The stores that will get the most value from Rollouts are the ones that never split tested at all. For them, this is the first time they can make data-informed decisions about their storefront without adding another app to the stack.
If you want to know exactly where your current stack overlaps with what Shopify now handles natively, book a free consultation with our solution architects.
