Podcast: Download (Duration: 51:23 — 59.1MB)
In this episode of the My Wife Quit Her Job podcast, I sit down with Shopify consultant Kurt Elster of Ethercycle to unpack the counterintuitive Shopify split test results that actually move revenue. The short version: removing the hero image beats keeping it on most stores, dropping free shipping often makes more profit than offering it, and hiding prices on category pages can lift both conversion and average order value.
Kurt runs the Unofficial Shopify Podcast and has spent years testing what real shoppers respond to versus what brand owners assume they will. Most of the wins in this episode come from taking things off the page rather than adding them, and from testing entire template variants instead of button colors.
Below I have organized the most useful Shopify split test findings from our conversation into a structure you can steal, run, and measure against your own store.
Get My Free Mini Course On How To Start A Successful Ecommerce Store
If you are interested in starting an ecommerce business, I put together a comprehensive package of resources that will help you launch your own online store from complete scratch. Be sure to grab it before you leave!
Table of Contents
Key takeaways
- Free shipping is not always the profit winner. In four separate stores Kurt tested, no free shipping produced the highest profit per visitor, roughly $20 vs. $16.50 at a $75 threshold and $13 at $25.
- Removing the category page hero image improved conversions on an apparel store for both mobile and desktop, and for both new and returning visitors.
- Hiding prices on category pages lifted both conversion rate and average order value for a sub-$100 AOV apparel store.
- “Recently viewed items” cut conversions 9% for new visitors and lifted them 33% for returning visitors on the same site.
- You need roughly 100 to 150 conversions per day and a minimum 14-day run window to trust a split test result.
- Test entire redesigned templates against the original, not single-element tweaks. Big changes with a clear hypothesis produce statistically significant results far more often.
What is Shopify split testing and when is it worth doing?
Shopify split testing is running two or more versions of the same page or element against live traffic to see which drives more conversions, revenue, or another chosen goal. It is worth doing once your store gets at least dozens of orders per day, and it becomes genuinely useful at 100 to 150 orders per day.
Kurt was resistant to split testing for years and leaned on user testing, heat maps, and surveys instead. That approach works, and it still does. The shift happened when clients started hitting the order velocity where statistically significant results were possible in about two weeks.
The other unlock was tooling. Enterprise tools like Optimizely and VWO were powerful but expensive and hard to use. Free tools that integrate directly with your existing analytics collapsed the barrier to entry for most Shopify merchants.
How much traffic do you need to run a Shopify split test?
You need dozens of conversions per day at a minimum, and 100 to 150 per day is where testing starts to feel comfortable. Even with enough traffic, plan on running each test for a full 14 days so you capture a complete pay-cycle window, since most shoppers get paid every other Friday.
Early on I made every classic mistake. I would see a “significant” result on day two and want to call the test right away. The tool would tell me not to, and it was right. Anyone announcing test wins after 48 hours is showing you noise, not signal.
Two weeks is the honest minimum for any test you plan to act on. If your traffic will not support that, either wait until it does or run tests only on your highest-traffic templates.
How to set up a Shopify split test at a high level
At a high level, connect a testing tool to your existing analytics, drop a single JavaScript snippet in the site head, and use the tool’s visual editor to build a variant of an existing page. Then pick a goal (conversions, revenue, page views, or bounce rate), pick your audience segmentation, and let it run for 14 days.
The visual editor lets you click elements on your page, remove them, rename text, or inject custom CSS. If you know enough CSS to be dangerous, you can restyle almost anything without touching the theme. If you do not, removing elements is the fastest place to start.
Segmentation is the second important lever. You can restrict a test to mobile only, desktop only, new visitors only, or returning visitors only. Some tests that look inconclusive across all traffic reveal clear wins once you split by device or visitor type.
Why removing the category page hero image usually wins
Removing the large lifestyle hero image at the top of a category page usually lifts conversions because it pushes products above the fold. Kurt tested this on an apparel site where aesthetics are core to the brand and the site performed better without the hero image for both mobile and desktop, and for both new and returning visitors.
The fold is still real, at least as far as revenue is concerned. Every pixel of hero image above your product grid is a pixel of purchase intent you are pushing down the page. Getting products in front of the visitor as quickly as possible is a durable win across most stores.
This holds true for blog posts too. If your post is about the top 17 ways to optimize a site, put the top 17 ways above the fold. Some readers skim the intro, decide it is filler, and bounce before they ever reach your list.
Should you hide prices on Shopify category pages?
Hiding prices on Shopify category pages is a must-test. On one sub-$100 AOV apparel store, hiding category prices lifted both conversion rate and average order value. On other stores it hurts, so treat it as store-specific rather than a universal best practice.
The likely mechanism is pricing psychology. When a shopper sees a product in the collection grid, they attach an expected price to it before they see the real number. If the actual price is above their expectation, conversion drops. If it is below, conversion rises.
A luxury brand like Tiffany hides prices entirely on category pages, and it works for them because the click through raises commitment. Someone who has clicked into a product page is far more likely to accept the price than someone who scanned it in a grid. Test it before you decide either way.
Free shipping split test results: no free shipping often wins on profit
Kurt has run the free shipping split test on four separate stores and no free shipping consistently produced the highest profit per visitor, even though it produced the lowest checkout conversion rate. The tradeoff between conversion and profit tips in favor of no free shipping more often than most merchants expect.
Here is the actual data from one store with a $7.50 fulfillment cost per order:
| Free shipping threshold | Checkout conversion | Profit per visitor |
|---|---|---|
| $25 threshold | ~80% | $13.00 |
| $75 threshold | ~75% | $16.50 |
| No free shipping | ~70% | $20.00 |
The checkout conversion story is exactly what you would predict. A low $25 threshold converts best, a high $75 threshold converts a little less well, and no free shipping is the worst. Profit tells the opposite story because every dollar of shipping you absorb comes out of margin.
Not offering free shipping also frees you up to use it as a promotional lever later, instead of leaning on price discounts or new product launches. If you do offer free shipping, set the threshold above your current average order value so the offer actively nudges people to add another item.
Font size and readability: test the whole readability system, not just the font
Font size split tests almost always come back inconclusive because font size alone is not what drives readability. What matters is the full readability system: line length of 60 to 80 characters, adequate line height, and strong contrast between text and background.
The Baymard Institute publishes readability guidelines that outperform bigger fonts alone. When Kurt tested a site with the previous font versus a full Baymard-compliant readability treatment, bounce rate dropped noticeably.
On my own store at Bumblebee Linens, a lot of our customers are over 55 and farsighted. We were getting emails complaining that the text was too small and people were calling in orders because they could not read the site. We enlarged the fonts on mobile and desktop, the calls stopped, and the site converted better.
The “recently viewed items” test: new vs. returning visitors respond very differently
A recently viewed items widget cut conversions by 9% for new visitors and lifted conversions by 33% for returning visitors on the same site. This is why segmenting split tests by new versus returning visitors matters as much as segmenting by device.
The best explanation is choice paralysis. A new visitor is still figuring out what your store is and what they want, and extra recommendations become distractions. A returning visitor already knows what they came back for, and the widget helps them get to it faster.
The fix is not to remove the widget. The fix is to show it only to returning visitors and hide it from new ones, which most testing tools can do with a personalization rule.
The homepage hero image and slider test: shopping-only pages often win
Removing the homepage hero image or slider often lifts conversion rate because it pushes shoppable categories and products above the fold. On one live Shopify test Kurt was running, a shopping-only version of the homepage converted at 2.3%, a version with the hero slider removed converted at 2%, and the original page with all the content converted at 1.25%.
Brand owners consistently overestimate how well their hero slides communicate what the store sells. A new visitor who cannot immediately tell what you offer will leave, and a slider that scrolls every four seconds does not solve that problem.
The exotic version of this test is to hide the hero and replace it with nothing. Just let the categories and products sit at the top of the page. It scares almost every brand owner, and it wins on almost every store where I have seen it tested.
What to test first on Shopify: cart, product, category, homepage
Start with the cart page, then work backward to product pages, category pages, and finally the homepage. The cart is the easiest place to start because the next-step goal is clear (proceed to checkout) and the page has usually accumulated cruft over the years.
Here is the practical order of operations Kurt recommends:
- Cart page: Strip cruft like on-cart shipping calculators and prominent coupon code fields. Both often hurt because they either duplicate work the checkout already does or send shoppers off looking for coupon codes.
- Product page: Focus on the product form above the fold. Try backgrounds and borders to draw attention. Remove elements that are not serving a purpose.
- Category page: Test hiding prices, removing the hero image, changing the filter sidebar, and shortening or hiding the category description.
- Homepage: Test a shopping-only version. Remove the hero, remove blog callouts, remove anything that is not directly shoppable and measure the impact.
Cart cruft is where the fastest wins live. A shipping calculator on the cart page just does the checkout’s job twice. A visible coupon code field sends bargain hunters off to Honey and RetailMeNot, and many of them never come back.
Test whole templates, not individual elements
Testing one element at a time is the fastest path to inconclusive results. Testing entire template variants with 5 to 10 coordinated changes and a clear hypothesis is where you actually get statistically significant answers.
The middle ground is redesigning one template at a time, then comparing the redesign against the original. You lose the ability to attribute the win to a single element, but you gain the ability to see wins at all. Over enough tests, a picture of what your audience prefers emerges.
Most successful split tests come from removing things rather than adding them. Every element on the page should serve a purpose, and split testing tends to expose the ones that do not in stark, unemotional data.
Ugly sites often convert better than beautiful sites
Ugly, straightforward sites frequently outperform beautiful, polished sites on revenue. The highest-revenue store Kurt has ever worked on was also the ugliest, and stripped-down sites that look almost broken continue to print money for their owners.
Good design is not a requirement of high revenue. Clear communication, fast product access, and trust signals are. If your beautiful hero image is pushing your products below the fold, your design is losing you money.
The mastermind story I shared on the episode is a real example. I redesigned my Bumblebee Linens site because a mastermind group told me it was ugly, and while the redesign changed 20 things at once, it converted about 50% better. The takeaway is not that beauty converts. The takeaway is that a site that gets out of the way converts.
Frequently asked questions
What is the best Shopify split testing tool for beginners?
The best free Shopify split testing tool for beginners is one that integrates with your existing Google Analytics setup, uses a single JavaScript snippet, and offers a visual editor for building variants. For shipping-specific tests on Shopify Plus, IntelliGems and Ship Scout are the standard picks.
How long should a Shopify split test run?
A Shopify split test should run for at least 14 days, even if statistical significance appears earlier. Two weeks captures a full pay-cycle window and smooths out weekday-versus-weekend behavior. Calling a test after two or three days almost always means you are acting on noise.
How many orders per day do I need to run a split test?
You need dozens of orders per day at a minimum, with 100 to 150 orders per day being the practical comfort zone. Below that, you can still test, but you will run fewer tests per year and hit inconclusive results more often. Below a dozen orders per day, focus on user testing, heat maps, and customer interviews instead.
Should I offer free shipping on my Shopify store?
Not necessarily. Testing across four separate stores showed that no free shipping produced the highest profit per visitor, even though it produced the lowest checkout conversion rate. If you do offer free shipping, set the threshold above your average order value so it nudges shoppers to add another item rather than subsidizing purchases they would have made anyway.
Should I hide prices on Shopify category pages?
Hide prices on Shopify category pages only if you have tested it against showing prices for your specific store. On one sub-$100 AOV apparel store, hiding prices lifted both conversion rate and average order value. On premium stores with strong brand equity, hiding prices often works because the click-through raises purchase commitment. On stores where shoppers expect transparency up front, hiding prices tanks conversion.
What is the single easiest Shopify split test to start with?
The single easiest Shopify split test to start with is a cart page cleanup test: original cart page versus a stripped-down version with the shipping calculator and any non-essential fields removed. It is easy to set up in a visual editor, the goal is clear (proceed to checkout), and cart pages tend to accumulate more cruft than any other template.
What is the biggest split testing mistake merchants make?
The biggest split testing mistake is testing single small elements like button color and calling tests after two or three days. Small-element tests almost always come back inconclusive, and early results are almost always noise. Test whole template variants with a clear hypothesis, and give every test a full 14-day window.
I Need Your Help
If you enjoyed listening to this podcast, then please support me with a review on Apple Podcasts. It's easy and takes 1 minute! Just click here to head to Apple Podcasts and leave an honest rating and review of the podcast. Every review helps!
Ready To Get Serious About Starting An Online Business?
If you are really considering starting your own online business, then you have to check out my free mini course on How To Create A Niche Online Store In 5 Easy Steps.
In this 6 day mini course, I reveal the steps that my wife and I took to earn 100 thousand dollars in the span of just a year. Best of all, it's absolutely free!
