Improving eCommerce Conversion Rate – Why Classic A/B Testing will Only Take You So Far

Posted on August 24, 2018

While A/B testing is a valuable tool in the arsenal of any ecommerce company, A/B testing, at least in the traditional sense, has limited utility for smaller stores.

For data-driven ecommerce managers, this may sound like sacrilege, but please hear me out.

At my last business, I ran one of the highest-volume ecommerce A/B testing teams on the planet.  Our store had around $100 million in annual revenue and around 10 million monthly visits. At our peak, we had a team of six designers and developers capable of running three or four A/B tests A DAY.  In the beginning, we were successful at finding small wins that, over time, dramatically increased conversion rate. In fact, we were so good at A/B testing, that our team lead was hired by Apple to build the first A/B testing team for!

Testing small things like button colors, homepage images, and font size certainly has their place for incredibly high volume stores like ours, but for the vast, vast majority of sites these tests will be inconclusive and will result in a massive waste of time and effort.  For a smaller store with say $1 or $2 million in revenue, there’s simply not enough traffic to run A/B tests around minutia – achieving statistical significance could take months or even years! As we found out first hand, testing iterative changes eventually result in very little improvement even for high volume stores.

We rapidly approached the “local maximum” of conversion rate, and winning tests became rare as hen’s teeth even as we ran nearly 100 tests a month.  Simple iterative changes no longer provided improvement, so we shifted our mindset to more radical, innovative, business model-level changes.

For a low to mid volume store, ignore the minutia.  Instead, time should be focused on testing potential transformational business model-level items such as the following:

  • Sell a consumable?  Test subscriptions like Dollar Shave Club or Quip.
  • Do you collect email addresses?  Test email-gating the entire site like Zulily.
  • Offer free shipping?  If not, test it. If you do, test the free shipping order value threshold.
  • Do you offer discounts?  If not, test them. If so, test %-off vs $$-off vs free shipping.
  • Do you have abandoned cart emails?  If not do them! If so, test cadence, timing, discounting, and frequency.  
  • What about email marketing?  Revamp your entire email game using event-driven flows – increasing your CPM by 1000% is possible!
  • Do you have a multi-page checkout?  Eliminate steps, testing single page checkout.
  • Do you do retargeting/remarketing?  If not, start doing it. Here’s how.

In addition to testing only high-leverage, more innovative items, be sure you use the correct data-driven approach to determine the winning variant.  Our article on building a $100 million ecommerce business using simple math provides a framework for determining (and improving) your value per visitor.

Unfortunately, many ecommerce platforms make testing large-scale paradigms like the above difficult.  To move to the next level and increase your conversion rate, you may need an even more radical change – an entirely new ecommerce platform.

Want us to help?  We specialize in taking ecommerce stores to the next level.  Email us –