Conversion Rate Optimization & A/B Testing

I feel as marketers, we’ve become a little sloppy with our marketing jargon! Most of these slip-ups are harmless, but there is one in particular that needs to be cleared up immediately:

A/B Split Testing & Conversion Rate Optimization (CRO) are not the same thing!

Prior to the rise in A/B Split testing, often referred to as A/B Testing, Split Testing, Testing, etc…, there was confusion between split testing and site functionality testing. Thankfully that verbiage mix up is well behind us with the rise in testing’s popularity.

A recent study by RedEye and Econsultancy found that 67% of companies are currently A/B testing and this is the fourth year in a row that testing was the #1 CRO activity. So it’s safe to say A/B split testing has hit the mainstream.

After the rise of A/B split testing, more buzzwords became prevalent in the community. One in particular is CRO. CRO is an umbrella term for all of the tools and methodologies marketers use to optimize their sites and campaigns. Pretty much every marketing tactic can fit under the CRO umbrella including personalization, web data analysis, eye tracking studies, user surveys, and even A/B split testing.

Why The Mix Up Is Problematic

I’ve met countless people who have said ‘I just don’t have the traffic for optimization’ and they continue to use the same old conversion-leaking site. Sure, they probably didn’t have enough traffic to run a traditional A/B split test, but that doesn’t mean they can’t optimize their site.

If you’re curious about how much traffic you need to run a test, check out this post on imaginary lifts and why sample size matters. If you already know how much traffic it takes, but want to know how long to run a test check out this test duration tool.

Yes, optimizing a page without testing can’t give you the data insights that A/B split testing provides practitioners. However, It’s important to make this fact very clear:

Just because a page is not test-worthy does not mean it is impossible to optimize!

CRO is about using tools and methods to squeeze out the most conversions from your website. Testing is about verifying these tools and methods.

Remember - when you can test, do it! Just because you have a hunch something will work, doesn’t mean it will.

Identify, Prioritize, Hypothesize, Optimize, Verify

CRO, much like testing, is all about a detailed strategy. That is another reason I think the two terms are used interchangeably, but now we know they are two different things altogether. In fact the two methodologies work with one another.

First you have the identification stage, a two-step process. The first step is to use all the data you have at your fingertips to find pages worth optimizing. The second step of the identification process is to figure out if these pages are test-able. Make a note about the page’s testing ability, and move on to step two.

I highly recommend creating an internal prioritization scale. Include factors such as conversion lift potential, ease of development, how important the page is to your overall strategy, and finally whether the page is testable. Note: if your optimize a non-testable page, Page A, while there is a test running on Page B, make sure to exclude the traffic from Page A to Page B.

Whether the page is testable or not, you can move onto the hypothesis construction phase. Simply put, your optimization goals should be put in the format ‘We believe that changing A for visitors B will make C happen.’ If you can’t put a hypothesis in this format, go back to the drawing boards!

The optimize step is where you either run your test or implement your new optimization strategy. From here you move on to ‘Verify’. In the case of A/B split testing, this is where you analyze the results. For the un-testable pages, this is where you look at how the new page is performing. Remember, to keep external factors such as time and traffic sources in mind when verifying un-testable pages!

If you take one thing away from this post, I hope it’s this:

A/B split testing is a part of your CRO tool kit, not the tool kit itself!