A/B Testing Without IT; Is It Really Possible?
By Hossein Tayebi | Jun 28, 2014
I get asked this question quite a lot. Mostly from product managers who prefer to run tests themselves rather than rely on development teams who are expensive, slow and sometimes annoying (I can say that because I am a software developer myself). I have also come across product owners who ask for my services; casually mentioning that no IT resources will be available. After all, they have signed up to Optimizely for this exact reason. Doesn’t it suffice?
My answer almost always annoys product owners: "it depends".
To help clients answer this pesky question, I usually ask them a series of questions:
- Purpose: what are you trying to achieve?
- Target: which pages are you testing?
- Test Types: what are you (or should you be) testing?
- Verdict: do you need IT?
Let’s dive in then.
People don’t normally start A/B testing for the sake of it. Conversion rate optimization seems to be the most common reason: to get more people to do x, to reduce bounce rate, to increase engagement rate, etc. Some startups use A/B testing to test the waters as well. They run tests to measure visitor behavior when presented with two products. One of my clients was a psychologist trying to gauge user reaction to different images. Whatever the purpose is, it’s important to clearly define it.
I worked with an e-commerce website selling auto parts (pseudo name AutoPartsCo). They were not happy with their conversion rate of 0.3% and were wondering if it could be improved. Before jumping to testing, I analyzed visitor flow on their website and asked what would be their ideal scenario (i.e. purpose).
Once purpose is clearly outlined, it’s time to narrow down the tests to a few pages. This might include removal and modification of existing pages or addition of new pages. Each test could target a single page or it could be a part of a sequence of pages in the funnel.
In the example of AutoPartsCo, I realized home page had an unusually high bounce rate (62%). Product details pages also seemed to be doing really badly even though prices on the site were quite competitive. The third page selected was cart page as abandonment rate was also quite high.
By now, you should know what the problem is and what pages need adjustments or overhaul. What tests should be run on these pages? This is where I have seen most companies fail. They select a series of (mostly random) elements in pages and create variations for them. Needless to say they end up with inconclusive results and minimal changes in user behavior. Answering this question correctly not only answers the question we asked at the beginning of this article, but also leads to meaningful data. It is beyond the scope of this article to define the types of tests required to achieve a successful and meaningful A/B testing campaign, but you can refer to the resources below to gain some insight into this half art half science.
Once your analysis of which test types to run is complete, they should be categorized into one of the followings:
- - Micro Refinements
- - Major Modifications
Micro refinements are small changes that do not affect the layout, function or purpose of a page. It’s merely changing the hero image, headline text, or color of buttons. In case of AutoPartsCo, these types of changes were made to product details page. Multiple alternate background image, font color and size, add to cart button size, color and text and a few icons were tested (multivariate testing).
Obama 2008 presidential campaign website is a famous and classic example of where micro refinements can be useful. In order to increase donations, hero image and button text of splash page were split tested. Below you can see a few variations.
images from optimizely blog
Major modifications are anything beyond micro refinements. What if we want to a/b test two completely different page layouts? What if there are different functions in each test? What if our test is comparing search box with or without autocomplete?
In case of AutoPartsCo, that’s what was done for home page. After some usability tests and having done some research on target audience, I formed the hypothesis that existing home page was not focusing on what visitors expect. As a result, a few alterations of the page were created from scratch. In most of them, image carousel was either removed or moved to a less significant part of the page. Browse products became a major part of the page and search box was introduced. These changes required change of layout and functions.
Little Green Light (a provider of CRM for donation sites) was trying to come up with a new home page and they came up with the following versions:
image from little green light blog
Indeed these pages present visitors with different functions and are well beyond micro refinements.
Hopefully by now, you have a pretty good idea of whether you need IT to run A/B tests or not. If micro refinements are all you need, it makes sense to use a tool like Optimizely. It is perfect for such scenarios and requires minimal involvement of IT team. And if you settle down on a winning variant and do not intend to a/b test it for a while, you can ask IT to apply those changes to the code base so that becomes the original version of your site. If some or most of your tests are categorized as major modifications, IT is indeed required and use of tools like Optimizely is not only of little value but also inefficient. In these cases, you would want to find a tool that makes developers’ life easier by streamlining the process and speeding up the cycle while providing user-friendly analytics for non-technical users such as product managers.
Speaking from experience, not-for-profit websites, where the funnel is simple and options are limited, can benefit the most from tools without IT. Websites that offer goods and services or want to generate leads normally require a more exhaustive process of conversion rate optimization and hence rely more heavily on IT to achieve meaningful results. AutoPartsCo was among the second group and its conversion rate jumped to %3.9 (130 times higher!) after four months of hard work.
Whether you need your software development team to run a/b tests or not very much depends on what you are trying to achieve and what steps are required to get there. Through a series of questions, you determine why you are running a/b tests in the first place, what parts of your site need modification and what test types are required to achieve the goal. If test types are mostly categorized as micro refinements, you could get away without involving IT as there are online tools that help you run such tests. For all other changes (major modifications), you definitely need you software development team and should choose a tool that facilitates a/b testing for developers while providing useful analytics for product managers and stakeholders.
Do you have any experience in split testing? What test types did you try? What tools did you use?