Split Testing Blog

The Developers’ Ultimate Guide to A/B Testing Tools and Software

By Hossein Tayebi | May 20, 2014

A/B testing helps you define alternate versions of your website and observe how users interact with the interface. If done properly, it could help lead start-up to gain an insight into your potential clients, or maybe increase your sales; it could do wonders.

As a developer, you could implement a bespoke framework to help you get there and we completely support that (after all, that’s what we’ve done). But if for any reason you decide to look at the community, you won’t be disappointed as there are several options. When choosing the A/B testing tool, you need to define your project parameters to select the one that is right for you. The following factors is a good starting point.

  • Product technology: Tools are built on different platforms. Choose the tool that supports the technology you use.
  • Type of tests: Some tools support testing only a single variable while some support multivariate testing.
  • Results and data storage: Some tools provide an attractive Dashboard while others provide only the raw data and do not provide data visualization.
  • Integration with third party software: Some tools integrate with Google Analytics and allow you to view results on the Analytics Dashboard.
  • Score type: Different tools use different statistical computations. Select one that is more appropriate for your tests.
  • Ease of installation
  • Cost

So before you begin evaluating A/B testing software, it might be a good idea to prepare a brief list outlining your project and test specifications.

Now let’s have a look at the list.

Ruby on Rails

1. A/Bingo

A/Bingo is an open source tool for interfaces developed with Ruby on Rails. It supports Rails 2.3 and 3.0. Testing but does not support chi-square test for result significance.

Works as a plugin
Supports multivariate testing
No response limitations
No data visualization support


2. Vanity

Vanity is another A/B testing tool that you can use for your Ruby on Rails applications. The tool supports versions of 2, 3, and 4 and is compatible with Redis, Mongo, and SQL databases. The default configuration works well if you are using Redis, for others you will need to tweak the configuration.

Allows multivariate testing
Allows Google Analytics Integration
Shows the same interface to the user across multiple visits
Good Documentation Available
Shows results for top two performers only


3. FluidFeatures

FluidFeatures is an open source Ruby on Rails gem that you can install with a single command. The framework is free to use and open to customization.

One line installation
Controlled rollout
Supports multiple versions
Dashboard for easy access to settings
Shows the same interface to the user across sessions
Development inactive
Online dashboard not in production anymore


4. Split

Split is one of the most current open source, free Rack based application and works with Ruby and other Rack based applications. It can work with Ruby 1.8, 2.3, and 3.0. Split has 13 branches and over 60 contributors.

Easy to install
Supports multivariate testing
Supports controlled rollout
Web-based dashboard
Extensionsible to export data to file or Google Analytics
Needs Redis 2.0 and later
Only shows result in Boolean terms
Experiment results with significance below 90% are shown only as “insufficient significance.”

PHP

5. PHP Scenario

PHP Scenario is an object oriented split testing library written for applications using PHP 5.2 and later. The tool is designed to function with minimum amount of code but still packs the necessary power you need.

Comprehensive documentation for developers
Supports PDO and Zend frameworks
Easy test setup
Does not support multivariate testing


6. PHP A/B

PHP A/B, as the name suggests is a PHP based, server-side A/B testing framework. The framework is free for personal and commercial use. The framework supports PHP version 5.0 and later and requires a Google Analytics account to record data.

Good integration with Google analytics
Ignores bots to keep the data accurate
Allows testing multiple variables
Does not work without Google Analytics. Cannot export result data elsewhere.
Limited documentation

JavaScript

7. Genetify

Genetify is an open source JavaScript library. You can use it as a service with minimal development or install the service on your server.

Easy to set up and use
Persistent interface across user sessions
Can affect page performance and cause flickering.
Has issues with IE<10


8. EasyAB

EasyAB is a JavaScript based open source plugin designed to help the tech savvy entrepreneur build better landing pages. Users with cookies disabled will always see the default version of the webpage.

Easy to set up
Supports multivariate testing
Google Analytics integration
Does not work without Google Analytics. Cannot export result data elsewhere.
Very little documentation


9. ABBA

ABBA is an open source JavaScript testing framework with a great API. Developed for internal use at Stripe, it is now available for the public to customize to their liking.

Easy installation
Data stored locally
Result visualization
Supports multivariate testing.
Requires Ruby 1.9.3 and MongoDB
Needs knowledge of all the technologies involved

ASP.NET

10. FairlyCertain

FairlyCertain is an open source library for ASP.Net that allows you to implement tests straight from the IDE with a single line of code. The framework has no external dependencies.

Shows if the results are statistically significant
Easy online code implementation
No data storage mechanism. Developer's responsibility to store data
No third party integration


11. A/B Test Master

A/B Test Master is a flexible open source testing framework for ASP.Net MVC projects. The tool requires .Net framework 4.0 or later and ASP.Net MVC > 3 to run. A/B Test Master uses action filters and allows you to save results in a format of your choice.

Easy installation via Nuget package manager
Supports multivariate testing
Persistent interface across user sessions
Result data can be stored to sql database, file or webservice.
Control variant ratios.
No third part integration
Advanced features (visitor segmentation, online dashboard available only in paid versions)


Whatever platform or technology you are working with, you are sure to find a tool that fits your needs from this list.

Have you tried any of these before? Or do you know any other framework?

About The Author

Author
Hossein Tayebi is a software engineer, web development enthusiast and an entrepreneur based in Melbourne, Australia. You can get in touch with him via his , Twitter or blog.

Comments (1)

  • User Avatar
    tarik Apr 30, 2015 11:40 AM

    Thank you for the article but sometimes we need to roll our own solution, it would be helpful if you write an article about how we can implement a such gem or plugin!

Add Comment

Share This Story

RECEIVE UPDATES

ABOUT THE BLOG

In this blog, you will find useful information on A/B testing, split testing and conversion rate optimization in general and how you can improve or kickstart your business using these techniques. Our posts are free and we are always interested to hear from you.