Getting Started

Your First Test

It’s super easy to get started with A/B Test Master. In this guide, we will walk you through a scenario and make use of various features.

Our Scenario

We have an ecommerce website and would like to run various split tests on different steps of checkout process in order to increase conversion rate.

For the purposes of this tutorial, we assume everything happens in a PurchaseController which looks like this:

public class PurchaseController : Controller
{
    public ActionResult Cart()
    {
        return View();
    }

    public ActionResult Payment()
    {
        return View();
    }

    [HttpPost]
    public ActionResult Confirm(PaymentData data)
    {
        return View("Receipt");
    }
}

There are three pages: Cart, Payment, and Receipt. The goal is for the visitors to navigate to Receipt page and our task is to create variants of Cart and Payment pages and compare the conversion rate.

First Things First

  • - Add A/B Test Master to your ASP.NET MVC project via the Nuget Package.
  • - Change PurchaseController to inherit from AbTestMasterController

Your First A/B Test

Let's say we have craeted alternative variants of Cart and Payment views. Now all we need to do is to set up A/B tests is:

               
public class PurchaseController : AbTestMasterController
{
    [SplitView("Version 1", "ShoppingCart")]
    public ActionResult Cart()
    {
        return View();
    }

    [SplitView("Version 2", "ShoppingCart")]
    public ActionResult Cart2()
    {
        return View();
    }

    [SplitView("Version 1", "Payment", .1)]
    public ActionResult Payment()
    {
        return View();
    }

    [SplitView("Version 2", "Payment", .9)]
    public ActionResult Payment2()
    {
        return View();
    }

    [HttpPost]
    public ActionResult Confirm(PaymentData data)
    {
        return View("Receipt");
    }
}

So, Cart and Cart2 will be shown to users alternatively. Whenever you call either of them, there is a 50% chance of each one being displayed. It's the same for Payment, except we have explicitly specified the ratios: 10% for version 1 and 90% for version 2.

Please note that returning users will keep getting the same view as we do not want to confuse them by showing them multiple versions of the same page across different visits. The ratios, as a result, will only apply to the new users.
Set up a goal

A common way of tracking the success of a split test is to set up goals. In our example, the goal is for Confirm action method to be called. All we need to do is annotate it.

public class PurchaseController : AbTestMasterController
{
    [SplitView("Version 1", "ShoppingCart", "Checkout")]
    public ActionResult Cart()
    {
        return View();
    }

    [SplitView("Version 2", "ShoppingCart", "Checkout")]
    public ActionResult Cart2()
    {
        return View();
    }

    [SplitView("Version 1", "Payment", "Checkout", .1)]
    public ActionResult Payment()
    {
        return View();
    }

    [SplitView("Version 2", "Payment", "Checkout", .9)]
    public ActionResult Payment2()
    {
        return View();
    }

    [SplitGoal("Checkout")]
    [HttpPost]
    public ActionResult Confirm(PaymentData data)
    {
        return View("Receipt");
    }
}

You will notice two sets of differences:

  • - Confirm action method is now annotated with SplitGoal filter.
  • - All split views now have an additional "Checkout" parameter, indicating they are all part of the same goal.

Having done all of that, we have a set of A/B tests and a goal. Next natural step would be to record the data and decide which multivariate path is converting best.

Define Targets

Targets are our way of recording the split view and goal data. When installing Nuget package, two csv files are added to App_Data by default: AB_TEST_MASTER_SplitViews.csv and AB_TEST_MASTER_SplitGolas.csv. You can have a look at these files yourself to see what information is saved in them. An ABTestMaster config section is also included in web.config file instructing the framework to write the data to these files.

You are not restricted to use these files. You can specify what data you would like written to targets. All of this can be done in AbTestMaster config section in web.config. There are two types of targets:

  • - csv file
  • - sql server database
If you have used NLog before, you will notice a lot of similarities in defining targets in config section. This is exactly the case because we took a lot of inspiration from NLog to write AbTestMaster config section!

Below is an example of configuration to write the split view data in csv and split goals in database.

<abtestmaster>
  <targets throwexceptions="false">
    <target name="fileviews" type="file" data="views" path="\App_Data\AB_TEST_MASTER_SplitViews.csv">
      <parameter name="@datetime" value="$currentdatetimeutc" />
      <parameter name="@group" value="$splitgroup" />
      <parameter name="@name" value="$splitname" />
      <parameter name="@sequence" value="$splitgoal" />
    </target>
    <target name="dbgoals" type="database" data="goals" connectionstringname="myConnectionString" commandtext="INSERT INTO AbGoalData (SplitGoal, SequenceTrail, Browser, UserAgent, IpAddress, CurrentDateTime) values(@goal, @sequence, @browser, @agent, @ip, @datetime)">
      <parameter name="@goal" value="$splitgoal" />
      <parameter name="@sequence" value="$splitsequencetrail" />
      <parameter name="@browser" value="$browser" />
      <parameter name="@agent" value="$useragent" />
      <parameter name="@ip" value="$ipaddress" />
      <parameter name="@datetime" value="$currentdatetimeutc" />
    </target>
  </targets>
</abtestmaster>

Depending on the type of target element, different attributes need to be provided:

Attribute file database Allowed Values
type "file"
"database"
name Any string value
data "views"
"goals"
path Any valid/accessible path
connectionStringName An existing SQL server connection string name
commandText Valid SQL server statement. Should be INSERT.

What can we write to Targets?

What you can write to the target depends on waht data you are recodring (i.e. "goals", or "views"). The table below lists all the possible variables:

variable views goals description
$browser Visitor's browser
$useragent Visitor's user agent
$ipadderss Visitor's IP Address
$currentdatetimeutc Date Time in UTC of when view/goal was accessed.
$splitgoal Goal name as specified in the action filter
$splitname Name of Split view as specified in the action filter (e.g. "Version 1")
$splitgroup Group name of Split view as specified in the action filter (e.g. "ShoppingCart")
$splitsequencetrail List of the goal's split views visited by the user before reaching the goal.