A/B Testing

You can send variants of a message to cross-sections of your audience using A/B Tests; each variant in the test returns analytic data to help you determine the most effective way to engage your audience. A/B tests support up to 26 variants and a control group per test.

Planning your A/B Test

Before you begin an A/B test, you should know the parameters of your experiment:

  1. Have an audience in mind that you want to test — including the control group. The control group will not receive a message.
  2. Have a control message — a message that represents common communications with your audience, that you can expect to return stable results. A control message can help you understand baseline message performance.
  3. Understand the changes or variants you want to test in your message. How can you modify your message, and which modifications do you think will increase engagement? You may want your variant messages to differ only slightly so that you can pinpoint the specific strategies that increase engagement — like link placement or the tone of your message. Your test can include up to 26 different variants.

Audience Groups in the A/B Test

When you create an A/B test, you select an audience — this is the complete audience of users in your A/B test.

The control group is a selection of your audience that will not receive a message. The control group reduces the total audience by a specified proportion. For example, if your control group is set to 20%, the remaining 80% of the audience receives your message variants as a part of the test. The control group represents baseline behavior; they do not get or open messages, so they cannot have a positive or negative response.

Your remaining audience receives a message variant. By default, your remaining audience is evenly proportioned; each variant goes to the same number of users. However, if you create your test through the /experiments API, you can assign weights to variants if you want a particular message variant to go to a larger percentage of your audience.

Audience Groups in the API

When you set up A/B Tests using the /experiments API, your audience is split across the variants in your message by weight properties. You can also set a control group.

The control group is a decimal (float) between 0 and 1 representing the portion of your audience who will not get a message. The remainder of your audience (after the control group is subtracted) receives messages according to their weight. If you don’t set weight properties, Airship splits your audience evenly across your variants.

Airship adds the weight properties in your payload and divides the total by an individual weight to determine the proportion of the audience that receives each variant. For example, if you set weights of 10, 10, and 5 for your variants, Airship splits your audience proportionally into subsets of 40%, 40%, and 20%.

VariantWeightAudience Percentage
A1040%
B1040%
C520%

Example experiment with control and weights
{
    "name": "Experiment 1",
    "audience": "all",
    "control": 0.2,
    "device_types": "all",
    "variants": [
        {
            "push": {
                "notification": {
                    "alert": "You're in a cool group"
                }
            },
            "weight": 20
        },
        {
            "push": {
                "notification": {
                    "alert": "You're in the coolest group"
                }
            },
            "weight": 40
        }
    ]
}

Interpreting A/B Test Results

Airship reports the results of your A/B test so you can discover which variant performed the best and modify your messages going forward to increase engagement.

The A/B test report contains information about the percentage of your audience, total send count, and open rate. You can even see a complete message report for each individual variant. Use the message reports to determine the quality of each variant and the message strategies that you can use to increase engagement.

Resources