A/B Testing

You can send variants of a message to cross-sections of your audience using A/B Tests; each variant in the test returns analytic data to help you determine the most effective way to engage your audience. A/B tests support up to 26 variants and a control group per test on iOS, Android, Amazon, and web platforms.

Planning your A/B Test

Before you begin an A/B test, you should know the parameters of your experiment:

  1. Have an audience in mind that you want to test — including the control group. The control group will not receive a message.
  2. Have a control message — a message that represents common communications with your audience, that you can expect to return stable results. A control message can help you understand baseline message performance.
  3. Understand the changes or variants you want to test in your message. How can you modify your message, and which modifications do you think will increase engagement? You may want your variant messages to differ only slightly so that you can pinpoint the specific strategies that increase engagement — like link placement or the tone of your message. Your test can include up to 26 different variants.

Audience Groups in the A/B Test

When you create an A/B test, you select an audience — this is the complete audience of users in your A/B test.

The control group is a selection of your audience that will not receive a message. The control group reduces the total audience by a specified proportion. For example, if your control group is set to 20%, the remaining 80% of the audience receives your message variants as a part of the test. The control group represents baseline behavior; they do not get or open messages, so they cannot have a positive or negative response.

Your remaining audience receives a message variant. By default, your remaining audience is evenly proportioned; each variant goes to the same number of users. However, if you create your test through the /experiments API, you can assign weights to variants if you want a particular message variant to go to a larger percentage of your audience.

Interpreting A/B Test Results

Airship reports the results of your A/B test so you can discover which variant performed the best and modify your messages going forward to increase engagement.

The A/B test report contains information about the percentage of your audience, total send count, and open rate. You can even see a complete message report for each individual variant. Use the message reports to determine the quality of each variant and the message strategies that you can use to increase engagement.