About A/B testing
You can send variants of a message to cross-sections of your audience using A/B Tests; each variant in the test returns analytic data to help you determine the most effective way to engage your audience. A/B tests support up to 26 variants and a control group per test.
You can also create A/B tests for SequencesA series of messages that is initiated by a trigger. Airship sends messages in the series based on your timing settings, and you can also set conditions that determine its audience and continuation. Sequences can be connected to each other and to other messaging components to create continuous user experiences in a Journey. and ScenesA single or multi-screen in-app experience cached on users’ devices and displayed when users meet certain conditions in your app or website, such as viewing a particular screen or when a Custom Event occurs. They can be presented in fullscreen, modal, or embedded format using the default swipe/click mode or as a Story. Scenes can also contain survey questions.. See: Sequence A/B Tests and Scene A/B Tests.
The A/B Test composer supports all channels and message types except for Message CenterA place in your app where you can display persistent rich messages, including HTML, video, etc. Similar to email, Message Center represents both the medium (the in-app inbox) and the message type (the messages you send to the inbox)..
Planning your A/B Test
Before you begin an A/B test, you should know the parameters of your experiment:
- Have an audience in mind that you want to test — including the control group. The control group will not receive a message.
- Have a control message — a message that represents common communications with your audience, that you can expect to return stable results. A control message can help you understand baseline message performance.
- Understand the changes or variants you want to test in your message. How can you modify your message, and which modifications do you think will increase engagement? You may want your variant messages to differ only slightly so that you can pinpoint the specific strategies that increase engagement — like link placement or the tone of your message. Your test can include up to 26 different variants.
Try personalizing variants in an A/B test using HandlebarsHandlebars is Airship’s templating language for personalization. Handlebars expressions use double curly braces wrapped around a content template, ranging from a simple variable, e.g., {{first_name}}, to complex evaluations of personalization data. referencing AttributesMetadata used for audience segmentation and personalization. They extend the concept of Tags by adding comparison operators and values to determine whether or not to target a user, helping you better evaluate your audience.. Personalized messages can help increase audience engagement.
Audience Groups in the A/B Test
When you create an A/B test, you select an audience — this is the complete audience of users in your A/B test.
The control group is a selection of your audience that will not receive a message. The control group reduces the total audience by a specified proportion. For example, if your control group is set to 20%, the remaining 80% of the audience receives your message variants as a part of the test. The control group represents baseline behavior; they do not get or open messages, so they cannot have a positive or negative response.
Your remaining audience receives a message variant. By default, your remaining audience is evenly proportioned; each variant goes to the same number of users. However, if you create your test through the /experiments API, you can assign weights to variants if you want a particular message variant to go to a larger percentage of your audience.
Audience Groups in the API
When you set up A/B Tests using the /experiments API, your audience is split across the variants in your message by weight properties. You can also set a control group.
The control group is a decimal (float) between 0 and 1 representing the portion of your audience who will not get a message. The remainder of your audience (after the control group is subtracted) receives messages according to their weight. If you don’t set weight properties, Airship splits your audience evenly across your variants.
Airship adds the weight properties in your payload and divides the total by an individual weight to determine the proportion of the audience that receives each variant. For example, if you set weights of 10, 10, and 5 for your variants, Airship splits your audience proportionally into subsets of 40%, 40%, and 20%.
| Variant | Weight | Audience percentage |
|---|---|---|
| A | 10 | 40% |
| B | 10 | 40% |
| C | 5 | 20% |
{
"name": "Experiment 1",
"audience": "all",
"control": 0.2,
"device_types": "all",
"variants": [
{
"push": {
"notification": {
"alert": "You're in a cool group"
}
},
"weight": 20
},
{
"push": {
"notification": {
"alert": "You're in the coolest group"
}
},
"weight": 40
}
]
}Interpreting A/B Test Results
Airship reports the results of your A/B test so you can discover which variant performed the best and modify your messages going forward to increase engagement.
The A/B test report contains information about the percentage of your audience, total send count, and open rate. You can even see a complete message report for each individual variant. Use the message reports to determine the quality of each variant and the message strategies that you can use to increase engagement.
Categories