Sequence A/B tests

Use an A/B test to determine which version of a message has the best impact on a Sequence’s conversions or engagement.

About A/B tests for Sequences

Create a variant for any message in a SequenceA series of messages that is initiated by a trigger. Airship sends messages in the series based on your timing settings, and you can also set conditions that determine its audience and continuation. Sequences can be connected to each other and to other messaging components to create continuous user experiences in a Journey.. The variant is a duplicate of the original message that you can then edit, changing its content, delivery settings, or Channel CoordinationA group of strategies for targeting users where they are most likely to receive your message. settings. Audience allocation is set to 50% for each variant by default, but you can change the percentages. After starting the test, you will wait till the Confidence level meets or exceeds 95% and then select the winning message. The Sequence is then republished with the winning message. Audience members who receive the variant message are randomly selected on entry to the Sequence.

Related events and conversions are recorded for both audiences, providing data you can use to evaluate Sequence performance based on your selected metric.

To prepare for your tests, see About A/B testing.

Tip

You can run Sequence A/B tests and control groups concurrently.

Creating a Sequence A/B test

Note

You must start the Sequence before you can create an A/B test.
You cannot start an A/B test for a Sequence that has unpublished changes.

Go to Messages, then Messages Overview, and select the pencil icon () or report icon () for a Sequence.
Select Experiments in the left sidebar, then Create an A/B test.

Configure settings, then select Save and continue:

Setting	Description
Name and description	Text that describe the purpose of the test
Primary metric	The metric to use for reporting experiment performance: Engagement does not require additional setup. Sequence conversion requires a conversion event as the Sequence's Outcome.
Variant	The message to duplicate for editing
Variant allocation	The percentage of the Sequence audience who will receive the variant

Select Create variant, then edit the Content or Delivery steps, or edit the Channel CoordinationA group of strategies for targeting users where they are most likely to receive your message. setting. For Channel Coordination, select the gear icon (), make a new selection, then select Save & continue.
Tip
- A test with a single variable is measurable. When you make multiple changes in the variant, you will not know which change had an effect.
- If your test’s primary metric is Sequence conversions, consider editing any part of the message. For example, for a push notification, you could edit the title OR timing OR change your Channel Coordination selection.
- If your test’s primary metric is Engagement, focus on what users can experience before they interact with the message. For example, for an email, you would change the subject line only.
Select the Review step and review the device preview and message summary. Select the arrows to page through the various previews. The channel and display type dynamically update in the dropdown menu above. You can also select a preview directly from the dropdown menu. If you would like to make further changes, return to Review after you finish editing.
Send a test message to verify its appearance and behavior on each channel the message is configured for. The message is sent to your selected recipients immediately, and it appears as a test in Messages OverviewA view of all your project’s messages, with options for editing their settings, content, status, and more..
Select Save & continue, and you will then see the original and variant on the A/B test summary screen.
Select Start A/B test to make the variant available to your audience or select Exit to save the test without starting it. To start a saved A/B test:
1. Go to Messages, then Messages Overview, and then select the pencil icon () or the report icon () for a Sequence to go to the Sequence ManagerA preview of the messages in a Sequence, with options for editing and testing, and for running experiments. or Performance ReportA report that compares audience behavior to a Sequence’s goal. It displays performance metrics and a link to the message report for each message in the Sequence..
2. Select Experiments in the left sidebar, then View detail.
3. Select Start A/B test.

While the test is running:

You cannot edit the Sequence settings or messages.
On the Manage and Performance screens, the message with the variant is labeled with a beaker icon ().
On the Performance screen, statistics are the aggregate of the variant and the original message.

Selecting the winning variant

After starting the A/B test, compare the performance of the original message and variant and determine which (or if either) message is having the expected impact on engagement or conversion.

You may want to end an A/B test early if you see a significant drop in conversions or engagement. If the drop is not significant or if it is observed early on in the test period, you may want to let the test continue, as the rate may correct itself. Another reason to end a test early is if you notice an error in your content. To end a test early, select a winner. This effectively cancels the test.

See also Implementing A/B tests, outcomes, and compliance in About A/B testing.

Go to Messages, then Messages Overview, and then select the pencil icon () or the report icon () for a Sequence to go to the Sequence ManagerA preview of the messages in a Sequence, with options for editing and testing, and for running experiments. or Performance ReportA report that compares audience behavior to a Sequence’s goal. It displays performance metrics and a link to the message report for each message in the Sequence..
(From the Manage screen) Select the beaker icon () for the message with the variant.
(From the Performance report) Select Experiments in the left sidebar to view basic report statistics. Then select View results to access the test summary.

In the test summary, compare the metrics for the original message and the variant:

Data	Description
Sample size	The number of users selected to receive the variant. The threshold is 10,000 users.
Lift	The percent increase or decrease of your primary metric for users who have received the variant. Presented after seven days or when the sample size of 10,000 users is reached.
Confidence	The probability that the same results would be obtained if the test were repeated. Presented after seven days or when the sample size of 10,000 users is reached.

Additional statistics are displayed for the original message and variant based on the test's primary metric. If your primary metric is Engagement, select a message type to view statistics for that type only.

After Confidence is at least 95%, select a winner. The Sequence will be republished with the winner, and the A/B test ends. First select Select a winner, then Select original or Select variant, then confirm your choice.
Note
The winning message is saved as configured. You cannot select the content or settings per channel or message type.

Viewing A/B test history

After selecting a winning message, the A/B test is added to the list of past experiments.

Go to Messages, then Messages Overview, and then select the pencil icon () or the report icon () for a Sequence to go to the Sequence ManagerA preview of the messages in a Sequence, with options for editing and testing, and for running experiments. or Performance ReportA report that compares audience behavior to a Sequence’s goal. It displays performance metrics and a link to the message report for each message in the Sequence..
Select Experiments in the left sidebar.
Select Past Experiments. Each A/B test is listed by name along with its end date. Select the report icon () for an A/B test to open its summary.

Feedback

Was this page helpful?