Message A/B tests

Experiment with up to 26 message variations to determine audience engagement.

See also Scene A/B tests and Sequence A/B tests .

About A/B tests for messages

Create variants of message content by duplicating the initial variant or from scratch. Each variant returns analytic data to help you determine the most effective way to engage your audience. Audience allocation is automatically evenly distributed between variants, but you can change the percentages.

A/B tests for messages support these channels and message types:

  • App — Push notifications, in-app messages, and Message Center
  • Web
  • Email
  • SMS
  • Open channel

Set up the test in two steps:

  1. Create two or more message variants — Just like in the Message composer, for each variant, select channels, configure content for each channel, and set up delivery.

  2. Allocate an audience — You can designate all users as eligible for the test or target specific users. Of that group, set the percentage that will be able to receive a variant. Audience members are randomly selected.

    Setting a percentage helps limit the audience so you can effectively manage feedback and interpret results. By default, the overall audience percentage is divided evenly between variants, but you can set your own values.

After creating variants and setting the audience, you can start the test and review its results.

When running A/B tests and a Holdout ExperimentMeasures the effects of excluding a group of audience members from all messages or messages with specific Campaign Categories. You can compare the performance of the two audience groups in reports for selected goal events. simultaneously, Airship prevents holdout group users from being included in the A/B tests. This eliminates potentially skewed data in cases where there are overlapping experimentation audiences. It also ensures that the most critical experiments maintain integrity.

To prepare for your tests, see About A/B testing.

Create a message A/B test

To get started, select the Create dropdown menu (), then A/B Test. Or you can start from your list of all A/B tests: Go to Experiments, then Message A/B Tests, and then select Create A/B Test.

Next, select the test name and change it to something descriptive, then select the check mark to save it.

To finish setting up your test, you must add message variants and determine the audience. You can configure them in any order.

Add message variants

You can add up to 26 variants to an A/B test:

  1. Select Add variant, configure a step, and select the next step in the header to move on.

    Configuration details per step:

    StepConfiguration
    ChannelsEnter a name for the variant and enable the channels you want to send the message to. Also enable Localization if you want to create localized App and/or Web content.
    ContentConfigure the message content per enabled channel. See the Content documentation per message type, Optional features, and Localization.
    DeliveryConfigure the message delivery timing and options. See Message delivery.
  2. In the Review step, review the device preview and message summary:

    When your review is complete, select Save Variant.

To add another variant from scratch, select Add variant. To duplicate an existing variant, select the three dots icon () at the end of a row and select Copy to variant.

Set the test audience

After creating an A/B test, select Audience and then set up your test audience:

  1. Choose and configure users:
    OptionDescriptionSteps
    All UsersThis option makes the test available to a percentage of your total audience.n/a
    Target Specific UsersThis option makes the test available to a percentage of users who meet specified conditions.Select and configure one or more conditions. Follow the steps in Target Specific Users.
  2. Under Audience allocation, limit the selected audience to your specified percentage.
  3. (Optional) To override the default variant distribution, enable Allow uneven allocations. Then, under Variant allocation, edit the percentage for each variant.
     Note

    If you later add more variants, also update your variant allocation settings.

  4. Select Save.

Start an A/B test

Once you’ve created your message variants and set the audience for your test, select Start A/B test and confirm. Each variant will send according to its delivery settings.

View test results

After starting an A/B test, discover which variant performed best. Use the test- and message-level reports to determine the quality of each variant and strategies for increasing engagement. See also Implementing A/B tests, outcomes, and compliance in About A/B testing.

To access test results, go to Experiments, then Message A/B Tests, select the three dots icon () for a test in the list, then View results. You can also select the name of a test from the list and then go to Results.

To export test data, select Download CSV, then Performance Data. If your test included Custom EventsEvents that indicate that a user performed a predefined action, such as adding an item to a shopping cart, viewing a screen, or clicking an Unsubscribe button. Custom Events can trigger automation, including Sequences and Scenes. You can code them into your app or website, or send them to Airship from an external source using the Custom Event API. Custom Events contain properties that you can use to personalize messages., you will also have the option to download Variant Event Data, which is a report of event conversions and associated values, broken out by variant.

 Note

Engagement data is sent to Airship as soon as it becomes available. Data may be delayed due to connectivity issues with a user’s carrier, wifi, power, etc. Wait at least 12 to 24 hours before acting on test data to allow for potential lags.

Managing A/B tests

Go to Experiments, then Message A/B Tests to view and manage your Message A/B Tests. Each test is listed by name with its status and the date it was last modified. Your last modified test is listed first. You can search for tests by name.

Options from the tests list:

OptionDescriptionSteps
ViewOpen the test to access its message variants, audience configuration, and results.Select a test's name. Or select its three dots icon () and then View test.
DuplicateMake a draft copy of a test with its message variants and audience configuration.Select a test's three dots icon () and then Duplicate.
View resultsOpen the test's performance reports.Select a test's three dots icon () and then View results. See View test results.

Editing message variants and audience

You can edit variants and audience settings for any test that has not yet been started. After opening a test from the message A/B Tests list, select the three dots icon () for a variant and select an option:

OptionDescription
EditModify the variant's channels, content, or delivery settings.
DuplicateCreate a copy of the variant as a starting point for a new variant.
DeleteRemove the variant from the test.

To modify the test audience, select Audience and adjust targeting or allocation settings. See Set the test audience for configuration details.