Message A/B tests

About A/B tests for messages

Create variants of message content by duplicating the initial variant or from scratch. Each variant returns analytic data to help you determine the most effective way to engage your audience. You can retain a control group or send to 100% of your selected audience.

A/B tests for messages support these channels and message types:

• App — Push notifications and in-app messages
• Web
• Email
• SMS
• Open channel

When running A/B tests and a Holdout ExperimentMeasures the effects of excluding a group of audience members from all messages or messages with specific Campaign Categories. You can compare the performance of the two audience groups in reports for selected goal events. simultaneously, Airship prevents holdout group users from being included in the A/B tests. This eliminates potentially skewed data in cases where there are overlapping experimentation audiences. It also ensures that the most critical experiments maintain integrity.

To prepare for your tests, see About A/B testing.

Audience groups in the API

When you set up A/B Tests using the /experiments API, your audience is split across the variants in your message by weight properties. You can also set a control group.

The control group is a decimal (float) between 0 and 1 representing the portion of your audience who will not get a message. The remainder of your audience (after the control group is subtracted) receives messages according to their weight. If you don’t set weight properties, Airship splits your audience evenly across your variants.

Airship adds the weight properties in your payload and divides the total by an individual weight to determine the proportion of the audience that receives each variant. For example, if you set weights of 10, 10, and 5 for your variants, Airship splits your audience proportionally into subsets of 40%, 40%, and 20%:

{
    "name": "Experiment 1",
    "audience": "all",
    "control": 0.2,
    "device_types": "all",
    "variants": [
        {
            "push": {
                "notification": {
                    "alert": "You're in a cool group"
                }
            },
            "weight": 20
        },
        {
            "push": {
                "notification": {
                    "alert": "You're in the coolest group"
                }
            },
            "weight": 40
        }
    ]
}

Create a message A/B test

The following steps walk you through creating a message A/B test in the dashboard. For the API, see A/B Tests in the API reference.

To get started, select the Create dropdown menu (), then A/B Test.

Each configuration step is labeled in the center of the header:

After completing a step, select the next one to move on. Select the gear icon () to change the test name or flag it as a test.

1. In the Audience step, enter a descriptive name for the test, then enable channels and select which users should receive the test. User groups:

   Option	Description	Steps
   All Users	Your entire audience for the selected channels	n/a
   Target Specific Users	Audience members in a SegmentA grouping of audience members selected by unique or shared identifiers. Multiple identifiers can be combined within a Segment. you create	Follow the steps in Target Specific Users to create a single-use Segment.
   Test Users	Members of a Test GroupA preview group is audience group used for previewing personalized content in the dashboard. Wherever a personalization preview is available, you can select a preview group, and its group members’ attributes will appear for any Handlebars references to attributes. You can enable any preview group as a test group so you can send test messages to its group members. These messages appear as tests in Messages Overview.	Select a Test Group.

2. Select the Variants step, then select the number of variants you want to create and set the percentage of your target audience to test. By default we send your test to 80% of your target audience, keeping a control group of 20%. You can change the number of variants later.

   

3. Select the Content step, then enter a name for variant A and configure the message content per enabled channel. See Creating content and Optional Features.

   For additional variants, select a lettered tab, choose whether to copy content from an existing variant or start with a blank message, and complete message configuration. Select the plus icon () or remove icon () to add or remove variants. You cannot remove the last remaining variant.

4. Select the Delivery step, then set up delivery timing:

   Option	Description	Steps
   Send Now	Send the message immediately after review.	n/a
   Schedule1	Send the message on a specific date and time in a specific time zone or in each user's time zone. For delivery by time zone, a push notification scheduled for 9 a.m. will arrive for people on the east coast at 9 a.m. Eastern Time, in the midwest an hour later at 9 a.m. Central Time, then on the west coast two hours after that, at 9 a.m. Pacific Time.	Enter a date in YYYY-MM-DD format and select the time, then select a time zone or check the box for Delivery By Time Zone.
   Optimize	Send the message on a specific date and at each user's Optimal Send TimeAn algorithm that determines the best hour for optimal engagement activity — when each individual member of your audience is most likely to receive and act on your message. iOS, Android, and Fire OS platforms only.. iOS, Android, and Fire OS only. Airship recommends scheduling your message at least three days in advance due to the combination of time zones and optimal times. You can reduce the lead time if your audience is more localized, e.g., only in the United States or in a certain European region.	Enter a date in YYYY-MM-DD format.

   ^{1. Messages are only delivered by time zone to channels that have a time zone set. App and Web channels have their time zone set automatically by the SDK. Email, SMS, and Open channels will only have a time zone if set through the Channel Registration API. To do so, enter a value for the "timezone" key in the request body. See user registration information for Email, SMS, and Open channels. The API equivalent of Delivery By Time Zone is Push to Local Time.}

   After selecting timing, configure:

   Section	Description	Steps
   Purpose1	Set or verify the Message PurposeA dashboard setting that designates a message as commercial or transactional.. This option only appears if Message Purpose is enabled for the project.	Select Commercial or Transactional.
   Options	Various options are available depending on the message types, channels, and platforms selected for your test.	See Optional Features.
   External data feed options	If your message includes External Data FeedsA connection to an external API. When you send a message, Airship uses a response from that API to personalize messages., you must determine how the message is handled if the feed fails. Additionally, the default value is displayed for each send time variable in the feed URL. You can enter new values to override the default value for this message only.	For Failure behavior, select Abort sending the message or Send message without this data. For any Default value for <var>, enter a new value.
   Ban List	If your project has a Ban ListAn externally maintained record of users that should not be included in Airship message audiences. Before sending a message, Airship validates the audience members with the Ban List. Any matching recipients are dropped from the audience before sending the message. enabled and its request URL includes send time variables, you can override their default values for this message only. This setting does not appear if Bypass Ban List is also enabled.	For any Default value for \<variable>, enter a new value.

   ^{1. When Message Purpose is enabled and email and at least one other channel are selected for a message, Purpose is disabled in the Delivery step. Instead, set the purpose in the email's Sender Information: In Content step, select the Email tab, select Edit for Sender Information, and enable Transactional or leave it disabled if the message contains commercial content only. The commercial/transactional designation set in the email Sender Information will apply to all channels selected for the message.}

5. Select the Review step, then review the device preview and message summary. You can select a variant in the Content section or above the preview. Select the arrows to page through the various previews. The channel and display type dynamically update in the dropdown menu above. You can also select a preview directly from the menu. If you want to make changes, select the associated step in the header, make your changes, then return to Review.

   You can send a test message for each variant and verify its appearance and behavior on each channel the message is configured for. The message is sent to your selected recipients immediately, and it appears as a test in Messages OverviewA view of all your project’s messages, with options for editing their settings, content, status, and more..

   1. Select Send Test.
   2. Select a variant.
   3. Under Test audience, enter at least one Named UserA customer-provided identifier used for mapping multiple devices and channels to a specific individual. or Test GroupA preview group is audience group used for previewing personalized content in the dashboard. Wherever a personalization preview is available, you can select a preview group, and its group members’ attributes will appear for any Handlebars references to attributes. You can enable any preview group as a test group so you can send test messages to its group members. These messages appear as tests in Messages Overview. and select from the results. If your message includes email, you can also search for email addresses. If no matches appear for an address, you can select Create channel for <address>, and the channel will be registered for your project and opted in to transactional messaging.
   4. (If your message contains HandlebarsHandlebars is Airship’s templating language for personalization. Handlebars expressions use double curly braces wrapped around a content template, ranging from a simple variable, e.g., {{first_name}}, to complex evaluations of personalization data.) Under Personalization, select and configure a personalization data source:

      Data source	Description	Steps
      Test message recipient	The message will be personalized using information associated with each test audience member.	n/a
      Preview Data tool	The message will be personalized using the data currently entered in the Preview Data tool. The same values will apply to all test message recipients. You can also manually edit the JSON.	(Optional) Edit the JSON data.

   5. Select Send.

   When your review is complete, select Send Message or Schedule Message.

A/B test reports

After sending an A/B test, discover which variant performed best. Use the test- and message-level reports to determine the quality of each variant and strategies for increasing engagement. See also Implementing A/B tests, outcomes, and compliance in About A/B testing.

Go to Messages, then Messages Overview, and then select the report icon () for an A/B test. Report sections:

To export test data, select Download CSV, then Performance Data. If your test included Custom EventsEvents that indicate that a user performed a predefined action, such as adding an item to a shopping cart, viewing a screen, or clicking an Unsubscribe button. Custom Events can trigger automation, including Sequences and Scenes. You can code them into your app or website, or send them to Airship from an external source using the Custom Event API. Custom Events contain properties that you can use to personalize messages., you will also have the option to download Variant Event Data, which is a report of event conversions and associated values, broken out by variant or control group.