A/B Test Composer Tutorial

Use A/B Tests to experiment with different variations, or variants, of a message for a given audience, and provide engagement reporting data for each variant.

See the Message Reference for allowed message types and combinations. See also: Message and Composer Types and Composer Overview.

 Tip

You can configure the default appearance of your In-App Messages in Settings » Configuration » In-App Message Design. This includes button text and message color, screen position, and more.

Audience

The audience you select in this step defines the entire set of users for the A/B test, which will include the control group (users who receive no message) and users who will receive different messages, according to the variants that you define.

  1. Open your messaging project, then click Create and select A/B Test.
  2. (Optional) Add a name and/or flag as a test.
    1. Click in the header.
    2. Enter the name.
    3. Enable the Test toggle.
    4. Click outside the box to close it.
  3. Select which Channels are included in your audience.All channels configured for your project that are available for A/B testing will be listed and enabled by default, but you can toggle to limit your audience to specific channels. You must select at least one.
  4. Select your desired recipient group.
    • All Users: Sends the message to your entire audience.
    • Target Specific Users: Create a recipient group based on segmentation data. See: Target Specific Users.
    • Test Users: Predefined recipient groups. After choosing Test Users, select from the Test Groups dropdown menu that appears. See: Test Groups.
  5. Click Variants in the header to move on.

Variants

  1. Enter a descriptive Test Name if you didn't enter a name in the Audience step.
  2. Select the Number of Variants for your test, up to 26. The default is two. You will be able to add or delete variants in later steps.
  3. Use the slider to adjust the percentages of your target audience vs. control group. By default we send your test to 80% of your audience, keeping a control group of 20%

Content

Variant A Setup

  1. Enter a name for the first Variant.
  2. Choose between sending a Push Notification or an In-App Message, or choose Silent Push Notification to send a push without notification text. After you make your selection you can combine message types, then click Continue to confirm your choices.
  3. Enter the text that will display in your message. The preview will display as you type.
     Note

    If you chose to combine a Push Notification and In-App Message, an In-App Message Text section appears below the message truncation indicator. By default, the In-App Message's alert text is the same as the Push Notification text. If you'd prefer to use a different message, select the radio button for Write Alternative and enter your text.

  4. Select a Message Action from the dropdown menu. For Landing Page, follow the steps in Create Rich Pages.
     Note

    The Message Actions listed in the dropdown menu differ depending on which message type you chose and if you chose to combine message types. See: Message Reference: Actions.

  5. (Optional) Set a tag.
  6. Set Optional Features.
  7. Configure additional variants. Select the next variant by using the lettered tabs, and complete the name and message setup for each. For variants B-Z, you can either use the content from a previously created variant, or start from a blank message. You may also add and delete variants using the tab X and + options.
  8. Click Delivery in the header to move on.

Delivery

  1. Select Send Now or Schedule. For Schedule, enter a date in YYYY-MM-DD format and make your selections from the dropdown menus.
  2. Set Optional Features.
  3. Click Review in the header to move on.

Review

  1. Select a variant from the dropdown menu in the Content section or above the preview.
  2. Review the device preview and message summary. Click the arrows to page through the various previews. The channel and display type dynamically update in the dropdown menu above. You can also select a preview directly from the dropdown menu.If you would like to make changes, click the associated step in the header, make your changes, then return to Review.
  3. Click Send Message or Schedule Message.