Holdout Experiments

A Holdout Experiment measures the effects of excluding a group of audience members from all messages or messages with specific Campaign Categories. You can compare the performance of the two audience groups in reports for selected goal events.

About Holdout Experiments

When creating a Holdout Experiment, you first define its purpose, hypothesis, and a definition of success. This step serves as a guideline for designing an effective experiment, since your answers can influence what messages you choose include or exclude, what to measure, and the duration. The information also serves as a reference when evaluating reports.

Holdout Experiments can be open-ended or time-bound, starting immediately or at a scheduled time and date. Only one experiment can be active at any time, and you cannot create or schedule an experiment while another is active.

When you create your first experiment, Message PurposeA dashboard setting that designates a message as commercial or transactional. is automatically enabled for your project, and you must then select a purpose when creating any message in the dashboard.

Required for In-App AutomationsMessages cached on users’ devices and displayed when users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times. and ScenesA single or multi-screen in-app experience cached on users’ devices and displayed when users meet certain conditions in your app or website, such as viewing a particular screen or when a Custom Event occurs. They can be presented in fullscreen, modal, or embedded format using the default swipe/click mode or as a Story. Scenes can also contain survey questions. only: iOS SDK 17.2+Android SDK 17.8+

Holdout group

An experiment’s holdout group is the percentage of your total audience that is excluded from messaging. The remaining audience is the treatment group. Audience members in a holdout group are randomly selected.

Holdout groups are applied at the user level. No messages will send from your project to a ChannelAn instance representing an entity addressable via the Airship service, e.g., an iOS device, email address, SMS number or web browser. The channel instance or channel object contains all relevant information about a channel, including metadata used for targeting, opt-in status, device-specific information, and, importantly, a unique identifier for the channel, the Channel ID. associated with a Named UserA customer-provided identifier used for mapping multiple devices and channels to a specific individual. in an active holdout group until your experiment ends.

Depending on when your experiment starts, Holdout Experiments might be applied partially to scheduled messages.

You can view a user’s holdout group status by viewing their viewing channel details in Contact Management.

Message exclusion and allowances

You can exclude all messages from holdout group members or only messages with specific Campaign CategoriesAn optional delivery feature used to group messages of similar types or messaging strategies for aggregate reporting. Campaigns help you track the efficacy of both your individual messages and a messaging campaign as a whole.. For example, retailers could exclude purchase_journey campaigns to learn how their onboarding, abandoned cart, product rating requests, and other purchase-related messages impact conversion rates.

If your experiment excludes sending all messages, you can set exceptions that allow sending:

  • Transactional messages — When sending messages from the API, you can bypass this allowance. See IMPORTANT note in step 5 in Creating Holdout Experiments.

  • Messages with specific Campaign Categories — This flexibility helps ensure your business-critical or other required messages still reach your intended audience.

Goals and reports

Goals are the events you want to measure in your experiment. You can select from project-level GoalsSelected events that generate a set of performance reports. They can also be used for measurement in Holdout Experiments. or create new ones for that experiment only. Maximum 10 goals per experiment.

You can create Goals based on Custom or Predefined Events or these Default Events:

Event nameActivity nameDefinition
App openapp_openUser opened your mobile app.
First openfirst_openUser opened your mobile app for the first time.
First seenfirst_seenUser opted in to notifications or opened your mobile app for the first time.
First opt-infirst_opt_inUser opted in to a channel for the first time. For Email (commercial), SMS, and Open channels only.
UninstalluninstallUser uninstalled your mobile app in response to a push.
Web sessionweb_sessionUser generated a web session.

As an experiment runs, reports for each Goal show the performance of the holdout and treatment groups. Holdout Experiments generate the same reports as project-level Goals. They do not include events attributed to Named Users that are not associated with a platform and Channel IDAn Airship-specific unique identifier used to address a channel instance, e.g., a smartphone, web browser, email address..

After the experiment ends, or after a period of time relevant to the purpose of the experiment ends, evaluate the reports to determine the impact your messaging has on driving conversion goals or KPIs.

If there is no significant difference between holdout and treatment group performance, you may want to consider your campaigns and experiments for areas of improvement. Even with significant differences, this data can help you make informed decisions on how to best evolve your marketing strategy.

Workflow

The following is the general workflow for Holdout Experiments.

  1. Handle Goals prerequisites — If the events or project-level Goals you want to use for your experiment don’t already exist in your project, you’ll need to add them.

  2. Create the experiment — Define the experiment and set the holdout group percentage and options, Goals, and duration.

  3. View reports and evaluate

Adding events and creating Goals

You must add Custom and Predefined Events to your project before you can select them for Goals. You do not need to add Default Events to your project before selecting them for Goals.

If you want to use project-level Goals in an experiment, you must first create them in your project settings. See Goals in Engagement Reports. Otherwise, you can create Goals as you create the experiment.

Creating Holdout Experiments

  1. Go to Experiments, then Holdout Experiments.

  2. Select Create Holdout Experiment. After completing each step, select Next to move on.

  3. Define the experiment. All fields are required: Name, Purpose, Hypothesis, and Definition of success.

  4. Search for and select Goals, or create one. You can add up to 10 Goals.

    To create a Goal:

    1. Enter a Goal name in the search field, then select Create Goal.
    2. Configure fields:
      FieldDescriptionSteps
      Goal nameUsed for identification within the experimentEnter text.
      DescriptionAdditional information about the GoalEnter text.
      EventThe event you want to measure in the experimentSearch for and select an event. If the event does not have a category assigned, select from the list or select Custom category and enter a category name.
    3. Select Create Goal.
  5. Set up the holdout group and message options:

    SettingDescriptionSteps
    Holdout group allocationThe percentage of your audience to exclude from messaging.Set a percentage.
    Messages to withholdDetermines which messages to withhold from members of the holdout group. All messages withholds all messages sent from your project. Specific campaigns withholds messages with specific Campaign Categories.

    You cannot select Specific campaigns if allowances for Campaign Categories and/or Transactional messages are selected.

    Select All messages or Specific campaigns. For Specific campaigns, enter a Campaign Category and then select its name below the entry field. Repeat for additional categories.
    Allowances: Campaign CategoriesAllows sending messages with specific Campaign Categories to members of the holdout group. Cannot be selected if Withhold by Campaign Category is selected.Enter a category and then select its name below the entry field. Repeat for additional categories.
    Allowances: Transactional messagesAllows sending transactional messages to members of the holdout group. Cannot be selected if Withhold by Campaign Category is selected. For more information, see Commercial vs. Transactional Email. The information also applies to other message types.No configuration is required.
     Important

    If your experiment does not allow sending transactional messages to the holdout group, you can use the bypass_holdout_groups boolean to send them anyway.

    When set to true, the message, whether commercial or transactional, will be sent to holdout group members who are part of the message audience. This option is available for messages sent using the API only.

  6. Set the duration. For specific times and dates, also specify the time zone. The UTC conversion displays below the settings and updates as you make changes.

  7. Select Save.

Viewing reports

Go to Experiments, then Holdout Experiments. Information about the most recent experiment appears next to the report for the first Goal added when setting up the experiment. Ended experiments are listed under Past Holdout Experiments with their start and end dates and times. Select a date a column header to sort. You can search for experiments by name.

Select the report icon () for an experiment.

Performance

The Performance section contains the following reports per Goal:

Report nameDescription
GoalThe number of times the event occurred per day and the 7-day average.
Channels per goalThe number of ChannelsA communication medium supported by the Airship service. Supported channels include app, web, email, SMS, and Open Channels. Within some channels there may be specific platforms with individual characteristics. Example platforms include Chrome for the web channel and Android for the mobile app channel. that performed the event at least one time. You can filter by "greater than or equal to" and "is between" and enter values.
Goal frequency per channelThe frequency of event occurrence per ChannelA communication medium supported by the Airship service. Supported channels include app, web, email, SMS, and Open Channels. Within some channels there may be specific platforms with individual characteristics. Example platforms include Chrome for the web channel and Android for the mobile app channel.. Data points displayed: 50th (median), 75th, and 99th percentiles.
Goals per platformThe percentage of events that occurred per platform. Only appears if multiple platforms are configured for the project.

The default view is the last 30 days of data. You can select a new time frame, and the reports will reload with the data for that period. For reports for multiple platforms, you can filter by one or more platforms.

To export data, hover over a report, then select the gear icon () and select Download. You can select from various output and other formatting options.

Reports do not include events attributed to Named UsersAn Airship-specific unique identifier used to address a channel instance, e.g., a smartphone, web browser, email address. that are not associated with a platform and Channel IDAn Airship-specific unique identifier used to address a channel instance, e.g., a smartphone, web browser, email address..

Experiment detail

The Experiment Detail section contains a summary of the experiment’s settings and displays the number of Channel IDsAn Airship-specific unique identifier used to address a channel instance, e.g., a smartphone, web browser, email address. in the holdout group. Select the count to access a list of the Channel IDs. To export the list, select Download. If the count exceeds viewable rows, you can export the full list by selecting Download results to see more.

Managing Holdout Experiments

You can manage your most recent experiment. Go to Experiments, then Holdout Experiments. Information about the most recent experiment appears next to the report for the first Goal added when setting up the experiment.

Experiment management options:

OptionDescriptionSteps
EditYou can change the settings of an Active or Scheduled experiment. For Active experiments, you cannot change the holdout group allocation or transactional message status.Select the pencil icon (), make your changes, then select Save in the final step.
Delete a Scheduled experimentRemoves the experiment from your project. Deleted experiments cannot be recovered.Select the trash can icon ().
Start a Scheduled experimentChanges the start date and time of the experiment to the current date and time.Select Start now.
End an Active experimentChanges the end date and time of the experiment to the current date and time. You cannot restart an ended experiment.Select End now.

Setting Campaign Categories

You can set Campaign Categories per message. For the API, see the Campaigns Object documentation. In the dashboard, these are the locations per composer:

ComposerComposer stepDocumentation
Message, A/B Test, Automation, SequenceDeliveryCampaign Categories in Optional features
In-App Automation, SceneSettingsCampaign Categories in Set optional message features