Holdout Experiments

About Holdout Experiments

When creating a Holdout Experiment, you first define its purpose, hypothesis, and a definition of success. This step serves as a guideline for designing an effective experiment, since your answers can influence what messages you choose include or exclude, what to measure, and the duration. The information also serves as a reference when evaluating reports.

Holdout Experiments can be open-ended or time-bound, starting immediately or at a scheduled time and date. Only one experiment can be active at any time, and you cannot create or schedule an experiment while another is active.

When you create your first experiment, Message PurposeA dashboard setting that designates a message as commercial or transactional. is automatically enabled for your project, and you must then select a purpose when creating any message in the dashboard.

Required for In-App AutomationsMessages cached on users’ devices and displayed when users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times. and ScenesA single or multi-screen in-app experience cached on users’ devices and displayed when users meet certain conditions in your app or website, such as viewing a particular screen or when a Custom Event occurs. They can be presented in fullscreen, modal, or embedded format using the default swipe/click mode or as a Story. Scenes can also contain survey questions. only: iOS SDK 17.2+Android SDK 17.8+

Holdout group

An experiment’s holdout group is the percentage of your total audience that is excluded from messaging. The remaining audience is the treatment group. Audience members in a holdout group are randomly selected.

Holdout groups are applied at the user level. No messages will send from your project to a ChannelAn instance representing an entity addressable via the Airship service, e.g., an iOS device, email address, SMS number or web browser. The channel instance or channel object contains all relevant information about a channel, including metadata used for targeting, opt-in status, device-specific information, and, importantly, a unique identifier for the channel, the Channel ID. associated with a Named UserA customer-provided identifier used for mapping multiple devices and channels to a specific individual. in an active holdout group until your experiment ends.

Depending on when your experiment starts, Holdout Experiments might be applied partially to scheduled messages.

You can view a user’s holdout group status by viewing their viewing channel details in Contact Management.

Message exclusion and allowances

You can exclude all messages from holdout group members or only messages with specific Campaign CategoriesAn optional delivery feature used to group messages of similar types or messaging strategies for aggregate reporting. Campaigns help you track the efficacy of both your individual messages and a messaging campaign as a whole.. For example, retailers could exclude purchase_journey campaigns to learn how their onboarding, abandoned cart, product rating requests, and other purchase-related messages impact conversion rates.

If your experiment excludes sending all messages, you can set exceptions that allow sending:

• Transactional messages — When sending messages from the API, you can bypass this allowance. See IMPORTANT note in step 5 in Creating Holdout Experiments.

• Messages with specific Campaign Categories — This flexibility helps ensure your business-critical or other required messages still reach your intended audience.

Goals and reports

Goals are the events you want to measure in your experiment. You can select from project-level GoalsSelected events that generate a set of performance reports. They can also be used for measurement in Holdout Experiments. or create new ones for that experiment only. Maximum 10 goals per experiment.

You can create Goals based on Custom or Predefined Events or these Default Events:

As an experiment runs, reports for each Goal show the performance of the holdout and treatment groups. Holdout Experiments generate the same reports as project-level Goals. They do not include events attributed to Named Users that are not associated with a platform and Channel IDAn Airship-specific unique identifier used to address a channel instance, e.g., a smartphone, web browser, email address..

After the experiment ends, or after a period of time relevant to the purpose of the experiment ends, evaluate the reports to determine the impact your messaging has on driving conversion goals or KPIs.

If there is no significant difference between holdout and treatment group performance, you may want to consider your campaigns and experiments for areas of improvement. Even with significant differences, this data can help you make informed decisions on how to best evolve your marketing strategy.

Workflow

The following is the general workflow for Holdout Experiments.

1. Handle Goals prerequisites — If the events or project-level Goals you want to use for your experiment don’t already exist in your project, you’ll need to add them.

2. Create the experiment — Define the experiment and set the holdout group percentage and options, Goals, and duration.

3. View reports and evaluate

Adding events and creating Goals

You must add Custom and Predefined Events to your project before you can select them for Goals. You do not need to add Default Events to your project before selecting them for Goals.

If you want to use project-level Goals in an experiment, you must first create them in your project settings. See Goals in Engagement Reports. Otherwise, you can create Goals as you create the experiment.

Creating Holdout Experiments

1. Go to Experiments, then Holdout Experiments.

2. Select Create Holdout Experiment. After completing each step, select Next to move on.

3. Define the experiment. All fields are required: Name, Purpose, Hypothesis, and Definition of success.

4. Search for and select Goals, or create one. You can add up to 10 Goals.

   To create a Goal:

   1. Enter a Goal name in the search field, then select Create Goal.
   2. Configure fields:

      Field	Description	Steps
      Goal name	Used for identification within the experiment	Enter text.
      Description	Additional information about the Goal	Enter text.
      Event	The event you want to measure in the experiment	Search for and select an event. If the event does not have a category assigned, select from the list or select Custom category and enter a category name.

   3. Select Create Goal.

5. Set up the holdout group and message options:

   Setting	Description	Steps
   Holdout group allocation	The percentage of your audience to exclude from messaging.	Set a percentage.
   Messages to withhold	Determines which messages to withhold from members of the holdout group. All messages withholds all messages sent from your project. Specific campaigns withholds messages with specific Campaign Categories. You cannot select Specific campaigns if allowances for Campaign Categories and/or Transactional messages are selected.	Select All messages or Specific campaigns. For Specific campaigns, enter a Campaign Category and then select its name below the entry field. Repeat for additional categories.
   Allowances: Campaign Categories	Allows sending messages with specific Campaign Categories to members of the holdout group. Cannot be selected if Withhold by Campaign Category is selected.	Enter a category and then select its name below the entry field. Repeat for additional categories.
   Allowances: Transactional messages	Allows sending transactional messages to members of the holdout group. Cannot be selected if Withhold by Campaign Category is selected. For more information, see Commercial vs. Transactional Email. The information also applies to other message types.	No configuration is required.

    Important

   If your experiment does not allow sending transactional messages to the holdout group, you can use the bypass_holdout_groups boolean to send them anyway.

   When set to true, the message, whether commercial or transactional, will be sent to holdout group members who are part of the message audience. This option is available for messages sent using the API only.

6. Set the duration. For specific times and dates, also specify the time zone. The UTC conversion displays below the settings and updates as you make changes.

7. Select Save.

Viewing reports

Go to Experiments, then Holdout Experiments. Information about the most recent experiment appears next to the report for the first Goal added when setting up the experiment. Ended experiments are listed under Past Holdout Experiments with their start and end dates and times. Select a date a column header to sort. You can search for experiments by name.

Select the report icon () for an experiment.

Performance

The Performance section contains the following reports per Goal:

The default view is the last 30 days of data. You can select a new time frame, and the reports will reload with the data for that period. For reports for multiple platforms, you can filter by one or more platforms.

To export data, hover over a report, then select the gear icon () and select Download. You can select from various output and other formatting options.

Reports do not include events attributed to Named UsersAn Airship-specific unique identifier used to address a channel instance, e.g., a smartphone, web browser, email address. that are not associated with a platform and Channel IDAn Airship-specific unique identifier used to address a channel instance, e.g., a smartphone, web browser, email address..

Experiment detail

The Experiment Detail section of an experiment report contains a summary of the experiment’s settings and displays the number of users in the holdout and treatment groups. Select a count to access a list of the users’ Channel IDsAn Airship-specific unique identifier used to address a channel instance, e.g., a smartphone, web browser, email address. and Named User IDsA customer-provided identifier used for mapping multiple devices and channels to a specific individual., and then select Download to export the list.

Managing Holdout Experiments

You can manage your most recent experiment. Go to Experiments, then Holdout Experiments. Information about the most recent experiment appears next to the report for the first Goal added when setting up the experiment.

Experiment management options:

Setting Campaign Categories

You can set Campaign Categories per message. For the API, see the Campaigns Object documentation. In the dashboard, these are the locations per composer: