1. Home
  2. /Guides
  3. /Messaging
  4. /User guide
  5. /Tools + Features
  6. /Holdout experiments

Holdout experiments

Holdout experiments can measure campaign impact on your business goals.

About holdout experiments

A Holdout Experiment compares the performance of audience members who receive messaging (the treatment group) with those who do not (the holdout group) for the duration of the experiment.

The primary components of a holdout experiment are:

  1. Holdout group — The percentage of your total audience that will be excluded from messaging. Audience members in the holdout group are randomly selected.

    You can ensure delivery of transactional messages to users in a holdout group by setting Allow transactional messages when creating the experiment. Using the API, you can bypass this behavior by setting a push option. See IMPORTANT note in step 4 in Creating holdout experiments.

  2. Goals — The events to measure in your experiment. You can select up to 10 goals per experiment.

  3. Duration — The period when the experiment is active. You can create open-ended or time-bound experiments, starting immediately or at a scheduled time and date.

As the experiment runs, reports for each goal show the performance of the holdout and treatment groups. After the experiment ends, or after a period of time relevant to the purpose of the experiment ends, you can evaluate the reports to determine the impact your messaging has on driving conversion goals or KPIs.

If there is no significant difference between holdout and treatment group performance, you may want to consider your campaigns and experiments for areas of improvement. Even with significant differences, this data can help you make informed decisions on how to best evolve your marketing strategy.

Only one experiment can be active at any time, and you cannot create or schedule an experiment while another is active.

Minimum SDKs required for In-App AutomationMessages that are cached on users’ devices and displayed when your users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times. , ScenesMulti-screen experiences that are cached on users’ devices and displayed when your users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times. , and SurveysQuestion-and-answer experiences used to collect and aggregate feedback or generate a net promoter score. They are cached on users’ devices and displayed when your users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times. only: iOS 17.2.0, Android 17.2.0.

Holdout group application

Holdout groups are applied at the user level. Any channel associated with a named user in an active holdout group will not receive messages until your experiment ends.

Depending on when your experiment starts, holdout experiments might be applied partially to scheduled messages.

You can view a user’s holdout group status in Contact Management.

Message purpose

Message Purpose is a designation of a message as commercial or transactional. When you create your first experiment, Message Purpose is automatically enabled for your project, and you must then select a purpose for each message. For more information, see: Enable Features: Message Purpose.

Creating goals

When you create an experiment, you can create new goals or select from goals already added to your project.

If you create new goals while creating a holdout experiment, they are contained within that holdout experiment only. They are not saved at the project level.

You can create goals based on a custom or predefined event or these default events:

  • App Open app_open — User opened your mobile app.
  • First Open first_open — User opened your mobile app for the first time.
  • First Seen first_seen — User opted in to notifications or opened your mobile app for the first time.
  • First Opt In first_opt_in — User opted in to a channel for the first time. For Email (commercial), SMS, and Open channels only.
  • Uninstall uninstall — User uninstalled your mobile app in response to a push.
  • Web Session web_session — User generated a web sessionA session that occurs when an opted-in user directly visits a website with the Airship Web SDK present, or by clicking a web notification that leads the visitor to the site. The page the user visits must have the Web SDK installed to track sessions. A new session is generated after 30 minutes of inactivity. .

Adding events

You do not need to add default events to your project before selecting them for goals. You must add custom and predefined events to your project before you can select them for goals.

Creating holdout experiments

  1. Go to Audience » Holdout Experiments and click Create holdout experiment.

  2. Define the experiment. All fields are required:

    • Name
    • Purpose
    • Hypothesis
    • Definition of success

    Click Next to move on.

  3. Search for and select goals, or create one. You can add up to 10 goals. To create a goal:

    1. Enter a goal name in the search field and click Create goal.
    2. Configure fields:
      • Goal name — Used for identification within the experiment.
      • Description — Additional information about the goal.
      • Event — Search for and select an event. If the event does not have a category assigned, select from the list or select Custom category and enter a category name.
    3. Click Create goal.

    Click Next to move on.

  4. Set the percentage of your audience to include in the holdout group. If you want to ensure delivery of transactional messages to audience members in the holdout group, check the box for Allow transactional messages.

    Click Next to move on.

     Important

    When sending messages through the API, if your holdout experiment has Allow transactional messages enabled, any API-sent message with message_type set to transactional will be sent to your holdout group.

    If you want any message (commercial or transactional) to bypass the holdout group behavior, you can set the bypass_holdout_groups push option for the message, and it will be sent to holdout group members.

  5. Set the duration. For specific times and dates, also specify the time zone. The UTC conversion displays below the settings and updates as you make changes. You can manually end an experiment at any time.

  6. Click Save.

Viewing reports

Go to Audience » Holdout Experiments. The upper portion of the page shows the name, definition, and set duration of the most recent experiment, indicated as Active, Scheduled, or Ended. It also displays the Goal report for the first goal added when setting up the experiment. Following these are the list of past experiments, with their start and end dates and times. Click a column header to sort. You can search for an experiment by name.

Holdout experiments
Holdout experiment reporting

Holdout experiments contain the same reports as GoalsSelected events that generate a set of performance reports. They are also used for measurement in Holdout Experiments. . Click   to view reports:

  • Goal — The number of times the event occurred per day and the 7-day average.
  • Channels per goal — The number of ChannelsA communication medium supported by the Airship service. Supported channels include app, web, email, SMS, and Open Channels. Within some channels there may be specific platforms with individual characteristics. Example platforms include Chrome for the web channel and Android for the mobile app channel. that performed the event at least one time. You can filter by greater than or equal to or is between and enter values.
  • Goal frequency per channel — The frequency of event occurrence per ChannelA communication medium supported by the Airship service. Supported channels include app, web, email, SMS, and Open Channels. Within some channels there may be specific platforms with individual characteristics. Example platforms include Chrome for the web channel and Android for the mobile app channel. . Data points displayed: 50th (median), 75th, and 99th percentiles.
  • Goal per platform or named user — The percentage of events that occurred per platform or Named UserA customer-provided identifier used for mapping multiple devices and channels to a specific individual. . Only appears if multiple platforms are configured for the project.

Reports will appear empty until enough data is available to populate them. The default view is the last three months of data. You can select a new time frame, and the reports will reload with the data for that period. For reports displaying multiple platforms, you can filter by one or more platforms.

To export the data, hover over the report, click  , and select Download. You can select from various output and other formatting options.

Following the reports is a summary of the experiment, listing:

  • Name
  • Purpose
  • Hypothesis
  • Success
  • Goals
  • Holdout group allocation percentage
  • Allow transactional messages
  • Start and end dates and times

Managing holdout experiments

Go to Audience » Holdout Experiments. The upper portion of the page shows the name, definition, and set duration of the most recent experiment, indicated as Active, Scheduled, or Ended.

Options:

  • Edit — Change the settings of an Active or Scheduled experiment. For Active experiments, you cannot change the holdout group allocation or transactional message status. Click  , make your changes, then click Save in the final step.

  • Delete — Cancel a Scheduled experiment before it starts. Click  .

  • Start — Change the start of a Scheduled experiment to the current date and time. Click Start now.

  • End — Immediately end an Active experiment. You cannot restart an ended experiment. Click End now.