Holdout Experiments

When you create a holdout experiment, you set up:

1. Holdout group — The percentage of your total audience that will be excluded from messaging
2. Goals — The events to measure in your experiment
3. Duration — The period when the experiment is active

As an experiment runs, reports for each goal show the performance of the holdout and treatment groups. After the experiment ends, or after a period of time relevant to the purpose of the experiment ends, you can evaluate the reports to determine the impact your messaging has on driving conversion goals or KPIs.

If there is no significant difference between holdout and treatment group performance, you may want to consider your campaigns and experiments for areas of improvement. Even with significant differences, this data can help you make informed decisions on how to best evolve your marketing strategy.

Requirements and documentation

Minimum SDKs required for In-App AutomationMessages cached on users’ devices and displayed when users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times., ScenesA single or multi-screen in-app experience cached on users’ devices and displayed when users meet certain conditions in your app or website, such as viewing a particular screen or when a Custom Event occurs. They can be presented in fullscreen, modal, or embedded format using the default swipe/click mode or as a Story. Scenes can also contain survey questions., and SurveysA question-and-answer experience used to collect and aggregate feedback or generate a net promoter score. The Survey composer was deprecated September 9, 2024. All Survey composer functionality is available in Scenes. only: iOS 17.2.0, Android 17.2.0.

Get all the details in our Holdout Experiments docs and start experimenting! 🥽 🥼 🧪