Holdout Experiments

When you create a holdout experiment, you set up:

1. Holdout group — The percentage of your total audience that will be excluded from messaging
2. Goals — The events to measure in your experiment
3. Duration — The period when the experiment is active

As an experiment runs, reports for each goal show the performance of the holdout and treatment groups. After the experiment ends, or after a period of time relevant to the purpose of the experiment ends, you can evaluate the reports to determine the impact your messaging has on driving conversion goals or KPIs.

If there is no significant difference between holdout and treatment group performance, you may want to consider your campaigns and experiments for areas of improvement. Even with significant differences, this data can help you make informed decisions on how to best evolve your marketing strategy.

Requirements and documentation

Minimum SDKs required for In-App AutomationMessages that are cached on users’ devices and displayed when your users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times., ScenesMulti-screen experiences that are cached on users’ devices and displayed when your users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times. Scenes can include survey questions or be presented as a story., and SurveysQuestion-and-answer experiences used to collect and aggregate feedback or generate a net promoter score. They are cached on users’ devices and displayed when your users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times. only: iOS 17.2.0, Android 17.2.0.

Get all the details in our Holdout Experiments docs and start experimenting! 🥽 🥼 🧪