Campaign Categories in Holdout Experiments

A Holdout Experiment measures the effects of excluding a group of audience members from messaging. You can compare the performance of the two audience groups in reports for selected goal events. Today’s release gives you two new options when configuring your experiments:

1. Withhold by Campaign Category — Instead of withholding all messages from holdout group members, withhold only messages with specific Campaign CategoriesAn optional delivery feature used to group messages of similar types or messaging strategies for aggregate reporting. Campaigns help you track the efficacy of both your individual messages and a messaging campaign as a whole.. For example, retailers could exclude purchase_journey campaigns to learn how their onboarding, abandoned cart, product rating requests, and other purchase-related messages impact conversion rates.

   After setting the holdout group percentage, choose to withhold by Campaign Category, then enter a category and select its name below the entry field. Repeat for additional categories.

2. Allow by Campaign Category — If your experiment is set to withhold all messages, you can allow messages with specific Campaign Categories. This flexibility helps ensure your business-critical or other required messages still reach your intended audience. You can also pair it with the existing option to allow transactional messages.

   Under Allowances, select Campaign Categories and add categories as described above for withholding by Campaign Category.

When creating messages, you can set Campaign Categories per message using the dashboard or API.

Requirements and documentation

Minimum SDKs required for In-App AutomationMessages that are cached on users’ devices and displayed when your users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times., ScenesMulti-screen experiences that are cached on users’ devices and displayed when your users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times. Scenes can include survey questions or be presented as a story., and SurveysQuestion-and-answer experiences used to collect and aggregate feedback or generate a net promoter score. They are cached on users’ devices and displayed when your users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times. only: iOS 17.2.0, Android 17.2.0.

Get all the details in our Holdout Experiments docs and start experimenting! 🥽 🥼 🧪