Campaign Categories in Holdout Experiments

A Holdout Experiment measures the effects of excluding a group of audience members from messaging. You can compare the performance of the two audience groups in reports for selected goal events. Today’s release gives you two new options when configuring your experiments:

1. Withhold by Campaign Category — Instead of withholding all messages from holdout group members, withhold only messages with specific Campaign CategoriesAn optional delivery feature used to group messages of similar types or messaging strategies for aggregate reporting. Campaigns help you track the efficacy of both your individual messages and a messaging campaign as a whole.. For example, retailers could exclude purchase_journey campaigns to learn how their onboarding, abandoned cart, product rating requests, and other purchase-related messages impact conversion rates.

   After setting the holdout group percentage, choose to withhold by Campaign Category, then enter a category and select its name below the entry field. Repeat for additional categories.

2. Allow by Campaign Category — If your experiment is set to withhold all messages, you can allow messages with specific Campaign Categories. This flexibility helps ensure your business-critical or other required messages still reach your intended audience. You can also pair it with the existing option to allow transactional messages.

   Under Allowances, select Campaign Categories and add categories as described above for withholding by Campaign Category.

When creating messages, you can set Campaign Categories per message using the dashboard or API.

Requirements and documentation

Minimum SDKs required for In-App AutomationMessages cached on users’ devices and displayed when users meet certain conditions within your app, such as viewing a particular screen or opening the app a certain number of times., ScenesA single or multi-screen in-app experience cached on users’ devices and displayed when users meet certain conditions in your app or website, such as viewing a particular screen or when a Custom Event occurs. They can be presented in fullscreen, modal, or embedded format using the default swipe/click mode or as a Story. Scenes can also contain survey questions., and SurveysA question-and-answer experience used to collect and aggregate feedback or generate a net promoter score. The Survey composer was deprecated September 9, 2024. All Survey composer functionality is available in Scenes. only: iOS 17.2.0, Android 17.2.0.

Get all the details in our Holdout Experiments docs and start experimenting! 🥽 🥼 🧪