Sequence Control Groups

A control group is a percentage of an audience that is excluded from receiving messages. Audience members in the control group are randomly selected on entry to the sequence, and they continue the sequence just like active audience members, only without receiving any of its messages. Related events and conversions are recorded for both audiences, providing data you can use to evaluate sequence performance. Using control groups in this way can help you:

• Determine the attribution of conversions to, as well as the potentially negative impacts of, various marketing efforts.
• Evaluate the impact your messaging has on driving conversion goals, KPIs, or uninstalls/opt-outs, to make informed decisions on how to best evolve your marketing strategy.
• Preview and set the pace for a controlled rollout.

Create a control group

As a best practice, you should create a control group before starting a sequence, but you can create a control group at any time. From the sequence ManageA preview of the messages in a Sequence, with options for editing and testing, and for running experiments. or PerformanceA report that compares audience behavior to a Sequence’s goal. It displays performance metrics and a link to the message report for each message in the Sequence. screen:

1. Click Experiments in the leftside drawer.
2. Click Create a control group, and enter the percentage of users to exclude from messaging.
3. Click Save.

After saving, the Experiments drawer will update differently, depending on sequence configuration and status:

• For sequences without a conversion event, you will see Control group allocation and options to remove the control group and adjust the allocation.

• For sequences with a conversion event, you will see only Control group allocation until after you start the sequence. After starting, you will instead see Control group performance, with the following data, and options to remove the control group, adjust the allocation, and set a baseline.

  • Allocation: The current control group percentage.
  • Sample size: The number of users in the control group. Updated upon hard refresh and when allocation is changed.
  • Conversions: The number of users who exited the sequence by a conversion event. Updated upon hard refresh.
  • Conversion rate: The number of conversions divided by the sample size. Updated upon hard refresh.

Compare Performance reporting

The Performance report shows audience behavior compared to the sequence’s goal. While a control group is enabled, you can review the performance of the active group compared to the control group and determine if messaging is having the expected impact on the sequence’s conversion events.

The Performance report is available after you start the sequence. For full documentation, see: Sequence Performance.

1. Go to Messages » Messages Overview.
2. Click for a sequence. The report has the same layout as the Sequence ManagerA preview of the messages in a Sequence, with options for editing and testing, and for running experiments. and contains these main elements:
   • Performance data: Statistical data for the sequence and each message.
   • Message reports: Click Report to open an individual message report.
3. Select Active audience / Control group to change the viewable data.
4. (Optional) Select a time frame to update the viewable data.

Manage a controlled rollout

You can use a controlled rollout to increase the availability of a sequence over time. This can be helpful for sequences that are considered high-risk, such as promoting a new feature launch or product line. If you want to preview the performance before deployment, set the initial control group allocation to 100%, and compare performance data before reducing the group size.

1. Create a sequence, but do not start it.
2. Go to the ManageA preview of the messages in a Sequence, with options for editing and testing, and for running experiments. or PerformanceA report that compares audience behavior to a Sequence’s goal. It displays performance metrics and a link to the message report for each message in the Sequence. screen.
3. Click Experiments in the leftside drawer.
4. Click Create a control group, and enter the percentage of users to exclude from messaging.
5. Click Save.
6. Click Start.

When you are ready to increase the availability of the sequence:

1. Go to the ManageA preview of the messages in a Sequence, with options for editing and testing, and for running experiments. or PerformanceA report that compares audience behavior to a Sequence’s goal. It displays performance metrics and a link to the message report for each message in the Sequence. screen.
2. Click Experiments in the leftside drawer.
3. Click Adjust allocation and enter a lower percentage.
4. Click Save.

Continue to reduce the control group size over time, and click Remove control group when you want to make the sequence available to your entire audience.

Measure campaign lift

Lift attempts to determine the efficacy of a sequence by comparing the conversion rate of an active audience with that of a control group who received no messages. This can give you an idea of how your audience performs with and without your marketing efforts. Key terms:

Baseline

The baseline is the benchmark conversion rate of a Sequence’s control group. It represents the conversion rate you would expect to see without messaging, and it is used to calculate the lift rate.

Lift rate

The lift rate is the percent increase or decrease in the active audience conversion rate against the baseline.

Conversion trend

The conversion trend is the difference between the baseline and current conversion rates for a selected time frame.

For example, a hotel chain may want to measure the impact of sending their audience a discount code for $150 off reservations for any stay of 2+ nights, using a sequence conversion goal of hotel bookings. After creating the control group, they set a baseline when their desired sample size has been met. At the end of the 90-day campaign they find that the campaign lift rate is +86%:

• Baseline: 7% — The booking rate they would expect to see without messaging.
• Conversion rate: 13% — The conversion rate of the active audience.
• Conversion trend: +6% — The difference between the baseline conversion rate and the campaign-end conversion rate.
• Lift rate: +86% — For users who received the discount code, hotel bookings increased 86% over the baseline.

As your campaign runs and at its end, you can determine the impact of your messaging by looking at the lift rate, conversion trend, and other factors specific to the campaign. Keep in mind that:

1. The baseline is intended to represent the behavior of users without marketing influence (the messages in your sequence), however, they may be exposed to your marketing through other channels.
2. The lift rate is neutral data — even though a lift rate may be high, if the campaign cost to achieve your sequence goal is also high, you may find the ROI is too low to continue the campaign.

To get started, set up a sequence and its control group.

1. Create a sequence that has a conversion event, but do not start it.
2. Go to the ManageA preview of the messages in a Sequence, with options for editing and testing, and for running experiments. or PerformanceA report that compares audience behavior to a Sequence’s goal. It displays performance metrics and a link to the message report for each message in the Sequence. screen.
3. Click Experiments in the leftside drawer.
4. Click Create a control group, and enter the percentage of users to exclude from messaging.
5. Click Save.
6. Click Start.

Next you will establish a baseline for reporting. You decide when to set the baseline, by sample size or time — either wait till the number of users in the control group is representative of your sequence’s goal, or at least 7 days or 3 times the sequence length. When you set the baseline, the control group allocation is set to 0%, making the sequence available to your entire audience, and Airship starts generating reporting data using the baseline.

1. Click Experiments in the leftside drawer.
2. Click Set baseline and confirm.

After setting a baseline, Sequence lift is added to the Experiments drawer, which displays the lift rate, the baseline rate, and the date the baseline was set. You can view the lift rate and conversion trend in the Journey MapA visualization of messaging components that can be connected to create a continuous user experience. You can also create and edit from the map..

1. Go to Journeys.
2. Select the sequence from the sidebar or in the map. The card in the map view displays the conversion trend next to the conversion rate. It appears in green for a positive trend and red for a negative trend. Click the trend to see the lift rate, baseline, and the date the baseline was set.

Now you can let the campaign run to completion. As a best practice, do not edit a sequence while it has a control group enabled.

Create a new control group

When you need to change a sequence’s content and settings, you will likely also want to cancel your current control group and create a new one. For instance, you might update a sequence for seasonality, e.g., changing from winter to spring campaigns.

Creating a new control group follows the same procedure as creating the first control group — from the sequence ManageA preview of the messages in a Sequence, with options for editing and testing, and for running experiments. or PerformanceA report that compares audience behavior to a Sequence’s goal. It displays performance metrics and a link to the message report for each message in the Sequence. screen:

1. Click Experiments in the leftside drawer.
2. Click Create a control group, and enter the percentage of users to exclude from messaging.
3. Click Save.