Journey Control Groups

Use a control group to measure campaign efficacy and manage controlled rollouts.

A control group is a percentage of an audience that is excluded from receiving messages. Audience members in the control group are randomly selected on entry to the journey, and they continue the journey just like active audience members, only without receiving any of its messages. Related events and conversions are recorded for both audiences, providing data you can use to evaluate journey performance. Using control groups in this way can help you:

  • Determine the attribution of conversions to, as well as the potentially negative impacts of, various marketing efforts.
  • Evaluate the impact your messaging has on driving conversion goals, KPIs, or uninstalls/opt-outs, to make informed decisions on how to best evolve your marketing strategy.
  • Preview and set the pace for a controlled rollout.

Create a control group

As a best practice, you should create a control group before starting a journey, but you can create a control group at any time. From the journey ManageA preview of the messages in a journey, with options for editing its settings, content, and status. , PerformanceA report that compares audience behavior to a journey’s goal. It displays performance metrics and a link to the message report for each message in the journey. , or Test screen:

  1. Click Outcomes in the leftside drawer.
  2. Click Create a control group, and enter the percentage of users to exclude from messaging.
  3. Click Save.

After saving, Outcomes will update differently, depending on journey configuration and status:

  • For journeys without a conversion event, you will see Control group allocation and options to remove the control group and adjust the allocation.

  • For journeys with a conversion event, you will see only Control group allocation until after you start the journey. After starting, you will instead see Control group performance, with the following data, and options to remove the control group, adjust the allocation, and set a baseline.

    • Allocation: The current control group percentage.
    • Sample size: The number of users in the control group. Updated upon hard refresh and when allocation is changed.
    • Conversions: The number of users who exited the journey by a conversion event. Updated upon hard refresh.
    • Conversion rate: The number of conversions divided by the sample size. Updated upon hard refresh.

Compare Performance reporting

The Performance report shows audience behavior compared to the journey’s goal. While a control group is enabled, you can review the performance of the active group compared to the control group and determine if messaging is having the expected impact on the journey’s conversion events.

The Performance report is available after you start the journey. For full documentation, see: View Journey Performance Reports.

  1. Go to Messages » Messages Overview.
  2. Click   for a journey. The report has the same layout as the Journey ManagerA preview of the messages in a journey, with options for editing its settings, content, and status. and contains these main elements:
    • Performance data: Statistical data for the journey and each message.
    • Message reports: Click   Report to open an individual message report.
  3. Select Active audience / Control group to change the viewable data.
  4. (Optional) Select a time frame to update the viewable data.

Manage a controlled rollout

You can use a controlled rollout to increase the availability of a journey over time. This can be helpful for journeys that are considered high-risk, such as promoting a new feature launch or product line. If you want to preview the performance before deployment, set the initial control group allocation to 100%, and compare performance data before reducing the group size.

  1. Create a journey, but do not start it.
  2. Click Outcomes in the leftside drawer.
  3. Click Create a control group, and enter the percentage of users to exclude from messaging.
  4. Click Save.
  5. Click Start journey.

When you are ready to increase the availability of the journey:

  1. Click Outcomes in the leftside drawer.
  2. Click Adjust allocation and enter a lower percentage.
  3. Click Save.

Continue to reduce the control group size over time, and click Remove control group when you want to make the journey available to your entire audience.

Measure campaign lift

Lift attempts to determine the efficacy of a journey by comparing the conversion rate of an active audience with that of a control group who received no messages. This can give you an idea of how your audience performs with and without your marketing efforts. Key terms:

Baseline
The baseline is the benchmark conversion rate of a journey’s control group. It represents the conversion rate you would expect to see without messaging, and it is used to calculate the lift rate.
Lift rate
The lift rate is the percent increase or decrease in the active audience conversion rate against the baseline.
Conversion trend
The conversion trend is the difference between the baseline and current conversion rates for a selected time frame.

For example, a hotel chain may want to measure the impact of sending their audience a discount code for $150 off reservations for any stay of 2+ nights, using a journey conversion goal of hotel bookings. After creating the control group, they set a baseline when their desired sample size has been met. At the end of the 90-day campaign they find that the campaign lift rate is +86%:

  • Baseline: 7% — The booking rate they would expect to see without messaging.
  • Conversion rate: 13% — The conversion rate of the active audience.
  • Conversion trend: +6% — The difference between the baseline conversion rate and the campaign-end conversion rate.
  • Lift rate: +86% — For users who received the discount code, hotel bookings increased 86% over the baseline.

As your campaign runs and at its end, you can determine the impact of your messaging by looking at the lift rate, conversion trend, and other factors specific to the campaign. Keep in mind that:

  1. The baseline is intended to represent the behavior of users without marketing influence (the messages in your journey), however, they may be exposed to your marketing through other channels.
  2. The lift rate is neutral data — even though a lift rate may be high, if the campaign cost to achieve your journey goal is also high, you may find the ROI is too low to continue the campaign.
 Note

Airship does not provide historical reporting data for control groups. If you intend to compare the performance of a journey based on multiple control groups and baselines, you will need to record the data yoursef.

To get started, set up a journey and its control group.

  1. Create a journey that has a conversion event, but do not start it.
  2. Click Outcomes in the leftside drawer.
  3. Click Create a control group, and enter the percentage of users to exclude from messaging.
  4. Click Save.
  5. Click Start journey.

Next you will establish a baseline for reporting. You decide when to set the baseline, by sample size or time — either wait till the number of users in the control group is representative of your journey’s goal, or at least 7 days or 3 times the journey length. When you set the baseline, the control group allocation is set to 0%, making the journey available to your entire audience, and Airship starts generating reporting data using the baseline.

  1. Click Outcomes in the leftside drawer.
  2. Click Set baseline and confirm.

After setting a baseline, Journey lift is added to Outcomes, which displays the lift rate, the baseline rate, and the date the baseline was set. You can view the lift rate and conversion trend in the Journey MapShows the relationships between journeys. .

  1. Go to Messages » Journeys.
  2. Select the journey from the sidebar or in the map. The card in the map view displays the conversion trend next to the conversion rate. It appears in green for a positive trend and red for a negative trend. Click the trend to see the lift rate, baseline, and the date the baseline was set.

Now you can let the campaign run to completion. As a best practice, do not edit a journey while it has a control group enabled.

Create a new control group

When you need to change a journey’s content and settings, you will likely also want to cancel your current control group and create a new one. For instance, you might update a journey for seasonality, e.g., changing from winter to spring campaigns.

Creating a new control group follows the same procedure as creating the first control group — from the journey ManageA preview of the messages in a journey, with options for editing its settings, content, and status. , PerformanceA report that compares audience behavior to a journey’s goal. It displays performance metrics and a link to the message report for each message in the journey. , or Test screen:

  1. Click Outcomes in the leftside drawer.
  2. Click Create a control group, and enter the percentage of users to exclude from messaging.
  3. Click Save.
 Note

The lift rate is always based on the latest set baseline. Even if you create a new control group, the baseline is not affected until you set a new baseline.