View A/B Test Reports

A/B Tests have an overview report and a message report for each variant.

  1. Open your messaging project from the dashboard, then go to Messages » A/B Tests.
  2. Click on the History tab. The report contains these main elements:
    • Header: Displays the test name, sent date, time, and time zone.
    • Performance: Contains statistical data for each variant and the control group. Click a variant name to view its message report.
    • Message Detail: Contains the same information shown in the Review step of the A/B Test composer and the Message Detail section of a message report. Click the arrows to page through the various previews. The platform and display type dynamically update in the dropdown menu above. You can also select a preview or variant directly from the dropdown menus.
  3. (Optional) Click Download CSV and select Performance Data or Variant Event Data.
     Note

    Engagement data is sent to Airship as soon as it becomes available. Often data is delayed because of connectivity issues with a user's carrier, wifi, power, etc. You should wait at least 12 to 24 hours before acting on this data to allow for potential lags.

Performance Data

The following data are included in the A/B Test Performance Data export:

FieldDefinition
Project NameYour project name.
KeyUnique authentication key for your app.
Channel IdentifierUnique identifier that refers to the entire push send. The channel identifier will be the same for each variant within an A/B Test, with N/A for users in the control group who do not receive a notification.
Sent Time (UTC)UTC timestamp.
Test NameUser-friendly name for your A/B Test.
VariantThe letter corresponding to the message variant. A, B, C, etc.
AudienceDescription of the selected audience for the A/B Test. One of All Users, Target Specific Users, or Test Users.
Notification AlertAlert text for the given variant.
Notifications Sent/Control Sample SizeAudience send counts per variant/control group size.
Indirect OpensOpens that occur within 12 hours of the notification, regardless of attribution.
Direct OpensOpens that are attributed directly to interacting with the notification.
Influenced OpensSee: Reports: Influenced Opens.

Variant Event Data

If you are using Custom Events, you can download a report of event conversions and associated values, broken out by variant or control group.

The following data are included in the A/B Test Variant Event Data export:

Project Name
Key
Channel Identifier
Test Name
Variant
Event Name
Notification Attribution
Location
Count
Value