1. Home
  2. /Guides
  3. /Messaging
  4. /User guide
  5. /Data + Analytics
  6. /Reporting
  7. /A/B Test Reports

A/B Test Reports

A/B Tests have an overview report, and a message report for each variant.

Go to Messages » Messages Overview, and click for an A/B test. The report contains these main elements:

  • Header: Displays the test name, sent date, time, and time zone.
  • Performance: Contains statistical data for each variant and the control group. Click a variant name to view its message report.
  • Message Detail: Contains the same information shown in the Review step of the A/B Test composer and the Message Detail section of a message report. Click the arrows to page through the various previews. The platform and display type dynamically update in the dropdown menu above. You can also select a preview or variant directly from the dropdown menus.

To export, click Download CSV and select Performance Data or Variant Event Data.

 Note

Engagement data is sent to Airship as soon as it becomes available. Often data is delayed because of connectivity issues with a user’s carrier, wifi, power, etc. You should wait at least 12 to 24 hours before acting on this data to allow for potential lags.

Performance Data

The following data are included in the A/B Test Performance Data export:

FieldDefinition
Project NameYour project name.
KeyUnique authentication key for your app.
Channel IdentifierUnique identifier that refers to the entire push send. The channel identifier will be the same for each variant within an A/B Test, with N/A for users in the control group who do not receive a notification.
Sent Time (UTC)UTC timestamp.
Test NameUser-friendly name for your A/B Test.
VariantThe letter corresponding to the message variant. A, B, C, etc.
AudienceDescription of the selected audience for the A/B Test. One of All Users, Target Specific Users, or Test Users.
Notification AlertAlert text for the given variant.
SendsThe number of sends of each variant.
Sample SizeThe number of audience members in the control group.
Indirect OpensThe number of opens that occurred within 12 hours of the notification, regardless of attribution.
Direct OpensThe number of opens that were attributed directly to interacting with the notification.
Influenced OpensSee Reports: Influenced Opens.
Opens & SessionsThe number of opens and sessions for the control group within 12 hours of the notification. The percentage is the Opens & Sessions count divided by the Sample Size count.

Variant Event Data

If you are using Custom EventsEvents that indicate that a user performed a predefined action, such as adding an item to a shopping cart, viewing a screen, or clicking an Unsubscribe button. Custom events can trigger automation, including Sequences, Scenes, and Surveys. You can code them into your app or website, or send them to Airship from an external source using the custom event API. Custom events contain properties that you can use to personalize messages., you can download a report of event conversions and associated values, broken out by variant or control group.

The following data are included in the A/B Test Variant Event Data export:

  • Project Name
  • Key
  • Channel Identifier
  • Test Name
  • Variant
  • Event Name
  • Notification Attribution
  • Location
  • Count
  • Value