Amazon Web Services

Airship Real-Time Data StreamingA service that delivers engagement events in real time via the Data Streaming API or an Airship partner integration. currently supports two integrations with Amazon Web Services: S3 and Kinesis. This document walks through the process of connecting to each service.

AWS Integration Requirements

• Accounts
  1. AWS S3 or Kinesis
  2. Airship — Must include both:
     • Messaging
     • Real-Time Data Streaming

Amazon S3

S3 is a cloud storage service. Once you have integrated Real-Time Data Streaming with S3, your event stream data should begin funnelling into a bucket. From there, how you use the data is up to you, but some potential ideas are:

• Output CSV files with user-level send and open information, and import these files into your CRM system.
• Output your data in JSON format, and use that data along with Amazon Redshift  to perform detailed analysis of your users.

Setting up AWS S3

1. Log in to Amazon Web Services . Click Services and go to S3.

2. Create a bucket. Follow Amazon’s process for creating an S3 bucket, selecting the appropriate S3 Region for your project. For more information on S3 regions, see Amazon Regions .

3. Grant permissions to your newly-created bucket.

   1. Go to Permissions » Access Control List.
   2. Click Add Account, and use Airship’s ID as the canonical ID / account field: 7e7585ea39ccec40d8297a9038ba7f211b1c4a48994c2c702298aca8732f9f0e
   3. Select List objects and Write objects. You can select additional permissions, but your bucket must have these permissions to support an Airship integration.
   4. Click Save.

Configuring the AWS S3 Integration

You will need the bucket name from Setting up AWS S3.

In Airship:

1. Go to Settings » Project Configuration and click Manage for Real-Time Data Streaming.
2. Under Real-Time Data Streaming, select Amazon S3. Previously configured integrations are listed under Enabled Integrations.
3. Follow the onscreen instructions to configure the integration.

   You can select JSON or CSV as the output format and have the option to encrypt your data using server-side encryption  or export as a compressed gzip file. Tag Changes are not supported by CSV because their dynamic nature cannot be easily represented in a single row column structure.

Output Structure and Files

Your S3 bucket’s directory structure and files will be named using the following patterns:

CSV

appKey + “/” + integrationId + “/S3_CSV/” + eventType + “/” + year + “_” + month + “_” + day + “/” + year + “_” + month + “_” + day + “_” + hour + “_” + minute + “_” + second + “.csv”

For a list of headers for each supported event type, see S3 CSV Headers below.

JSON

appKey + “/” + integrationId + “/S3_JSON/” + year + “_” + month + “_” + day + “/” + year + “_” + month + “_” + day + “_” + hour + “_” + minute + “_” + second + “.json”

There will be one file generated per hour, assuming the relevant event occurred during that hour.

Sample File

Here are a few lines from a sample CSV file for LOCATION events:

Nonet.id,event.type,event.occurred,device.platform,device.channel_id,device.named_user_id,LOCATION.latitude,LOCATION.longitude,LOCATION.foreground,LOCATION.session_id
00000000-0000-0000-0000-000000000000,LOCATION,2015-11-18T01:21:33.180Z,IOS,90823094-1234-94b2-sb39-099s9018gx55,,14.5224123,-22.1236212,false,
00000000-0000-0000-0000-000000000000,LOCATION,2015-11-18T01:21:33.171Z,IOS,90823094-1234-94b2-sb39-099s9018gx55,,14.5224123,-22.1236212,false,
00000000-0000-0000-0000-000000000000,LOCATION,2015-11-18T01:21:33.162Z,IOS,90823094-1234-94b2-sb39-099s9018gx55,,14.5224123,-22.1236212,false,
...

If you choose to output your data as JSON, the files will follow the format laid out in the Data Streaming API Reference.

Amazon Kinesis

Kinesis is AWS’s conduit for streaming data. Once you have integrated Real-Time Data Streaming with Kinesis, you will be able to load and analyze mobile events in real time.

Once you have Airship data running through a functioning Kinesis stream, you will likely want to begin analyzing and processing that data. To that end, AWS Lambda  is a backend compute service that can process events in real time. Lambda responds to events that occur on other AWS products, such as a Kinesis stream.

As an example, let’s say you have built out a mapping between Named User IDs and associated email addresses, and you would like to send an email to any user that chooses to uninstall your app. Here’s a typical app uninstall event:

{
   "id" : "ff76bb85-74bc-4511-a3bf-11b6117784db",
   "type": "UNINSTALL",
   "offset": 1235,
   "occurred": "2015-05-03T02:32:12.088Z",
   "processed": "2015-05-03T12:12:43.180Z",
   "device": {"named_user_id": "named-user-id-123"}
}

You would then use Lambda to create a function associated with your Airship-Kinesis stream. The function should search for UNINSTALL type events that occurred on devices with associated Named User IDs and then send those devices an email:

def uninstall_email(event, context):
   if (event['type'] == 'UNINSTALL' and 'named_user_id' in event['device']):
      send_email(event['device']['named_user_id'])

The above pseudo-code finds events that fit the parameters and then uses the send_email function to send an email to the appropriate Named User.

There are thousands of ways to process and analyze your Airship data with Kinesis and Lambda, e.g., editing your CRM database when certain events come in, writing streams into Redshift or Dynamo DB, monitoring tag changes. The possibilities are endless.

Setting up AWS Kinesis

1. Log in to the AWS Console.

2. Create a Kinesis Stream. Go to Kinesis in the AWS console, click Create Data Stream, and follow the steps. When complete, copy the ARN; you will need the ARN when creating a policy.

3. Create a Policy. The following represents the minimum policy to support Airship Real-Time Data Streaming.

   1. Go to Services and search for IAM — Identity and Access Management.
   2. Go to Policies, and click Create Policy
   3. For the Service select Kinesis
   4. Assign Actions for the policy. At a minimum, you should assign DescribeStream, GetShardIterator, GenerateKeyData,GetRecords, PutRecord, and PutRecords (found under Read and Write categories, respectively).
   5. For Resources provide the ARN.
   6. Review and then submit the policy.

4. Create a User to attach the policy to. This user will represent Airship in your Kinesis instance.

   1. Go to Users and click Add User.
   2. Enter a User name and select Programmatic Access for the AWS Access Type, and then click Next.
   3. Under Set permissions boundary, select Use a permissions boundary to control the maximum user permissions and select the policy you created in previous steps.
   4. When you finish creating your user, copy the Access key ID and Secret access key — you will need these when enabling your Kinesis integration in Airship.

Configuring the AWS Kinesis Integration

You will need your Access Key, Access Secret, and Kinesis Stream Name. You can get the Access Key ID and Secret Access ID from the User you created for your integration.

In Airship:

1. Go to Settings » Project Configuration and click Manage for Real-Time Data Streaming.
2. Under Real-Time Data Streaming, select Amazon Web Services / Amazon Kinesis. Previously configured integrations are listed under Enabled Integrations.
3. Follow the onscreen instructions to configure the integration.

Appendix: S3 CSV Headers

You may choose the CSV format for your Amazon S3 data output. These are the headers for supported event types.

OPEN

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, LAST_DELIVERED_PUSH_ID, LAST_DELIVERED_GROUP_ID, LAST_DELIVERED_VARIANT_ID, LAST_DELIVERED_TIME, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION

CLOSE

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION

CUSTOM

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, LAST_DELIVERED_PUSH_ID, LAST_DELIVERED_GROUP_ID, LAST_DELIVERED_VARIANT_ID, LAST_DELIVERED_TIME, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION, NAME, VALUE, TRANSACTION, INTERACTION_ID, INTERACTION_TYPE, CUSTOMER_ID

LOCATION

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION, LATITUDE, LONGITUDE, FOREGROUND

SEND

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID

TAG_CHANGE

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE

RICH_DELETE

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID

RICH_DELIVERY

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID

RICH_READ

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID

REGION

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION, ACTION, REGION_ID, NAME, SOURCE, SOURCE_ID

IN_APP_MESSAGE_DISPLAY

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION

IN_APP_MESSAGE_EXPIRATION

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION, TIME_SENT, RESOLUTION_TYPE, TIME_EXPIRED, REPLACING_PUSH_PUSH_ID, REPLACING_PUSH_GROUP_ID, REPLACING_PUSH_VARIANT_ID, REPLACING_PUSH_TIME

IN_APP_MESSAGE_RESOLUTION

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION, TIME_SENT, RESOLUTION_TYPE, BUTTON_ID, BUTTON_DESCRIPTION, DURATION

CONTROL

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID

SCREEN_VIEWED

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION, DURATION, PREVIOUS_SCREEN, VIEWED_SCREEN

FIRST_OPEN

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE

UNINSTALL

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE

PUSH_BODY

ID, TYPE, OCCURRED, PROCESSED, OFFSET, PUSH_ID, GROUP_ID, VARIANT_ID, PAYLOAD, TRIMMED, RESOURCE

WEB_CLICK

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID

WEB_SESSION

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, LAST_DELIVERED_PUSH_ID, LAST_DELIVERED_GROUP_ID, LAST_DELIVERED_VARIANT_ID, LAST_DELIVERED_TIME, SESSION

IN_APP_BUTTON_TAP

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, BUTTON_ID, GROUP_ID, RENDERED_LOCALE, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME

IN_APP_EXPERIENCES

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, EVENT_NAME, GROUP_ID, PUSH_ID, SESSION, SURVEY_TYPE, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, VARIANT_ID

IN_APP_FORM_DISPLAY

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, GROUP_ID, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, IN_APP_FORM_EVENT_TYPE

IN_APP_FORM_RESULT

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, GROUP_ID, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, IN_APP_FORM_EVENT_TYPE

IN_APP_PAGE_SWIPE

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, FROM_PAGE_INDEX, FROM_PAGE_IDENTIFIER, GROUP_ID, RENDERED_LOCALE, PAGER_IDENTIFIER, PUSH_ID, SESSION, TIME_SENT, TO_PAGE_IDENTIFIER, TO_PAGE_INDEX, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME

IN_APP_PAGE_VIEW

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, COMPLETED, GROUP_ID, RENDERED_LOCALE, PAGE_COUNT, PAGE_IDENTIFIER, PAGE_INDEX, PAGER_IDENTIFIER, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, VIEWED_COUNT

IN_APP_PAGER_COMPLETED

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, COMPLETED, GROUP_ID, RENDERED_LOCALE, PAGE_COUNT, PAGE_IDENTIFIER, PAGE_INDEX, PAGER_IDENTIFIER, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME

IN_APP_PAGER_SUMMARY

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, COMPLETED, GROUP_ID, RENDERED_LOCALE, PAGE_COUNT, PAGE_IDENTIFIER, PAGER_IDENTIFIER, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME

SEND_ABORTED

ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, GROUP_ID, PUSH_ID, REASON