Amazon Web Services

Airship Real-Time Data Streaming currently supports two integrations with Amazon Web Services: S3 and Kinesis. This document walks through the process of connecting to each service.

Amazon S3

S3 is a cloud storage service. Once you have integrated Real-Time Data Streaming with S3, your event stream data should begin funnelling into a bucket. From there, how you use the data is up to you, but some potential ideas are:

  • Output CSV files with user-level send and open information, and import these files into your CRM system.

  • Output your data in JSON format, and use that data along with Amazon Redshift to perform detailed analysis of your users.
 Tip

Be sure to regularly audit your event stream S3 bucket. Real-Time Data Streaming outputs large amounts of data, which can lead to expensive AWS bills if not managed appropriately.

Setup AWS S3

  1. Log in to Amazon Web Services. Click Services and go to S3.
  2. Create a bucket. Follow Amazon's process for creating an S3 bucket, selecting US Standard for S3 Region. For more information on S3 regions, see Amazon Regions.

  3. Grant permissions to your newly-created bucket.
    1. Go to Permissions » Access Control List.
    2. Click Add Account, and use Airship’s ID as the canonical ID / account field: 7e7585ea39ccec40d8297a9038ba7f211b1c4a48994c2c702298aca8732f9f0e
    3. Select List objects and Write objects. You can select additional permissions, but your bucket must have these permissions to support an Airship integration.
    4. Click Save.

Configure your S3 Integration

  1. Open your messaging project and go to Settings » Real-Time Data Streaming.
  2. Under Real-Time Data Streaming, click Amazon S3.
     Tip

    Previously configured integrations are listed under Enabled Integrations.

  3. Set up a new Amazon S3 integration:
    • Enter a user-friendly name and description.
    • Enter the S3 Bucket Name for the bucket you created in the AWS setup steps above.
    • Select the Output Format.
    • Optionally check the boxes to Encrypt your data using server-side encryption or export as a compressed gzip file. If you select gzip, the file will have the extension .gz.
    • Select the event types you want to send to your S3 bucket.

 Note

Tag Changes are not supported by CSV because their dynamic nature cannot be easily represented in a single row column structure.

1. Click the Save button.

Output Structure and Files

Your S3 bucket's directory structure and files will be named using the following patterns:

CSV
appKey + "/" + integrationId + "/S3_CSV/" + eventType + "/" + year + "_" + month + "_" + day + "/" + year + "_" + month + "_" + day + "_" + hour + "_" + minute + "_" + second + ".csv"

For a list of headers for each supported event type, see S3 CSV Headers below.
JSON
appKey + "/" + integrationId + "/S3_JSON/" + year + "_" + month + "_" + day + "/" + year + "_" + month + "_" + day + "_" + hour + "_" + minute + "_" + second + ".json"

There will be one file generated per hour, assuming the relevant event occurred during that hour.

 Note

File size maximum is 15 GB. If the volume of events for an app results in a file exceeding 15 GB, you may instead see multiple files.

Sample File

Here are a few lines from a sample CSV file for LOCATION events:

Nonet.id,event.type,event.occurred,device.platform,device.channel_id,device.named_user_id,LOCATION.latitude,LOCATION.longitude,LOCATION.foreground,LOCATION.session_id
00000000-0000-0000-0000-000000000000,LOCATION,2015-11-18T01:21:33.180Z,IOS,90823094-1234-94b2-sb39-099s9018gx55,,14.5224123,-22.1236212,false,
00000000-0000-0000-0000-000000000000,LOCATION,2015-11-18T01:21:33.171Z,IOS,90823094-1234-94b2-sb39-099s9018gx55,,14.5224123,-22.1236212,false,
00000000-0000-0000-0000-000000000000,LOCATION,2015-11-18T01:21:33.162Z,IOS,90823094-1234-94b2-sb39-099s9018gx55,,14.5224123,-22.1236212,false,
...

If you choose to output your data as JSON, the files will follow the format laid out in the Data Streaming API Reference.

Amazon Kinesis

Kinesis is AWS's conduit for streaming data. Once you have integrated Real-Time Data Streaming with Kinesis, you will be able to load and analyze mobile events in real time.

Once you have Airship data running through a functioning Kinesis stream, you will likely want to begin analyzing and processing that data. To that end, AWS Lambda is a backend compute service that can process events in real time. Lambda responds to events that occur on other AWS products, such as a Kinesis stream.

As an example, let's say you have built out a mapping between Named User IDs and associated email addresses, and you would like to send an email to any user that chooses to uninstall your app. Here's a typical app uninstall event:

{
   "id" : "ff76bb85-74bc-4511-a3bf-11b6117784db",
   "type": "UNINSTALL",
   "offset": 1235,
   "occurred": "2015-05-03T02:32:12.088Z",
   "processed": "2015-05-03T12:12:43.180Z",
   "device": {"named_user_id": "named-user-id-123"}
}

You would then use Lambda to create a function associated with your Airship-Kinesis stream. The function should search for UNINSTALL type events that occurred on devices with associated Named User IDs and then send those devices an email:

def uninstall_email(event, context):
   if (event['type'] == 'UNINSTALL' and 'named_user_id' in event['device']):
      send_email(event['device']['named_user_id'])

The above pseudo-code finds events that fit the parameters and then uses the send_email function to send an email to the appropriate Named User.

There are thousands of ways to process and analyze your Airship data with Kinesis and Lambda, e.g., editing your CRM database when certain events come in, writing streams into Redshift or Dynamo DB, monitoring tag changes. The possibilities are endless.

Set up AWS Kinesis

  1. Log in to the AWS Console via US East Region/Availability Zone 1.
     Note

    You must be in us-east-1 to set up this integration. Verify this by checking the URL.

  2. Create a Kinesis Stream. Go to Kinesis in the AWS console, click Create Data Stream, and follow the steps. When complete, copy the ARN; you will need the ARN when creating a policy.
  3. Create a Policy. The following represents the minimum policy to support Airship Real-Time Data Streaming.
    1. Go to Services and search for IAM — Identity and Access Management.
    2. Go to Policies, and click Create Policy
    3. For the Service select Kinesis
    4. Assign Actions for the policy. At a minimum, you should assign DescribeStream, GetShardIterator, GetRecords, PutRecord, and PutRecords (found under Read and Write categories, respectively).
    5. For Resources provide the ARN.
    6. Review and then submit the policy.
  4. Create a User to attach the policy to. This user will represent Airship in your Kinesis instance.
    1. Go to Users and click Add User.
    2. Enter a User name and select Programmatic Access for the AWS Access Type, and then click Next.
    3. Under Set permissions boundary, select Use a permissions boundary to control the maximum user permissions and select the policy you created in previous steps.
    4. When you finish creating your user, copy the Access key ID and Secret access key — you will need these when enabling your Kinesis integration in Airship.

Configure your Kinesis Integration

  1. Open your messaging project and go to Settings » Real-Time Data Streaming.
  2. Under Real-Time Data Streaming, click Amazon Web Services / Amazon Kinesis.
     Tip

    Previously configured integrations are listed under Enabled Integrations.

  3. Configure a new Amazon Kinesis integration:
    • Enter a user-friendly Name and Description.
    • Enter the Access Key, Access Secret, and Kinesis Stream Name. You can get the Access key ID and Secret Access ID from the User you created for your integration.
    • Select the events you want to send to your Kinesis stream.
  4. Click Save.

Appendix: S3 CSV Headers

You may choose the CSV format for your Amazon S3 data output. These are the headers for supported event types.

OPEN
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, LAST_DELIVERED_PUSH_ID, LAST_DELIVERED_GROUP_ID, LAST_DELIVERED_VARIANT_ID, LAST_DELIVERED_TIME, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION
CLOSE
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION
CUSTOM
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, LAST_DELIVERED_PUSH_ID, LAST_DELIVERED_GROUP_ID, LAST_DELIVERED_VARIANT_ID, LAST_DELIVERED_TIME, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION, NAME, VALUE, TRANSACTION, INTERACTION_ID, INTERACTION_TYPE, CUSTOMER_ID
LOCATION
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION, LATITUDE, LONGITUDE, FOREGROUND
SEND
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
TAG_CHANGE
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE
RICH_DELETE
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
RICH_DELIVERY
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
RICH_READ
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
REGION
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION, ACTION, REGION_ID, NAME, SOURCE, SOURCE_ID
IN_APP_MESSAGE_DISPLAY
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION
IN_APP_MESSAGE_EXPIRATION
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION, TIME_SENT, RESOLUTION_TYPE, TIME_EXPIRED, REPLACING_PUSH_PUSH_ID, REPLACING_PUSH_GROUP_ID, REPLACING_PUSH_VARIANT_ID, REPLACING_PUSH_TIME
IN_APP_MESSAGE_RESOLUTION
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION, TIME_SENT, RESOLUTION_TYPE, BUTTON_ID, BUTTON_DESCRIPTION, BUTTON_GROUP, DURATION
CONTROL
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
SCREEN_VIEWED
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION, DURATION, PREVIOUS_SCREEN, VIEWED_SCREEN
FIRST_OPEN
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE
UNINSTALL
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE
PUSH_BODY
ID, TYPE, OCCURRED, PROCESSED, OFFSET, PUSH_ID, GROUP_ID, VARIANT_ID, PAYLOAD, TRIMMED, RESOURCE
WEB_CLICK
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
WEB_SESSION
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, LAST_DELIVERED_PUSH_ID, LAST_DELIVERED_GROUP_ID, LAST_DELIVERED_VARIANT_ID, LAST_DELIVERED_TIME, SESSION