1. Home
  2. /Integrations
  3. /Amazon Web Services

Amazon Web Services

AIRSHIP MAINTAINED INTEGRATION

This integration is maintained by Airship. Please contact Airship for support.

Airship Real-Time Data StreamingA service that delivers engagement events in real time via the Data Streaming API or an Airship partner integration. currently supports two integrations with Amazon Web Services: S3 and Kinesis. This document walks through the process of connecting to each service.

AWS Integration Requirements

Amazon S3

S3 is a cloud storage service. Once you have integrated Real-Time Data Streaming with S3, your event stream data should begin funnelling into a bucket. From there, how you use the data is up to you, but some potential ideas are:

  • Output CSV files with user-level send and open information, and import these files into your CRM system.
  • Output your data in JSON format, and use that data along with Amazon Redshift  to perform detailed analysis of your users.
 Tip

Be sure to regularly audit your event stream S3 bucket. Real-Time Data Streaming outputs large amounts of data, which can lead to expensive AWS bills if not managed appropriately.

Setting up AWS S3

First, create an S3 bucket, then grant permissions.

  1. Log in to Amazon Web Services .
  2. Select Services, then S3.
  3. Create an S3 bucket, selecting the appropriate S3 Region for your project. For more information on S3 regions, see AWS service endpoints .
  4. Go to Permissions, then Access Control List.
  5. Select Add Account.
  6. In the canonical ID / account field, enter Airship’s ID: 7e7585ea39ccec40d8297a9038ba7f211b1c4a48994c2c702298aca8732f9f0e
  7. Select List objects and Write objects. You can select additional permissions, but your bucket must have these permissions to support an Airship integration.
  8. Select Save.

Configuring the AWS S3 Integration

You will need the bucket name from Setting up AWS S3.

In Airship:

  1. Go to Settings.
  2. Under Project Configuration, select Manage for Real-Time Data Streaming.
  3. Under Real-Time Data Streaming, select Amazon S3.
  4. Follow the onscreen instructions to configure the integration.

    You can select JSON or CSV as the output format and have the option to encrypt your data using server-side encryption  or export as a compressed gzip file. Tag changes are not supported by CSV because their dynamic nature cannot be easily represented in a single row column structure.

Output Structure and Files

Your S3 bucket’s directory structure and files will be named using the following patterns:

CSV
appKey + “/” + integrationId + “/S3_CSV/” + eventType + “/” + year + “_” + month + “_” + day + “/” + year + “_” + month + “_” + day + “_” + hour + “_” + minute + “_” + second + “.csv”

For a list of headers for each supported event type, see S3 CSV Headers below.

JSON
appKey + “/” + integrationId + “/S3_JSON/” + year + “_” + month + “_” + day + “/” + year + “_” + month + “_” + day + “_” + hour + “_” + minute + “_” + second + “.json”

There will be one file generated per hour, assuming the relevant event occurred during that hour.

 Note

File size maximum is 15 GB. If the volume of events for an app results in a file exceeding 15 GB, you may instead see multiple files.

Sample File

Here are a few lines from a sample CSV file for LOCATION events:

Nonet.id,event.type,event.occurred,device.platform,device.channel_id,device.named_user_id,LOCATION.latitude,LOCATION.longitude,LOCATION.foreground,LOCATION.session_id
00000000-0000-0000-0000-000000000000,LOCATION,2015-11-18T01:21:33.180Z,IOS,90823094-1234-94b2-sb39-099s9018gx55,,14.5224123,-22.1236212,false,
00000000-0000-0000-0000-000000000000,LOCATION,2015-11-18T01:21:33.171Z,IOS,90823094-1234-94b2-sb39-099s9018gx55,,14.5224123,-22.1236212,false,
00000000-0000-0000-0000-000000000000,LOCATION,2015-11-18T01:21:33.162Z,IOS,90823094-1234-94b2-sb39-099s9018gx55,,14.5224123,-22.1236212,false,
...

If you choose to output your data as JSON, the files will follow the format laid out in the Data Streaming API Reference.

Amazon Kinesis

Kinesis is AWS’s conduit for streaming data. Once you have integrated Real-Time Data Streaming with Kinesis, you will be able to load and analyze mobile events in real time.

Once you have Airship data running through a functioning Kinesis stream, you will likely want to begin analyzing and processing that data. To that end, AWS Lambda  is a backend compute service that can process events in real time. Lambda responds to events that occur on other AWS products, such as a Kinesis stream.

As an example, let’s say you have built out a mapping between Named User IDs and associated email addresses, and you would like to send an email to any user that chooses to uninstall your app. Here’s a typical app uninstall event:

{
   "id" : "ff76bb85-74bc-4511-a3bf-11b6117784db",
   "type": "UNINSTALL",
   "offset": 1235,
   "occurred": "2015-05-03T02:32:12.088Z",
   "processed": "2015-05-03T12:12:43.180Z",
   "device": {"named_user_id": "named-user-id-123"}
}

You would then use Lambda to create a function associated with your Airship-Kinesis stream. The function should search for UNINSTALL type events that occurred on devices with associated Named User IDs and then send those devices an email:

def uninstall_email(event, context):
   if (event['type'] == 'UNINSTALL' and 'named_user_id' in event['device']):
      send_email(event['device']['named_user_id'])

The above pseudo-code finds events that fit the parameters and then uses the send_email function to send an email to the appropriate Named User.

There are thousands of ways to process and analyze your Airship data with Kinesis and Lambda, e.g., editing your CRM database when certain events come in, writing streams into Redshift or Dynamo DB, monitoring tag changes. The possibilities are endless.

Setting up AWS Kinesis

First, create a Kinesis stream:

  1. Log in to the AWS Console.
  2. Go to Kinesis.
  3. Select Create Data Stream, and follow the steps. When complete, copy the ARN. You will need it when creating a policy.

Next, create a policy. The following represents the minimum policy to support Airship Real-Time Data Streaming:

  1. Go to Services, then search for and select “IAM” (Identity and Access Management).
  2. Go to Policies and select Create Policy.
  3. For the Service, select Kinesis.
  4. Assign actions for the policy. At a minimum, you should assign DescribeStream, GetShardIterator, GenerateKeyData,GetRecords, PutRecord, and PutRecords (found under Read and Write categories).
  5. For Resources, enter the ARN.
  6. Review, the submit the policy.

Now you can create a user to attach the policy to and to represent Airship in your Kinesis instance:

  1. Go to Users and select Add User.
  2. Enter a user name
  3. For AWS Access Type, select Programmatic Access
  4. Select Next.
  5. Under Set permissions boundary, select Use a permissions boundary to control the maximum user permissions, and select the policy you created in previous steps.
  6. When you finish creating your user, copy the Access key ID and Secret access key. You will need these when configuring your Kinesis integration in Airship.

Configuring the AWS Kinesis Integration

You will need your Access Key, Access Secret, and Kinesis Stream Name. You can get the Access Key ID and Secret Access ID from the User you created for your integration.

In Airship:

  1. Go to Settings.
  2. Under Project Configuration, select Manage for Real-Time Data Streaming.
  3. Under Real-Time Data Streaming, select Amazon Web Services / Amazon Kinesis.
  4. Follow the onscreen instructions to configure the integration.

Appendix: S3 CSV Headers

You may choose the CSV format for your Amazon S3 data output. These are the headers for supported event types.

OPEN
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, LAST_DELIVERED_PUSH_ID, LAST_DELIVERED_GROUP_ID, LAST_DELIVERED_VARIANT_ID, LAST_DELIVERED_TIME, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION
CLOSE
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION
CUSTOM
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, LAST_DELIVERED_PUSH_ID, LAST_DELIVERED_GROUP_ID, LAST_DELIVERED_VARIANT_ID, LAST_DELIVERED_TIME, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION, NAME, VALUE, TRANSACTION, INTERACTION_ID, INTERACTION_TYPE, CUSTOMER_ID
LOCATION
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION, LATITUDE, LONGITUDE, FOREGROUND
SEND
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
TAG_CHANGE
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE
RICH_DELETE
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
RICH_DELIVERY
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
RICH_READ
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
REGION
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION, ACTION, REGION_ID, NAME, SOURCE, SOURCE_ID
IN_APP_MESSAGE_DISPLAY
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION
IN_APP_MESSAGE_EXPIRATION
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION, TIME_SENT, RESOLUTION_TYPE, TIME_EXPIRED, REPLACING_PUSH_PUSH_ID, REPLACING_PUSH_GROUP_ID, REPLACING_PUSH_VARIANT_ID, REPLACING_PUSH_TIME
IN_APP_MESSAGE_RESOLUTION
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION, TIME_SENT, RESOLUTION_TYPE, BUTTON_ID, BUTTON_DESCRIPTION, DURATION
CONTROL
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
SCREEN_VIEWED
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION, DURATION, PREVIOUS_SCREEN, VIEWED_SCREEN
FIRST_OPEN
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE
UNINSTALL
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE
PUSH_BODY
ID, TYPE, OCCURRED, PROCESSED, OFFSET, PUSH_ID, GROUP_ID, VARIANT_ID, PAYLOAD, TRIMMED, RESOURCE
WEB_CLICK
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
WEB_SESSION
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, LAST_DELIVERED_PUSH_ID, LAST_DELIVERED_GROUP_ID, LAST_DELIVERED_VARIANT_ID, LAST_DELIVERED_TIME, SESSION
IN_APP_BUTTON_TAP
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, BUTTON_ID, GROUP_ID, RENDERED_LOCALE, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME
IN_APP_EXPERIENCES
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, EVENT_NAME, GROUP_ID, PUSH_ID, SESSION, SURVEY_TYPE, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, VARIANT_ID
IN_APP_FORM_DISPLAY
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, GROUP_ID, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, IN_APP_FORM_EVENT_TYPE
IN_APP_FORM_RESULT
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, GROUP_ID, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, IN_APP_FORM_EVENT_TYPE
IN_APP_PAGE_SWIPE
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, FROM_PAGE_INDEX, FROM_PAGE_IDENTIFIER, GROUP_ID, RENDERED_LOCALE, PAGER_IDENTIFIER, PUSH_ID, SESSION, TIME_SENT, TO_PAGE_IDENTIFIER, TO_PAGE_INDEX, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME
IN_APP_PAGE_VIEW
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, COMPLETED, GROUP_ID, RENDERED_LOCALE, PAGE_COUNT, PAGE_IDENTIFIER, PAGE_INDEX, PAGER_IDENTIFIER, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, VIEWED_COUNT
IN_APP_PAGER_COMPLETED
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, COMPLETED, GROUP_ID, RENDERED_LOCALE, PAGE_COUNT, PAGE_IDENTIFIER, PAGE_INDEX, PAGER_IDENTIFIER, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME
IN_APP_PAGER_SUMMARY
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, COMPLETED, GROUP_ID, RENDERED_LOCALE, PAGE_COUNT, PAGE_IDENTIFIER, PAGER_IDENTIFIER, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME
SEND_ABORTED
ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, GROUP_ID, PUSH_ID, REASON