Amazon Web Services
AIRSHIP MAINTAINED INTEGRATION
This integration is maintained by Airship. Please contact Airship for support.Airship Real-Time Data StreamingA service that delivers engagement events in real time via the Data Streaming API or an Airship partner integration. currently supports two integrations with Amazon Web Services: S3 and Kinesis. This document walks through the process of connecting to each service.
AWS Integration Requirements
- Accounts
- AWS S3 or Kinesis
- Airship — Must include both:
- Messaging
- Real-Time Data Streaming
Amazon S3
S3 is a cloud storage service. Once you have integrated Real-Time Data Streaming with S3, your event stream data should begin funnelling into a bucket. From there, how you use the data is up to you, but some potential ideas are:
- Output CSV files with user-level send and open information, and import these files into your CRM system.
- Output your data in JSON format, and use that data along with Amazon Redshift to perform detailed analysis of your users.
Be sure to regularly audit your event stream S3 bucket. Real-Time Data Streaming outputs large amounts of data, which can lead to expensive AWS bills if not managed appropriately.
Setting up AWS S3
Log in to Amazon Web Services . Click Services and go to S3.
Create a bucket. Follow Amazon’s process for creating an S3 bucket, selecting the appropriate S3 Region for your project. For more information on S3 regions, see Amazon Regions .
Grant permissions to your newly-created bucket.
- Go to Permissions » Access Control List.
- Click Add Account, and use Airship’s ID as the canonical ID / account field:
7e7585ea39ccec40d8297a9038ba7f211b1c4a48994c2c702298aca8732f9f0e
- Select List objects and Write objects. You can select additional permissions, but your bucket must have these permissions to support an Airship integration.
- Click Save.
Configuring the AWS S3 Integration
You will need the bucket name from Setting up AWS S3.
In Airship:
- Go to Settings » Project Configuration and click Manage for Real-Time Data Streaming.
- Under Real-Time Data Streaming, select Amazon S3. Previously configured integrations are listed under Enabled Integrations.
- Follow the onscreen instructions to configure the integration.
You can select JSON or CSV as the output format and have the option to encrypt your data using server-side encryption or export as a compressed gzip file. Tag Changes are not supported by CSV because their dynamic nature cannot be easily represented in a single row column structure.
Output Structure and Files
Your S3 bucket’s directory structure and files will be named using the following patterns:
- CSV
- appKey + “/” + integrationId + “/S3_CSV/” + eventType + “/” + year + “_” + month + “_” + day + “/” + year + “_” + month + “_” + day + “_” + hour + “_” + minute + “_” + second + “.csv”
For a list of headers for each supported event type, see S3 CSV Headers below.
- JSON
- appKey + “/” + integrationId + “/S3_JSON/” + year + “_” + month + “_” + day + “/” + year + “_” + month + “_” + day + “_” + hour + “_” + minute + “_” + second + “.json”
There will be one file generated per hour, assuming the relevant event occurred during that hour.
File size maximum is 15 GB. If the volume of events for an app results in a file exceeding 15 GB, you may instead see multiple files.
Sample File
Here are a few lines from a sample CSV file for LOCATION
events:
Nonet.id,event.type,event.occurred,device.platform,device.channel_id,device.named_user_id,LOCATION.latitude,LOCATION.longitude,LOCATION.foreground,LOCATION.session_id
00000000-0000-0000-0000-000000000000,LOCATION,2015-11-18T01:21:33.180Z,IOS,90823094-1234-94b2-sb39-099s9018gx55,,14.5224123,-22.1236212,false,
00000000-0000-0000-0000-000000000000,LOCATION,2015-11-18T01:21:33.171Z,IOS,90823094-1234-94b2-sb39-099s9018gx55,,14.5224123,-22.1236212,false,
00000000-0000-0000-0000-000000000000,LOCATION,2015-11-18T01:21:33.162Z,IOS,90823094-1234-94b2-sb39-099s9018gx55,,14.5224123,-22.1236212,false,
...
If you choose to output your data as JSON, the files will follow the format laid out in the Data Streaming API Reference.
Amazon Kinesis
Kinesis is AWS’s conduit for streaming data. Once you have integrated Real-Time Data Streaming with Kinesis, you will be able to load and analyze mobile events in real time.
Once you have Airship data running through a functioning Kinesis stream, you will likely want to begin analyzing and processing that data. To that end, AWS Lambda is a backend compute service that can process events in real time. Lambda responds to events that occur on other AWS products, such as a Kinesis stream.
As an example, let’s say you have built out a mapping between Named User IDs and associated email addresses, and you would like to send an email to any user that chooses to uninstall your app. Here’s a typical app uninstall event:
{
"id" : "ff76bb85-74bc-4511-a3bf-11b6117784db",
"type": "UNINSTALL",
"offset": 1235,
"occurred": "2015-05-03T02:32:12.088Z",
"processed": "2015-05-03T12:12:43.180Z",
"device": {"named_user_id": "named-user-id-123"}
}
You would then use Lambda to create a function associated with your Airship-Kinesis stream.
The function should search for UNINSTALL
type events that occurred on devices with
associated Named User IDs and then send those devices an email:
def uninstall_email(event, context):
if (event['type'] == 'UNINSTALL' and 'named_user_id' in event['device']):
send_email(event['device']['named_user_id'])
The above pseudo-code finds events that fit the parameters and then uses the send_email
function to send an email to the appropriate Named User.
There are thousands of ways to process and analyze your Airship data with Kinesis and Lambda, e.g., editing your CRM database when certain events come in, writing streams into Redshift or Dynamo DB, monitoring tag changes. The possibilities are endless.
Setting up AWS Kinesis
Log in to the AWS Console.
Create a Kinesis Stream. Go to Kinesis in the AWS console, click Create Data Stream, and follow the steps. When complete, copy the ARN; you will need the ARN when creating a policy.
Create a Policy. The following represents the minimum policy to support Airship Real-Time Data Streaming.
- Go to Services and search for IAM — Identity and Access Management.
- Go to Policies, and click Create Policy
- For the Service select Kinesis
- Assign Actions for the policy. At a minimum, you should assign
DescribeStream
,GetShardIterator
,GenerateKeyData
,GetRecords
,PutRecord
, andPutRecords
(found under Read and Write categories, respectively). - For Resources provide the ARN.
- Review and then submit the policy.
Create a User to attach the policy to. This user will represent Airship in your Kinesis instance.
- Go to Users and click Add User.
- Enter a User name and select Programmatic Access for the AWS Access Type, and then click Next.
- Under Set permissions boundary, select Use a permissions boundary to control the maximum user permissions and select the policy you created in previous steps.
- When you finish creating your user, copy the Access key ID and Secret access key — you will need these when enabling your Kinesis integration in Airship.
Configuring the AWS Kinesis Integration
You will need your Access Key, Access Secret, and Kinesis Stream Name. You can get the Access Key ID and Secret Access ID from the User you created for your integration.
In Airship:
- Go to Settings » Project Configuration and click Manage for Real-Time Data Streaming.
- Under Real-Time Data Streaming, select Amazon Web Services / Amazon Kinesis. Previously configured integrations are listed under Enabled Integrations.
- Follow the onscreen instructions to configure the integration.
Appendix: S3 CSV Headers
You may choose the CSV format for your Amazon S3 data output. These are the headers for supported event types.
- OPEN
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, LAST_DELIVERED_PUSH_ID, LAST_DELIVERED_GROUP_ID, LAST_DELIVERED_VARIANT_ID, LAST_DELIVERED_TIME, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION
- CLOSE
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION
- CUSTOM
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, LAST_DELIVERED_PUSH_ID, LAST_DELIVERED_GROUP_ID, LAST_DELIVERED_VARIANT_ID, LAST_DELIVERED_TIME, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION, NAME, VALUE, TRANSACTION, INTERACTION_ID, INTERACTION_TYPE, CUSTOMER_ID
- LOCATION
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION, LATITUDE, LONGITUDE, FOREGROUND
- SEND
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
- TAG_CHANGE
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE
- RICH_DELETE
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
- RICH_DELIVERY
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
- RICH_READ
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
- REGION
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION, ACTION, REGION_ID, NAME, SOURCE, SOURCE_ID
- IN_APP_MESSAGE_DISPLAY
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION
- IN_APP_MESSAGE_EXPIRATION
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION, TIME_SENT, RESOLUTION_TYPE, TIME_EXPIRED, REPLACING_PUSH_PUSH_ID, REPLACING_PUSH_GROUP_ID, REPLACING_PUSH_VARIANT_ID, REPLACING_PUSH_TIME
- IN_APP_MESSAGE_RESOLUTION
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, SESSION, TIME_SENT, RESOLUTION_TYPE, BUTTON_ID, BUTTON_DESCRIPTION, DURATION
- CONTROL
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
- SCREEN_VIEWED
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, SESSION, DURATION, PREVIOUS_SCREEN, VIEWED_SCREEN
- FIRST_OPEN
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE
- UNINSTALL
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE
- PUSH_BODY
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, PUSH_ID, GROUP_ID, VARIANT_ID, PAYLOAD, TRIMMED, RESOURCE
- WEB_CLICK
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, PUSH_ID, GROUP_ID, VARIANT_ID
- WEB_SESSION
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, LAST_DELIVERED_PUSH_ID, LAST_DELIVERED_GROUP_ID, LAST_DELIVERED_VARIANT_ID, LAST_DELIVERED_TIME, SESSION
- IN_APP_BUTTON_TAP
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, BUTTON_ID, GROUP_ID, RENDERED_LOCALE, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME
- IN_APP_EXPERIENCES
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, EVENT_NAME, GROUP_ID, PUSH_ID, SESSION, SURVEY_TYPE, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, VARIANT_ID
- IN_APP_FORM_DISPLAY
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, GROUP_ID, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, IN_APP_FORM_EVENT_TYPE
- IN_APP_FORM_RESULT
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, GROUP_ID, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, IN_APP_FORM_EVENT_TYPE
- IN_APP_PAGE_SWIPE
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, FROM_PAGE_INDEX, FROM_PAGE_IDENTIFIER, GROUP_ID, RENDERED_LOCALE, PAGER_IDENTIFIER, PUSH_ID, SESSION, TIME_SENT, TO_PAGE_IDENTIFIER, TO_PAGE_INDEX, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME
- IN_APP_PAGE_VIEW
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, COMPLETED, GROUP_ID, RENDERED_LOCALE, PAGE_COUNT, PAGE_IDENTIFIER, PAGE_INDEX, PAGER_IDENTIFIER, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME, VIEWED_COUNT
- IN_APP_PAGER_COMPLETED
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, COMPLETED, GROUP_ID, RENDERED_LOCALE, PAGE_COUNT, PAGE_IDENTIFIER, PAGE_INDEX, PAGER_IDENTIFIER, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME
- IN_APP_PAGER_SUMMARY
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, APP_DEFINED_ID, COMPLETED, GROUP_ID, RENDERED_LOCALE, PAGE_COUNT, PAGE_IDENTIFIER, PAGER_IDENTIFIER, PUSH_ID, SESSION, TIME_SENT, TRIGGERING_PUSH_ID, TRIGGERING_GROUP_ID, TRIGGERING_VARIANT_ID, TRIGGERING_TIME
- SEND_ABORTED
- ID, TYPE, OCCURRED, PROCESSED, OFFSET, NAMED_USER_ID, ANDROID_CHANNEL, IOS_CHANNEL, AMAZON_CHANNEL, CHANNEL, DEVICE_TYPE, APP_PACKAGE_NAME, APP_VERSION DEVICE_MODEL, DEVICE_OS, CARRIER, PUSH_OPT_IN, BACKGROUND_PUSH_ENABLED, LOCATION_ENABLED, LOCALE_LANGUAGE_CODE, LOCALE_COUNTRY_CODE, LOCATION_PERMISSION, IANA_TIMEZONE, GROUP_ID, PUSH_ID, REASON
Categories