Real-Time Data Streaming

With Real-Time Data Streaming (RTDS), Airship is not only your digital engagement service provider, but your most valuable source of mobile engagement data.

About Real-Time Data Streaming

Real-Time Data Streaming (RTDS) delivers an Airship project’s events using the Data Streaming API. When you make a call to the API endpoint, it opens a stream of uninterrupted newline JSON, returning events as they occur. This stream remains open, continuously delivering events until you close your connection, so you can see how users interact with your messaging and app or website in real time.

Airship sends the events you select during setup, along with relevant identifiers that associate them with a particular user and notification or group of notifications. If you need to monitor a subset of events or specific types of events, you can apply filter parameters when opening an event stream to determine the type and quantity of events returned.

All data is available using a single endpoint, with separate base URLs for each data center:

• North America: https://connect.urbanairship.com/api/events
• Europe: https://connect.asnapieu.com/api/events

See the Real-Time Data Streaming API reference and the Data Formats section for a list of all events.

Compliance events For Email and SMS

Compliance events represent operations where users opt in to or out of email and SMS notifications. They are effectively records of your users’ consent to receive SMS or email notifications. You can open an event stream containing only these events for use in maintaining compliance records, proving compliance with data regulations, and ensuring that you respect your audience’s right to privacy.

Compliance events are available to all email and SMS customers through the /api/events/general endpoint. Each event is for a single email address or MSISDNThe mobile phone number of an individual in your Airship audience. Each MSISDN represents an individual mobile device..

For more information, see the following in the RTDS API reference:

• Compliance Event Stream
• Data Formats: Email compliance events
• Data Formats: SMS compliance events

Routing your data

You can send your Airship data to another system using direct, webhook, and partner integrations:

Connect to the test server

If you don’t have a project in the Airship system or just want to try a test connection, you can use our test server. The test server data is randomly generated and approximates a mobile user’s potential lifecycle. While connecting to the test server is not required, it is a way to see what the event stream looks like.

To connect to the test server, paste this code in your terminal window:

curl -vv https://connect-testing.urbanairship.com/api/events/ \
   --compressed \
   -u "sample_connection:sample_connection" \
   -H "Accept: application/vnd.urbanairship+x-ndjson; version=3;" \
   -d "{}"

After a short time, you should see Airship events appear.

Configure an RTDS direct integration

Connect to the event stream from a terminal window on MacOS or a Linux environment. You will construct a cURL command that connects to the appropriate Airship server, passing in no parameters. The POST call will route your event stream data into your terminal window.

First, create an access token and get your app key in the dashboard:

1. Next to your project name, select the dropdown menu (), then Settings.
2. Under Project settings, select Real-Time Data Streaming.
3. Under Real-Time Data Streaming, select Direct Integration.
4. Enter a user-friendly name and description.
5. Check the box if you’d like to send location events through this connection.
6. Select Create Access Token.
7. Copy the App Key and Access Token values and save them in a secure location.
    Warning

   You will not be able to view the Access Token after leaving this screen.

8. Select Save and exit.

Next, construct a cURL command to access the Data Streaming API, replacing <app-key> and <access-token> with your actual values:

curl -vv https://connect.urbanairship.com/api/events/ \
   --compressed \
   -H "Authorization: Bearer <access-token>" \
   -H "X-UA-Appkey: <app-key>" \
   -H "Accept: application/vnd.urbanairship+x-ndjson; version=3;" \
   -d "{}"

You should now be connected and receiving events. There may be a perceived lag between when events are delivered via the Airship event stream and when they appear in your terminal. This is due to how cURL processes compressed events and does not reflect the actual real-time delivery speed of the events.

Manage direct integrations

After you set up a direct integration you can edit its name and description, create more tokens, and delete tokens. You can also delete the integration.

In the dashboard:

1. Next to your project name, select the dropdown menu (), then Settings.
2. Under Project settings, select Real-Time Data Streaming.
3. Under Enabled Integrations, select the direct integration you want to edit.
4. Make your changes:

   Option	Description	Steps
   Edit the name or description	The name and description appear in the list of enabled integrations.	Select Edit, enter new text, and then select Save.
   Add a token	You can create a maximum of 25 access tokens per integration.	Next to Access Tokens, select Create Token, enter a name to identify it, select Create token. Next, copy the App Key and Access Token values and save them in a secure location, and then select Got it to close the window. You will not be able to view the Access Token after closing the window.
   Remove a token	Any direct connection using the token will no longer function.	Under Access Tokens, select Delete for a token, and then confirm your choice.
   Delete the integration	The integration will be removed from your project and its access token will no longer be valid.	Select Edit, then Delete, and then confirm your choice.

Configure an RTDS webhook integration

To connect to your project’s event stream using a webhook integration, configure it in the dashboard:

1. Next to your project name, select the dropdown menu (), then Settings.
2. Under Project settings, select Real-Time Data Streaming.
3. Under Real-Time Data Streaming, select Webhooks.
4. Configure:

   Section	Description	Steps
   Name and description	The name and description appear in the list of enabled integrations.	Enter text.
   Webhook request URL	This is the endpoint URL you want your events POSTed to. The host must support HTTPS. Query parameters are supported.	Enter your URL.
   Custom Headers	Optional. You can add key/value pairs for the custom header to be sent with the request URL.	Enter a key and value. Select Add another for more.
   Event types	The selected events will be sent to your webhook.	Check the boxes to select event types.

5. Select Activate.

Manage webhook integrations

After you set up a direct integration you can edit its name and description, create more tokens, and delete tokens. You can also delete the integration.

In the dashboard:

1. Next to your project name, select the dropdown menu (), then Settings.
2. Under Project settings, select Real-Time Data Streaming.
3. Under Enabled Integrations, select the webhook you want to edit.
4. Make your changes:

   Option	Description	Steps
   Edit the name or description	The name and description appear in the list of enabled integrations.	Select Edit, enter new text, and then select Save.
   Edit the stream configuration	You can change the endpoint URL, custom headers, or which events to send.	Edit fields or change selections, and then select Update.
   Delete the integration	The integration will be removed from your project.	Select Edit, then Delete, and then confirm your choice.

Example RTDS event

This is an example of a Custom Event as it would appear in your event stream:

POST https://example.com HTTP/1.1
Content-Length: 799
Content-Type: application/json; charset=UTF-8
Custom-Header1: Value1
Custom-Header2: Value2
Authorization: Basic dXNhcj0wYXNzd29yZA==

{
   "id":"7e7f5990-d277-4636-b22c-912cb2ca2ec9",
   "offset":"1245924",
   "occurred":"2016-12-08T21:27:01.125Z",
   "processed":"2016-12-08T21:27:03.000Z",
   "device":{
      "ios_channel":"1819298f-065f-46f1-81a0-1fea91f65ce7",
      "attributes":{
         "locale_variant":"",
         "app_version":"1.2.3",
         "device_model":"x86_64",
         "connection_type":"WIFI",
         "app_package_name":"com.company.app",
         "iana_timezone":"America/Los_Angeles",
         "push_opt_in":"false",
         "locale_country_code":"US",
         "device_os":"10.1",
         "locale_timezone":"-28800",
         "locale_language_code":"en",
         "location_enabled":"true",
         "background_push_enabled":"false",
         "ua_sdk_version":"8.0.1",
         "location_permission":"ALWAYS_ALLOWED"
      }
   },
   "body":{
      "name":"finished_game",
      "session_id":"e025bde1-f974-42fa-a925-aca7054218dc",
      "properties":{
         "game_name":"snaps",
         "game_score":1000
      }
   },
   "type":"CUSTOM"
}

RTDS FAQ

The following are answers to frequently asked questions about Real-Time Data Streaming.

Why do some events occur in the stream multiple times?

While we take measures to prevent them, duplicates can appear in the stream for various reasons. We provide an ID on every event, which you can use to de-duplicate if your use case requires it. For users of the PARTITION subset, note that partitioning is done by ID, ensuring duplicates will always be on the same stream partition.

Why do events appear in the stream with occurred_at from the fairly distant

past?

Devices that are unable to send events to Airship due to a lack of network connectivity will save generated events and dispatch them the next time the app is open and the device is connected to the internet. This can lead to events not reaching the Airship systems until long after they occur. We do not process or pass through any events older than 90 days. If you have stricter latency requirements, use the latency filter.

Why do events appear in my stream with occurred_at in the future, or with

occurred_at after processed_at?

Because device clocks can be unreliable, we sometimes receive events that a device claims occurred in the future. If an event’s time is more than five minutes in the future, we mark it as having occurred at the current server time.

Why do events come in with out of order occurred_at times?

Events may be out of time order for an app and for a single device. This can be due to the on-device latency mentioned above and because events are processed in parallel. Consequently, no guarantees are made about the order in which events appear in the stream.

Why do events come in with out of order processed_at times?

As mentioned above, the systems that process events and prepare them for consumption via the Data Streaming API run in parallel, so we cannot guarantee the events in the resulting stream will be ordered.

Why do some events seem to be missing, like no CLOSE event following an OPEN?

This can happen for multiple reasons:

• Events are batched on the device, but to limit the impact on device storage, they may be dropped if a device cannot communicate with Airship’s servers or has met internal storage limits.

• Events may arrive in different batches, with significant gaps between send times. While the Airship SDK will attempt to flush all buffered events on app close, it may not be able to do so. This could mean that the second event will not be received until the next time the app is opened during network connectivity.

When a push goes to an uninstalled app instance, why do we get both a SEND and an UNINSTALL on iOS but only an UNINSTALL on Android?

Firebase Cloud Messaging (FCM, Google’s notification transport system) sends feedback about uninstalled devices synchronously in their API response. Apple Push Notification Service (APNs) sends feedback later via a separate channel. Because of this differing behavior, we can immediately discover that an Android user has uninstalled and not mark them as being sent to, but we cannot do the same for an iOS user. This is platform-dependent behavior, so it is subject to change.

Why did my stream contain a PUSH_BODY for a push that went to 0 devices?

PUSH_BODY events are generated when a request is made to our push systems, from sending a message using the dashboard or the push API. At that point, we do not yet have the information to determine if anyone in the specified audience will be sent to.

Why does my stream contain a SEND event for a device, but that device never received anything?

We emit a SEND event when the push platform (APNs or FCM) informs us that it has accepted the push request. That push platform might then fail or otherwise be unable to send the push to the specified device, but we will not receive feedback on this unless the reason that the device is unreachable is that the app has been uninstalled.

Why do I get TLS handshake errors when I try to send a request to the Data Streaming API?

In order to protect your data, we support only the newest TLS version (1.2 and above) with a modern cipher:

• TLS_CHACHA20_POLY1305_SHA256
• TLS_AES_256_GCM_SHA384
• TLS_AES_128_GCM_SHA256
• TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
• TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
• TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

Some older HTTP libraries may not support this, but we have confirmed that the following work:

• Java 1.6 or newer, but will require the JCE Unlimited Strength package appropriate for your java version from Oracle
• Python 2.7.9 or newer
• Python 3.4 or newer
• Golang 1.5 or newer
• NodeJS 0.10.33 or newer
• Any HTTP library linked against OpenSSL 1.0.1f or newer (OpenSSL 1.0.1e-fips, which ships with Centos 6, also supports the needed ciphers)

Why do I get an HTTP 402 error code when I try to issue a request to the Data Streaming API?

The number of simultaneous connections for a given connection point is restricted, and any connection attempt that exceeds that limit will receive an HTTP 402 error response. If this continues to occur after retrying for at least 30 seconds, please contact Airship Support .