Remote Audio Data (RAD)

 V3.5 Spec

This document should be considered an “Internet-Draft”, as defined by the IETF, therefore it may be updated, replaced or obsoleted by other documents at any time and should not be cited other than as "work in progress.”

Background

This document outlines the specification as used by  on-demand audio clients for a client implementation of Remote Audio Data (RAD) framework. RAD requires a custom ID3 tag encoded with event metadata to allow clients to report valuable listening data to podcast content producers and sponsors.

Documentation for V1 of the Remote Audio Data (RAD) spec can be found on the NPR Dev Center. V2 of the RAD spec was developed in collaboration with working groups formed out of a RAD Winter Summit in February 2018. That document is still a working draft specification and should be referenced for context and historical background only.

Fundamental Requirements for RAD Implementation

1. Audio file published with custom ID3V2 tag that includes encoded metadata that describes RAD-formatted listening events and tracking URL.

Note: The encoded ID3 tags will not affect audio playback for any non-RAD clients.

2. Client application configured to report listening events to provided tracking URL.
3. Server configured to accept RAD events on tracking URL from RAD-enabled clients; this data will be then available for analysis.
4. Reporting infrastructure for displaying the results.

Please see the Appendix for more detail on the RAD life cycle and the interaction of these parties.

Security Requirements

All RAD data requests and responses should use the JSON data interchange format and are required to be sent over HTTPS. The HTTPS requirement also includes any publisher-provided tracking URLs.

RAD metadata is encoded as a parsable JSON string into custom ID3 tag named “RAD”,^{[a][b][c][d][e]} which may be readable by anyone who chooses to inspect the tag within the audio file. Metadata shall be encoded as value for “remoteAudioData” JSON key. Publishers that desire an additional layer of security may choose to encrypt the values within the JSON payload before encoding their RAD metadata into the ID3 tag. The tracking URLs and events timestamps must be readable by the client framework. (Encryption is, however, out of scope for this document.)

Event Requirements and Format

A series of listening events are saved as objects within the "events":[ ] array of the encoded RAD metadata.

RAD event labels (JSON keys) and their values may be customized to fit the needs of the publisher; examples might include a percentage of completed listening, segment breaks or advertising spots.

• Values assigned to event labels should be string types so that non-integer identifiers can be used, such as compound keys or GUIDs.
• There are no limits in the framework on the number of events created per audio file or event labels (key-value pairs).
• Event names should be published in camelCase format for ease of use by various analytics systems.

Mandatory keys: remoteAudioData, trackingUrls, events, eventTime

Example of metadata with mandatory info (JSON keys) only:

(RAD)

{

  "remoteAudioData": {

    ...

    "trackingUrls": [

      "https://tracking.publisher.org/remote_audio_data",

      ...

    ],

    "events": [

      {

        "eventTime":"hh:mm:ss.sss",

        ...

      },

      ...

    ]

  }

}

Note: ellipsis (...) suggest that additional custom JSON key-value pairs may be added at discretion

Example metadata for RAD custom tag

(RAD)

{

  "remoteAudioData": {

    "podcastId":"510298",

    "episodeId":"497679856",

    "trackingUrls": [

      "https://tracking.publisher1.org/remote_audio_data",

      "https://tracking.publisher2.org/remote_audio_data",

      "https://tracking.publisherN.org/remote_audio_data",

    ],

    "events": [

      {

        "eventTime":"00:00:00.000",

        "label":"podcastDownload",

        "spId":"0",

        "creativeId":"0",

        "adPosition":"0",

        "eventNum":"0"

      },

      {

        "eventTime":"00:00:05.000",

        "label":"podcastStart",

        "spId":"0",

        "creativeId":"0",

        "adPosition":"0",

        "eventNum":"1"

      },

      {

        "eventTime":"00:05:00.000",

        "label":"breakStart",

        "spId":"123456",

        "creativeId":"1234567",

        "adPosition":"1",

        "eventNum":"2"

      },

      {

        "eventTime":"00:05:15.000",

        "label":"breakEnd",

        "spId":"123456",

        "creativeId":"1234567",

        "adPosition":"1",

        "eventNum":"3"

      }

    ]

  }

}

Standardized Error Codes

Errors can occur in the transmission of RAD data from the client or to the tracking URL. Appropriate HTTP status codes should be used in response to client requests, and clients should respond with the agreed upon course of action:

• 2xx: Success
• 3xx: Redirection: Client application needs to re-execute the request to redirected URL.
• 4xx: Client error: Client application needs to drop the request and not re-execute it. Listened RAD event(s) will be lost.
• 5xx: Server error: Client application needs to retry later.

Audio Formats

The RAD spec currently supports implementation for MP3 and MP4 audio formats.

Client Considerations

RAD-enabled clients will be required to:

• Parse the payload of the encoded RAD metadata in the audio file’s ID3 tag.
• Observe audio playback to identify listened RAD events and store them locally.
• It is suggested that clients enable throttling to prevent excessive pings, however each client app has the liberty to adjust that threshold as appropriate.
• Every submission interval check for existing listened RAD events and send them to tracking server accessible by tracking URLs from metadata.
• Batch (limited size defined by batch size) multiple RAD events in a network reporting POST request.
• Offer support for offline mode (locally store events) and report to server(s) once connectivity reestablishes.
• Offer support for specifying custom value for User-Agent request header field

When reporting, listened RAD events shall include the following data:

• Full event data (all events’ key-value pairs) as defined in metadata.
• Device timestamp of when the event was listened in human readable format (ISO 8601) YYYY-MM-DDTHH:MM:SSzzzz (i.e 2018-09-21T16:42:37+03:00)
• Session ID, which is an UUID generated per media file within 24h
• All key/value pairs from metadata on level above events array (e.g. "podcastId":"510298", "episodeId":"497679856")excluding tracking URLs.

Example of reporting network request body:

{

  "audioSessions": [

    {

      "podcastId": "510313",

      "episodeId": "525083696",

      "sessionId": "A489C3AD-04AA-4B5F-8289-4D3D2CFE4CFB",

      "events": [

        {

          "sponsorId": "0",

          "creativeId": "0",

          "eventTime": "00:00:00.000",

          "adPosition": "0",

          "label": "podcastDownload",

          "eventNum": "0",

          "timestamp": "2018-10-24T11:23:07+04:00"

        },

        {

          "sponsorId": "0",

          "creativeId": "0",

          "eventTime": "00:00:05.000",

          "adPosition": "0",

          "label": "podcastStart",

          "eventNum": "1",

          "timestamp": "2018-10-24T11:23:08+04:00"

        },

        {

          "sponsorId": "111128",

          "eventTime": "00:00:05.000",

          "adPosition": "1",

          "label": "breakStart",

          "creativeId": "1111132",

          "eventNum": "2",

          "timestamp": "2018-10-24T11:23:09+04:00"

        },

        {

          "label": "breakEnd",

          "sponsorId": "111128",

          "eventTime": "00:00:05.000",

          "adPosition": "1",

          "creativeId": "1111132",

          "eventNum": "3",

          "timestamp": "2018-10-24T11:23:10+04:00"

          }

      ]

    },

    {

      "podcastId": "510314",

      "episodeId": "525083697",

      "sessionId": "778A4569-4B06-469B-8686-519C3B43C31F",

      "events": [

        {

          "sponsorId": "0",

          "eventTime": "00:00:00.000",

          "adPosition": "0",

          "creativeId": "0",

          "eventNum": "0",

          "timestamp": "2018-10-24T11:23:11+04:00"

        }

      ]

        },

    {

      "podcastId": "510315",

      "episodeId": "525083698",

      "sessionId": "F825BE2B-9759-438A-A67E-9C2D54874B4F",

      "events": [

        {

          "sponsorId": "0",

          "eventTime": "00:00:00.000",

          "adPosition": "0",

          "label": "podcastDownload",

          "creativeId": "0",

          "eventNum": "0",

          "timestamp": "2018-10-24T11:23:12+04:00"

        }

      ]

    }

  ]

}

Configurations

Offline listening

RAD-enabled clients should support offline mode; RAD events should be stored on local persistent storage and be reported to tracking URLs when network connectivity re-establishes.

These stored RAD events should be deleted from local persistent storage; if the age is greater than the configuration parameter (expiration time interval) they should not be reported to the tracking URL.

RAD framework SDK for mobile

Android-specific implementation

Android RAD implementation must locate the remoteAudioData tag from media file metadata and parse the content.

Android RAD framework is designed to be integrated into client applications that are using ExoPlayer version 2.9.1. Clients implementing custom players will need to customize their implementation of RAD library accordingly.

The parsing is done asynchronously while the framework is monitoring the playback of the media player. The extracted remoteAudioData, together with the playback events and session details, are sent to a persistence service (Android IntentService) to be persisted in a local SQLite database. The framework is also comprised of a reporting service that is scheduled at a configurable interval of time to report the persisted events.

iOS-specific implementation

The iOS media player supports the parsing of custom fields in the ID3 tags by default, so no extra effort is needed on the part of the client to locate and parse the metadata encoded into a tag within the file.

iOS RAD framework is designed to be integrated into client applications that are using AVPlayer. Clients implementing custom players will need to customize their implementation of library RAD accordingly.

Data Tracking and Analytics

RAD is not intended to replace download statistics as a point of measurement for the on-demand audio industry; rather, the reported listening events will complement this metric.

Publishers must provide a tracking URLs where clients can report RAD events and choose their own method for storing and analyzing client-reported RAD data. A tracking URL may point to an in-house analytics solution or to a solution provided by a third party.

By grouping listening events by UUID, the data consumer can determine his/her own analytics methods and protocols.

Listened RAD events can be grouped by listening sessions defined on the client. These listening sessions are distinguished by a session ID, which is an UUID generated per media file within 24 hours. There is one required session ID and one optional.

Required mediaSessionID: A GUID generated by the client for each played media file per day per client. A mediaSessionID remains constant for that media file until the next day. For example, if a user listens to five media files in one day, there will be five mediaSessionIDs. If they listen to the same media file on their phone and then on their tablet, there will be two mediaSessionIDs. If they listen to one media file twenty times in one day, there will be one mediaSessionID.

Optional sessionID: A GUID generated by the client per day per client. A sessionID remains constant for a client until the next day. For example, if a user listens to five media files in one day, there will be one sessionID. If they listen to the same media file on their phone and then on their tablet, there will be two sessionIDs. If they listen to one media file twenty times in one day, there will be one sessionID. The definition of “per day” can vary client to client; a rolling 24-hour window is recommended, though not required.

Publishers may download the reported data to an analytics solution, which may involve a text parser that classifies listening events by download in a queryable form. Ideally, the solution would allow for querying listening events by podcast download and for querying specific listening events across all downloads. A complementary database with podcast information would allow for more complicated queries.

Tracking URLs Requirements and Format

At least one valid tracking URL must be included in the encoded ID3 tag so that a RAD-enabled client may report listened RAD events. The tracking URLs will not be visible to the standard audio consumer, however publishers should be aware that they can be seen to anyone who inspects a ID3 tag.

Publishers may wish to include more than one tracking URL in the encoded ID3 tag so that a client can directly share event data.

Tracking URLs must be HTTPS; HTTP is not supported.

While publishers may configure listening events to track advertising playback, it is not currently in scope to include third-party tracking pixels from advertisers. Tracking URLs should be limited to publishers or vendor(s) designated by the publisher.

Recommendations for server setup

NPR’s RAD V1 pilot used existing web server resources, configured with an Apache virtual host entry similar to the following:

<VirtualHost *>

  ServerName tracking.example.org

  CustomLog /var/logs/apache2/tracking.example.org-access.log combined

  ErrorLog /var/logs/apache2/tracking.example.org-error.log

  DefaultType image/gif

  RewriteEngine on

  RewriteRule .* - [R=204,NS,L]

</VirtualHost>

Specific server configurations and CPU requirements will depend upon expected traffic and any preexisting system setup.

Privacy considerations

The RAD spec is not intended as a means to target individual listening behavior, but to provide anonymized, aggregated analytics information. Clients and publishers should adhere to applicable law, such as the EU’s General Data Protection Regulation’s data processing, privacy by design and privacy by default requirements.

Appendix

Workflow

Data Flow