Gnip API v2.1 (https://api-v21.gnip.com) (doc rev 21)

Gnip provides a set of services for accessing, filtering and integrating data from web APIs and feeds. To get started using the Gnip Platform first create an account. You can create one at the Gnip Developer site in seconds. The Gnip Platform API is based on the REST approach (see Wikipedia for a definition of REST if you need one), and we provide convenience libraries for several popular languages (PHP, Java, .NET, Python, Perl and Ruby), so you can be up and running in the language or your choice in no time. The Gnip API version of this document is version 2.1. At Gnip we use the hostname as the way to version the API, this means to access the 2.1 version of the Gnip API, you will need to use the host: https://api-v21.gnip.com. Also, if you connect to the default host URL for the Gnip platform at https://api.gnip.com it will redirect to the most recent version available.

There are two main data integration use cases supported by the Gnip Platform: Publishers & Data Consumers. Publishers represent the sources of data available through the Gnip Platform from different web APIs and feeds across the Internet, and Data Consumers, represent the developers and companies that use the Gnip Platform to access, filter and integrate social and business data into their applications. Developers have the choice of using the Gnip Developer Site or the Gnip Platform APIs to integrate data. In additioin, Gnip provides online support at the Gnip Developer Forums.

The Gnip Platform APIs provides access to normalized event data. An activity contains all event meta-data and the data payload. The data payload represents the full raw data, from a data source (e.g, such Digg, Delicious, Twitter, Sixapart, etc). A notification is the subset of an activity and contains all the event meta-data, just no data payload. By providing these two options developers obtain the flexibility to request the type of data needed for their application and use case. Also, the data payloads included in an activity provide the raw, original, untouched (though encoded) event information from the data source that Gnip maps to the normalized Gnip XML schema. Gnip maps as much of the relevant raw data to the Gnip XML schema so that Data Consumers can satisfy most of their use cases without the need to proces the raw activity data payload. In order to protect the integrity of the Gnip XML, the raw payload is gzip'd and base64 encoded within the xml document and is called the <raw> element. Data Consumers are responsible for decoding and un-gzip'ing the raw element if they need direct access to the original data and this can be easily done with the Gnip Convenience Libraries.

Overview of the Gnip Platform

Important Gnip Platform Concepts

It is important to note that Data Consumers also have two options on how to receive data, either by a subscription, which represents the entire available data from a publisher data stream or by filtering, which represents specific views of a publisher data stream. Data Consumers can create filters for a specific publisher for either notifications or activities.

A Gnip Subscription, provides access to an entire publisher data stream, sometimes called a firehose or public timeline, and represents the entire available data stream of all the activities from a given Publisher. (Note: not all Publishers support public timelines, and public timelines do not always contain full data.) To avoid Publisher name clashes and to show which publishers are 'bless and endorsed by Gnip", Publisher's are namespace scoped to either "gnip," "my," (private only to the Gnip account creator) or "public" (not supported at present).
A Gnip Filter provides access to a data stream containing all the activities or notifications that meet a set of defined criteria, called rules, that are defined by the Data Consumer. For example, Filters allow Data Consumers to create data streams containing just the information from the Publisher based on users they are interested in (using Gnip rule-type "actor"), or all the data from Publisher "foo," that are tagged with a given string (using the Gnip rule-type "tag"). When a Data Consumer creates a Filter they are the only one who is granted access to use the filter. Filters can be "polled" (HTTP GET) from a Gnip generated end-point, or "pushed" (HTTP POST) to a Data Consumer POST URL endpoint that is defined in the Filter.

In addition to these two approaches for defining data data consumption (see a Data Consumer example for more details), companies also can publish data into the Gnip Platform by mapping an existing data source to the Gnip XML schema. Here is an example of how to publish data to the Gnip Platform.

Before you can do anything you will need an account. You can create one at the Gnip developer website in a few seconds.
All requests to Gnip must be made over HTTPS (HTTP over SSL/TLS), eg https://api-v21.gnip.com
All requests to Gnip assume that you're sending user id and password (your gnip account) in HTTP BASIC AUTH'd requests to https://api-v21.gnip.com. This is done in our convenience libraries.
There are a variety of convenience libraries for many popular languages provided by Gnip to ease your integration. These make a lot easier, you can find them at GitHub.
The current list of Gnip scoped publishers can be found at https://api.gnip.com/publishers.html.
Published activities are in GMT/UTC time conversely, all notifications and activities are being consumed in UTC/GMT time.
Requests must be in UTF8 encoding.
When retrieving activities back in time, it is important to have your server's request syncronized with GNIP's server time. (This can be done using the Gnip convenience libraries.)
Requests must be set with content type: application/xml
Requests and responses must be compliant with Gnip's XML schema, it can be found at https://api-v21.gnip.com/schema/gnip.xsd .
Activity buckets can be large in size (2Mb+) and filters can have 100,000s of entries, so it is important to support gzip (compression) for sending and receiving of requests. This is done by default in our convenience libraries. Accept-Encoding: gzip and Content-Encoding: gzip. Accept-Encoding is to the Gnip servers to compress the response whereas content-encoding is sending request gzipped. Sending large filters updates should be gzipped, and polling for large activity buckets should request for this response to be gzipped. Network performance improvements by using Gzip for activities is over 50% (from real world testing), and should be orders of magnitude higher for filter updates. (XML compresses well!)
Gnip Activity & Notification access

To access activities, use the "/activity/" endpoint.
To access notifications, use the "/notification/" endpoint.
Individual filters support only one endpoint type ("activity" or "notification"); not both.
You can not directly access the activity endpoint of a publisher unless you create a filter or have a subscription.

Gnip Platform API

Response Codes

Typical HTTP response codes are used throughout the API.
Requests that do not need to return data or have failed return an XML document describing the result of the request.

Errors

In the event of Gnip being unable to fulfill a request it will return a response with the most appropriate HTTP response code and a human readable message in the body.

An error represented in XML looks like this:

<error>"foo_bar" is not a valid publisher name. Only letter, number, '.', '+' and '-' characters are allowed in publisher names.</error>

Remember, depending on your HTTP client, an <error/> message may only be available as part of an error input stream in the event that the server returns an HTTP response with a non-200 response code. For example, in the case of Java's java.net.HttpURLConnection, the <error/> message would be available via the getErrorStream() method.

If the request could not be fulfilled because there was something wrong with the request a 4xx series response code will be used. If the request was not fulfilled for any other reason a 5xx series response code will be used.

Most common HTTP Response codes used:

200 OK: Happy days!
400 Bad Request: your request is invalid. If this happens check the URL your sending the request to is correct and the data you are sending meets the GNIP Schema format. (Try validating the data via the Gnip schema XSD before sending).
401 Not Authorized: Can mean two things. 1st is most common you have means you have not sent your HTTP BASIC Auth credentials, or you have not entered correctly put your username:password correctly. Try logging into the the GNIP Web UI https://api-v21.gnip.com, if you can log in there then your password is correct. The second is you do not have permission to perform the action, such as deleting a gnip publisher, trying to publish information to a publisher you do not own, etc.
404 Not Found: either you're requesting an invalid URL or the resource in question doesn't exist, check that the URL exists. Use a browser and enter the URL and test this.

A full external reference list of HTTP response codes can be found at the W3C reference site.

Gnip REST URL format

Gnip REST URL formats are based on performing actions on objects. Below, is an example URL used for retrieving current activity based from a specific filter called 'buzzfilter' on the publisher 'digg'.

Gnip REST URL Explanation

HTTPS: Gnip uses SSL on all requests to use are encrypted

API-v21: The host name has the version number within the host name. For version 2.1 of the API the document is https://api-v21.gnip.com. This version is 2.1.

Scope: "Gnip" or "my" is supported. My is mapped to a specific user acocunt and can only be seen by that account. Gnip scoped publishers are maintained by Gnip and can be seen by all users of the Gnip platform.

Digg: one of the publishers supported on Gnip platform.

myfiltername: The spceific filter activity being retrieved.

activity: The type of data being retried. This can be "notification" or "activity".

current.xml: The specific bucket being retried. See bucket details for more information.

As a general rule when doing HTTP requests, setting the HTTP client method allows different actions:

GET is returns a list of publishers, filters or activities (basically read request)
POST is for creating of filters, publishers and publishing activities (write request)
PUT is for updating and not re-creating such as filters and filter rules
DELETE to delete a publisher or a filter

Three major areas on urls:

Publishers
Activities
Filters

Rules on a filter

Rest URLs have two scopes:

Gnip - endorsed by Gnip
my - private to only you, the creator.

Gnip Scoped Rest URLS

These URLS will be used the most for creating filters on. Most data consumers are after data from GNIP endorsed feeds such as digg, delicious, twitter, etc.

Publishers

URL	Method	Details
https://api-v21.gnip.com/gnip/publishers.xml	GET	Returns a list of Gnip endorsed publishers
https://api-v21.gnip.com/gnip/publishers.xml	POST	Creates a new publisher, defined in Publisher XML Schema sent
https://api-v21.gnip.com/gnip/PublisherName	Delete	Only Gnip can delete a Gnip endorsed publisher

Publishing Activity to the publisher created

URL	Method	Details
https://api-v21.gnip.com/gnip/PUBLISHER-NAME/activity.xml	GET	Returns publisher's activity based for PUBLISHER-NAME, for most publishers will return HTTP error 401, unauthorized. If your unauthorized use a filter.
https://api-v21.gnip.com/gnip/PUBLISHER-NAME/notification.xml	GET	Returns publisher's activity based for PUBLISHER-NAME, for some publishers will return HTTP error 401, unauthorized. If your unauthorized use a filter.
https://api-v21.gnip.com/gnip/PUBLISHER-NAME/activity.xml	POST	Only Gnip can post activity for a Gnip endorsed publisher
https://api-v21.gnip.com/gnip/publishers.xml	Delete	Only Gnip can delete a Gnip endorsed publisher

Activity retrieval

URL	Method	Details
https://api-v21.gnip.com/gnip/publishers/PUBLISHER- NAME/notification/current.xml	GET	Returns publisher's current notification activity for Gnip endorsed PUBLISHER-NAME, some publishers this will return a HTTP error 401, unauthorized
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/notification/BUCKET-ID.xml	GET	Returns publisher's notification activity from BucketID for PUBLISHER-NAME, for some publishers this will return a HTTP error 401, unauthorized.
https://api-v21.gnip.com/gnip/publishers/PUBLISHER- NAME/activity/current.xml	GET	Returns publisher's current notification activity for Gnip endorsed PUBLISHER-NAME, most publishers this will return a HTTP error 401, unauthorized
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/activity/BUCKET-ID.xml	GET	Returns publisher's full-data activity from BucketID for PUBLISHER-NAME, for nearly all publishers this will return a HTTP error 401, unauthorized.

Activities based on a filter

URL	Method	Details
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME/notification/current.xml	GET	Returns publisher's current notification activity for Gnip PUBLISHER-NAME
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME/notification/BUCKET-ID.xml	GET	Returns publisher's notification activity from BucketID for Gnip PUBLISHER-NAME
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME/activity/current.xml	GET	Returns publisher's full-data activity for Gnip PUBLISHER-NAME
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME/activity/BUCKET-ID.xml	GET	Returns publisher's full-data activity from BucketID for PUBLISHER-NAME

Filters

URL	Method	Details
https://api-v211.gnip.com/gnip/publishers/PUBLISHER-NAME/filters.xml	GET	Lists all the filters and the rules owned by you on a Gnip endorsed feed.
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters.xml	POST	Accepts a Filter XML document and creates a new filter based on that information
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME.xml	GET	Returns the FILTER-NAME and the XML definition of the filter
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME.xml	POST	Will create/over-ride a filter with Filter-NAME with a Filter XML instance posted
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME.xml	PUT	Accepts a Filter XML document and updates the specified Filter to match the data provided
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME.xml	DELETE	Deletes a based on FILTER-NAME on a publisher PUBLISHER-NAME on a Gnip endorsed data feed.
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME/activity/BUCKET-ID.xml	GET	Returns publisher's notification activity from BucketID on filter FILTER-NAME that is on publisher PUBLISHER-NAME
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME/notification/BUCKET-ID.xml	GET	Returns publisher's full data activity from BucketID on filter FILTER-NAME that is on publisher PUBLISHER-NAME

Filter rules

URL	Method	Details
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME/rules	GET	Accepts a rule description to search for in the Filter using a query string of the form ?type=actor&value=joe
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME/rules.xml	POST	Accepts either a <rule/> or <rules/> document and adds the rules to the filter named FILTER-NAME. Use the <rules/> document to batch add rules to a Filter.
https://api-v21.gnip.com/gnip/publishers/PUBLISHER-NAME/filters/FILTER-NAME/rules	DELETE	Accepts a <rules/> document that contains a batch of <rule/> objects to delete from the user's filter named FILTER-NAME. This should be used to batch delete rules from a Filter. Also accepts a query string to specify a single rule to delete in the form ?type=actor&value=joe.

My Scoped Rest URLS

These URLS will be used the most used to retrieve your filters and edit your filters / filter rules.

Publishers

URL	Method	Details
https://api-v21.gnip.com/my/publishers.xml	GET	Returns a list of your publishers
https://api-v21.gnip.com/my/publishers.xml	POST	Creates a new publisher, defined in Publisher XML Schema sent
https://api-v21.gnip.com/my/PublisherName	DELETE	Deletes a publisher with PublisherName

Publishing Activity to the publisher created

URL	Method	Details
https://api-v21.gnip.com/my/PUBLISHER-NAME/activity.xml	GET	Returns publisher's activity based for PUBLISHER-NAME
https://api-v21.gnip.com/my/PUBLISHER-NAME/notification.xml	GET	Returns notification methods based for PUBLISHER-NAME
https://api-v21.gnip.com/my/PUBLISHER-NAME/activity.xml	POST	Publishes an activity based on the activity schema. You only need to publish to activity.xml, Gnip will create another type of activity called notifications from this. (That is there will be full-data activity and notifications (activity) without payload
https://api-v21.gnip.com/my/publishers.xml	DELETE	Deletes a publisher, defined in the Publisher XML Schema

Activity retrieval

URL	Method	Details
https://api-v21.gnip.com/my/publishers/PUBLISHER- NAME/notification/current.xml	GET	Returns publisher's current notification activity for PUBLISHER-NAME
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/notification/BUCKET-ID.xml	GET	Returns publisher's notification activity from BucketID for PUBLISHER-NAME
https://api-v21.gnip.com/my/publishers/PUBLISHER- NAME/activity/current.xml	GET	Returns publisher's current full data activity for PUBLISHER-NAME
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/activity/BUCKET-ID.xml	GET	Returns publisher's full data activity from BucketID for PUBLISHER-NAME

Activities based on a filter

URL	Method	Details
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME/notification/current.xml	GET	Returns publisher's current notification activity for PUBLISHER-NAME
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME/notification/BUCKET-ID.xml	GET	Returns publisher's notification activity from BucketID for PUBLISHER-NAME
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME/activity/current.xml	GET	Returns publisher's full-data activity for PUBLISHER-NAME
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME/activity/BUCKET-ID.xml	GET	Returns publisher's full-data activity from BucketID for PUBLISHER-NAME

Filters

URL	Method	Details
https://api-v211.gnip.com/my/publishers/PUBLISHER-NAME/filters.xml	GET	Lists all the filters and the rules owned by you.
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters.xml	POST	Accepts a Filter XML document and creates a new filter based on that information
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME.xml	GET	Returns the FILTER-NAME and the XML definition of the filter
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME.xml	POST	Will create/over-ride a filter with Filter-NAME with a Filter XML instance posted
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME.xml	PUT	Accepts a Filter XML document and updates the specified Filter to match the data provided
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME.xml	DELETE	Deletes a filter based on FILTER-NAME on a publisher PUBLISHER-NAME you own.
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME/activity/BUCKET-ID.xml	GET	Returns publisher's notification activity from BucketID on filter FILTER-NAME that is on publisher PUBLISHER-NAME
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME/notification/BUCKET-ID.xml	GET	Returns publisher's full data activity from BucketID on filter FILTER-NAME that is on publisher PUBLISHER-NAME

Filter rules

URL	Method	Details
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME/rules	GET	Accepts a rule description to search for in the Filter using a query string of the form ?type=actor&value=joe
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME/rules.xml	POST	Accepts either a <rule/> or <rules/> document and adds the rules to the filter named FILTER-NAME. Use the <rules/> document to batch add rules to a Filter.
https://api-v21.gnip.com/my/publishers/PUBLISHER-NAME/filters/FILTER-NAME/rules	DELETE	Accepts a <rules/> document that contains a batch of <rule/> objects to delete from the user's filter named FILTER-NAME. This should be used to batch delete rules from a Filter. Also accepts a query string to specify a single rule to delete in the form ?type=actor&value=joe.

Buckets

Activity is put into buckets. It is important to understand the concepts behind activity buckets.

Time and buckets

Time

All time used in the Gnip system is in the UTC/GMT time zone. When doing polling for previous activity buckets , time between your Application server and Gnip's servers need to be the same. (This is handled in our convenience libraries.)

GNIP Activity Schema

The full GNIP schema can be at the Developer website, https://api.gnip.com/schema/gnip.xsd.

The most common schema you will be using is the Activity Schema and filters. Below is an example of an activities schema.

<?xml version="1.0" encoding="utf-8"?>
<activities publisher="gnip-sample">
    <activity>
        <at>2009-02-02T17:16:01.000Z</at>
        <action>post</action>
        <activityID>0.516958128858</activityID>
        <URL>http://gnipherald.com/stories/0.516958128858/jousting-gone-bad.xml</URL>
        <source>Vignette-Web-Publisher</source>
        <place>

           <elev>5,368 meters</point>
           <floor>2nd</floor>            <point>38.2638886 -106.126131</point>
            <featurename>Chloride Mine</featurename>

         <featuretypetag>Colorado</featuretypetag >

           <relationshiptag>middle</relationshiptag>

        </place>
        <actor uid="12339948@4994" metaURL="http://gnipherald.com/profiles/jvaleski">jvaleski</actor>
        <destinationURL>http://gnipherald.com/stories/0.516958128858/jousting-gone-bad.html</destinationURL>
        <tag metaURL="http://gnipherald/about/tag/armor">armor</tag>
        <tag metaURL="http://gnipherald/about/tag/staff">staff</tag>
        <tag metaURL="http://gnipherald/about/tag/injury">injury</tag>
        <to metaURL="http://gnipherald.com/profiles/barinek">barinek</to>
        <regardingURL metaURL="http://apnewswire.com/jousting">http://apnewswire.com/jousting/accident.html</regardingURL>
        <payload>
            <title>Jousting Event Went Awry</title>
            <body> Two drunken guys, two jousting sticks, blindfolds and running full bore down a hallway at each other, lead to a night in hospital for two university students. One with chest injury the other unharmed. REST of BODY </body>
            <mediaURL mimeType="jpg" height="600" width="400" >http://images.gnipherald.com/jousting-thumbnail.jpg</mediaURL>

            <raw>base64 encoded gzip compressed binary</raw>
        </payload>
    </activity>
</activities>

Each activity must have an <at> and an <action>, <at> must be in UTC time.  Every other element / attribute is optional. Note: the more information you provide in the schema the more rich the information, the greater syndication of your data. Our convenience libraries for most popular languages will put the time in UTC format, gzip the request and the response and base64 with gzip the <raw> part of the message. The convenience libraries can be  find them here.

Gnip Schema Explanation


Schema Explanation Details:

<at>is the time the activity was done
<action> what activity was performed
<activityID> is any value a publisher wishes to use to reference this information - be it a GUID, url, string, etc up to the discretion of the publisher 
<URL> contains the machine-readable metadata for the activity
<source> is the mechanism by which the event was published for example, web, mobile, txt, partner's name, etc

<actor> is the person who performed the activity
        <posts> is the number of updates made by the user
        <followers> is the number of people following the user
        <friends> is the number of people the user has friended
        <fullname> is fullname or screen name for the user
        <username> is username of the user on the service
        <metaURL> is the user profile link on the service
        <uid> is the user numeric ID on the service

<destinationURL> is the link to the actual activity, for example links to the actual post of a comment, tweet, video upload, etc If you use more than one destinationURL make sure the first destinationURL points to the human readable story of the action.
    <metaURL> is the link to the page about the result of the activity

<to> is the person the action is in reply to
    <metaURL> is the link to the user profile of the user on the service
         <uid> is the user numeric ID on the service

<regardingURL> is the link to what was replied to 
    <metaURL> is the link to the thread's root 

<tag> a tag
    <metaURL> is a link to the publisher's page on this tag 

<payload> 
This is where the title, body, and the raw base64 gzip message (can be in a different format than gnip's) with one ore more mediaURLs

    <title> the title of the activity
    <body> the body of the activity, or description if there's no body
    <raw> is the base64 gziped message, the unmodified publisher data/metadata for an activity.  Raw can be empty

    <mediaURL> (can have more than one mediaURL) 
        <mimeType> is the mime type of the media, eg, image/jpeg
        <type>  is the type of media, types today are: avatar, video, flash
        <height> is the height of the enclosure
        <width> is the width of the enclosure
        <duration> is the duration of the enclosure, it can be any string format today such as 2m and 3s, we may put in some formats in the future


<place> is based on the GeoRSS standard. 
     <point> is the location containing a single latitude-longitude pair, separated by whitespace
    <elev> is the meant to contain the "common" GPS elevation readings, i.e. height in meters 
    <floor> is meant to contain the floor number of a building 
     <featurename> is meant to refer to the location or feature
    <relationshipTag> is a way of relating Web content to the Earth features, see GeoRSS for more details. 
    <featureTypeTag> is meant to represent a real feature of the Earth's surface 


XSD of the activity Schema
Full Gnip XML schema can be found here.

...
    <xs:element name="activity">
        <xs:complexType>
            <xs:sequence>
                <xs:element name="at" type="xs:dateTime" minOccurs="1" maxOccurs="1" />
                <xs:element name="action" type="xs:string" minOccurs="1" maxOccurs="1" />
                <xs:element name="activityID" type="xs:string" minOccurs="0" maxOccurs="1" />
                <xs:element name="URL" type="xs:anyURI" minOccurs="0" maxOccurs="1" />
                <xs:element name="source" type="xs:string" minOccurs="0" maxOccurs="unbounded" />
                <xs:element name="keyword" type="xs:string" minOccurs="0" maxOccurs="unbounded" />
                <xs:element name="place" minOccurs="0" maxOccurs="unbounded">
                    <xs:complexType>
                        <xs:sequence>
                            <xs:element name="point" type="pointType" minOccurs="0" maxOccurs="1" />
                            <xs:element name="elev" type="xs:double" minOccurs="0" maxOccurs="1" />
                            <xs:element name="floor" type="xs:integer" minOccurs="0" maxOccurs="1" />
                            <xs:element name="featuretypetag" type="xs:string" minOccurs="0" maxOccurs="1" />
                            <xs:element name="featurename" type="xs:string" minOccurs="0" maxOccurs="1" />
                            <xs:element name="relationshiptag" type="xs:string" minOccurs="0" maxOccurs="1" />
                        </xs:sequence>
                    </xs:complexType>
                </xs:element>
                <xs:element name="actor" minOccurs="0" maxOccurs="unbounded">
                    <xs:complexType>
                        <xs:simpleContent>
                            <xs:extension base="xs:string">
                                <xs:attribute name="metaURL" use="optional" type="xs:anyURI" />
                                <xs:attribute name="uid" use="optional" type="xs:string" />
                            </xs:extension>
                        </xs:simpleContent>
                    </xs:complexType>
                </xs:element>
                <xs:element name="destinationURL" minOccurs="0" maxOccurs="unbounded">
                    <xs:complexType>
                        <xs:simpleContent>
                            <xs:extension base="xs:anyURI">
                                <xs:attribute name="metaURL" use="optional" type="xs:anyURI" />
                            </xs:extension>
                        </xs:simpleContent>
                    </xs:complexType>
                </xs:element>
                <xs:element name="tag" minOccurs="0" maxOccurs="unbounded">
                    <xs:complexType>
                        <xs:simpleContent>
                            <xs:extension base="xs:string">
                                <xs:attribute name="metaURL" use="optional" type="xs:anyURI" />
                            </xs:extension>
                        </xs:simpleContent>
                    </xs:complexType>
                </xs:element>
                <xs:element name="to" minOccurs="0" maxOccurs="unbounded">
                    <xs:complexType>
                        <xs:simpleContent>
                            <xs:extension base="xs:string">
                                <xs:attribute name="metaURL" use="optional" type="xs:anyURI" />
                            </xs:extension>
                        </xs:simpleContent>
                    </xs:complexType>
                </xs:element>
                <xs:element name="regardingURL" minOccurs="0" maxOccurs="unbounded">
                    <xs:complexType>
                        <xs:simpleContent>
                            <xs:extension base="xs:anyURI">
                                <xs:attribute name="metaURL" use="optional" type="xs:anyURI" />
                            </xs:extension>
                        </xs:simpleContent>
                    </xs:complexType>
                </xs:element>
                <xs:element name="payload" minOccurs="0" maxOccurs="1">
                    <xs:complexType>
                        <xs:sequence>
                            <xs:element name="title" type="xs:string" minOccurs="0" maxOccurs="1" />
                            <xs:element name="body" type="xs:normalizedString" minOccurs="0" maxOccurs="1" />
                            <xs:element name="mediaURL" minOccurs="0" maxOccurs="unbounded">
                                <xs:complexType>
                                    <xs:simpleContent>
                                        <xs:extension base="xs:anyURI">
                                            <xs:attribute name="height" use="optional" type="xs:string" />
                                            <xs:attribute name="width" use="optional" type="xs:string" />
                                            <xs:attribute name="duration" use="optional" type="xs:string" />
                                            <xs:attribute name="mimeType" use="optional" type="xs:string" />
                                            <xs:attribute name="type" use="optional" type="xs:string" />
                                        </xs:extension>
                                    </xs:simpleContent>
                                </xs:complexType>
                            </xs:element>
                            <xs:element name="raw" type="xs:base64Binary" minOccurs="1" maxOccurs="1" />
                        </xs:sequence>
                    </xs:complexType>
                </xs:element>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
...
</xs:schema>

Activity Streams

Activity streams are collections of activities which may be retrieved and, in some situations, to which activities may be published. Publishers POST new activities to activity streams; they are the start of data flow through the system. Subscribers can retrieve activity data via activity buckets (see the Activity Buckets section below). There is no synchronizing of time between the Gnip server and your local server is needed on this request as it lists the buckets to traverse to retrieve information.

https://api-v21.gnip.com/my[gnip]/publishers/PUBLISHER-NAME/activity[notification].xml

(Replace PUBLISHER-NAME with the name of an actual publisher. The list of current Publishers can be found here)

GET:

Returns an XML representation (Activity Stream XML) of the activity Publisher activity stream.
Response Content-Type: application/xml

Activity Stream XML

An activity stream represented in XML looks like the following. Full Gnip XML schema can be found here.

<?xml version="1.0" encoding="utf-8"?>
<activityStream>
    <activitiesAddedAt>2008-07-28T11:32:50Z</activitiesAddedAt>
    <buckets>
        <bucket href="/my[gnip]/publishers/PUBLISHER-NAME/activity/200807281130.xml" />
        <bucket href="/my[gnip]/publishers/PUBLISHER-NAME/activity/200807281125.xml" />
        <bucket href="/my[gnip]publishers/PUBLISHER-NAME/activity/200807281130.xml" />
        
    </buckets>
</activityStream>

Activity Buckets

Activity streams are segmented into buckets based on time. Each bucket contains all the activities for the stream that are reported during the bucket’s duration. Each bucket has an ID that is the time stamp, in UTC, of the first minute bucket. The bucket ID format is %Y%m%d%H%M%S (e.g. the strftime conversion spec). For example, bucket with ID 200806090913 would contain all activity reported during between 9:13 am and 9:14 UTC am, inclusive, on June 9, 2008. Activity buckets are only valid for the last 60 minutes. The "current.xml" bucket holds the "current" activities up to one minute. Please note that the current bucket is not static, so retrieving data from this bucket may not be reliable for your application. The same goes for the 0-1 minute old bucket. Until the full minute is up, the bucket will continue to accrue data.

https://api-v21.gnip.com/my[gnip]/publishers/PUBLISHER-NAME/notification/current.xml
https://api-v21.gnip.com/my[gnip]/publishers/PUBLISHER-NAME/notification/BUCKET-ID.xml

https://api-v21.gnip.com/my[gnip]/publishers/PUBLISHER-NAME/filters/FILTER-NAME/activity[notification]/current.xml

https://api-v21.gnip.com/my[gnip]/publishers/PUBLISHER-NAME/filters/FILTER-NAME/activity[notification]/BUCKET-ID.xml

GET:

Returns a representation (Activities XML) of the activity bucket specified.
Response Content-Type: application/xml

Request Accept-Encoding: gzip   (optional but recommended, our convenience libraries support this)
Request Content-Encoding: gzip  (optional but recommended, our convenience libraries support this)

Activities XML

Activity definitions are represented as XML documents similar to this.

<activities publisher="gnip-sample">
    <activity>
        <at>2009-02-02T17:16:01.000Z</at>
        <action>post</action>
        <activityID>0.516958128858</activityID>
        <URL>http://gnipherald.com/stories/0.516958128858/jousting-gone-bad.xml</URL>
        <source>Vignette-Web-Publisher</source>
        <place>
            <point>38.2638886 -106.126131</point>
            <featurename>Chloride Mine</featurename>
        </place>
        <actor uid="12339948@4994" metaURL="http://gnipherald.com/profiles/jvaleski">jvaleski</actor>
        <destinationURL>http://gnipherald.com/stories/0.516958128858/jousting-gone-bad.html</destinationURL>
        <tag metaURL="http://gnipherald/about/tag/armor">armor</tag>
        <tag metaURL="http://gnipherald/about/tag/staff">staff</tag>
        <tag metaURL="http://gnipherald/about/tag/injury">injury</tag>
        <to metaURL="http://gnipherald.com/profiles/barinek">barinek</to>
        <regardingURL metaURL="http://apnewswire.com/jousting">http://apnewswire.com/jousting/accident.html</regardingURL>
        <payload>
            <title>Jousting Event Went Awry</title>
            <body> Two drunken guys, two jousting sticks, blindfolds and running full bore down a hallway at each other, lead to a night in hospital for two university students. One with chest injury the other unharmed. REST of BODY </body>
            <mediaURL>http://images.gnipherald.com/jousting-thumbnail.jpg</mediaURL>
            <raw>base64 encoded gzip compressed binary</raw>
        </payload>
    </activity>
</activities>

Descriptions of element and attribute schema details can be found here .

Activity XML document conform to the following schema.

Filters

A Filter consists of rules that are used to match activities flowing through a Publisher's activity stream. Filters may be created, edited, and deleted at will by their owner. Filter rules are OR'd together; there is no AND support. Since your filter is created on a per publisher basis, the URI will reflect the namespace of that publisher ("my" or "gnip"). Filters can be created and edited through the Developer website, a walkthru is found in the "Configuring filters through the Gnip Developer Site".

The following rule-types are supported:

Actor
Regarding
Source
Tag
To
Keyword

Not all publishers support the all of the rule-types. Specific rule-type support for each publisher can be viewed in the Gnip platform web application at http://api.api.com/publishers.html.

There are times when entering rule-type values into a filter characters, such as, commas or quotes around the actual value. Entering these special cases require specific formatting based on being entered via the Gnip API or using the Gnip Platform web application. The table below the highlights these differences.

Entering values	Gnip REST API	Gnip Web Application
Political News	<rule type="actor">Political News</rule> Note: no quotes needed	Political News Quote not needed needed as comma separated values
Boulder, CO	<rule type="actor">Boulder, CO</rule> Note: no quotes needed	"Boulder, CO" Note:Quotes are needed as our values are comma separated
"Search Term" needs to have quotes - literal	<rule type="actor">"Search Term"</rule> Note: quotes needed	"""Search Term""" To have the search term surrounded by quotes there need to be three quotes three quotes as the second quote escapes the third quote
World & Business	<rule type="actor">World & Business</rule> Note: needs to be URL encoded	World & Business We URL encode this correctly

For any rule value, be it actor, keyword or tag, we will URL encode all URLs to the respective publishers.

URL encoding

GNIP will URL encode all string values in keywords, actors, source only using the Gnip Web UI. Note: When using the XML API it needs to be well formed XML, so properly encode the html entities.

If a Filter specifies a postURL, new activities that meet the collection criteria will be posted to the specified URL. POSTed activities include full data by default.

https://api-v21.gnip.com/my[gnip]/publishers/PUBLISHER-NAME/filters.xml

GET:
Returns the Filters XML document which lists all the filters, including their rules, you own for the specified Publisher.
Response Content-Type: application/xml

POST:

Accepts a Filter XML document and creates a new filter based on that information. If your Filter document is large it should be gzip'd, and sent with the "Content-Encoding: gzip" header. If you wish to update a filter, use PUT, described below. POSTing a Filter twice with the same name will result in an error response from the server.
Request Content-Type: application/xml

Request Accept-Encoding: gzip   (optional but recommended if a lot of filters, our convenience libraries support this)
Request Content-Encoding: gzip  (optional but recommended, our convenience libraries support this)

Response Content-Type: application/xml

Response Content-Encoding: gzip (if gzip is set on request)

https://api-v21.gnip.com/my[gnip]/publishers/PUBLISHER-NAME/filters/FILTER-NAME.xml

GET:

Returns a Filter XML representation of the specified Filter.
Response Content-Type: application/xml

PUT:

Accepts a Filter XML document and updates the specified Filter to match the data provided. Gnip will replace the existing Filter with this document. If you're trying to create a Filter for the first time, use POST, described above. For incremental updates of a Filter, see the next section "Editing Filter Rules."
Request Content-Type: application/xml

Request Accept-Encoding: gzip   (optional but recommended, our convenience libraries support this)
Request Content-Encoding: gzip  (optional but recommended, our convenience libraries support this)

Response Content-Type: application/xml

Response Content-Encoding: gzip (if gzip is set on request)

DELETE:

Removes the specified Filter.

Filter XML

Filter definitions are represented as XML documents similar to this.

<filter name="example" fullData="true|false">
    <postURL>optional</postURL>
    <rule type="actor">joe</rule>
    <rule type="tag">political news</rule>

<rule type="tag">world news</rule>

</filter>

Filter XML document conform to the following schema. Full Gnip XML schema can be found here.

    <xs:element name="filter">
        <xs:complexType>
            <xs:sequence>
                <xs:element name="postURL" type="xs:anyURI" minOccurs="0" maxOccurs="1" />
                <xs:element name="rule" type="rule" maxOccurs="unbounded" minOccurs="1" />
            </xs:sequence>
            <xs:attribute name="name" type="uriSafeType" use="required" />
            <xs:attribute name="fullData" type="xs:boolean" use="required" />
        </xs:complexType>
    </xs:element>

Editing Filter Rules

A Filter's rule set can be incrementally updated and queried via the Filter's /rules endpoint. The rules endpoint can be used to add a single rule or to batch add a set of rules. The rules endpoint is namespaced based on the Publisher scope (either "my" or "gnip").

Each publisher supports a different set of one or more rules. To review the list of supported rule types, you can view the schema document. Currently, the types are:

actor

tag

regarding

source

To find out if a publisher supports the rule types, you can look at the list of Publishers, the action types they support, and example XML output.

https://api-v21.gnip.com/my[gnip]/publishers/PUBLISHER-NAME/filters/FILTER-NAME/rules

GET:
        Accepts a query string consisting of a single rule to search for in the Filter named FILTER-NAME.
        For example: .../my[gnip]/publishers/PUBLISHER-NAME/filters/FILTER-NAME/rules?type=actor&value=joe
        Request Content-Type: application/xml

        Request Accept-Encoding: gzip   (optional but recommended, our convenience libraries support this)
        Request Content-Encoding: gzip  (optional but recommended, our convenience libraries support this)

Response Content-Type: application/xml

Response Content-Encoding: gzip

        Response Codes:
            200: if the rule is found in the Filter. The response body will contain the XML of found <rule/>.
            404: if the rule could not be found.
POST:
       Accepts two XML document types:
           Send a rule XML document (e.g. <rule type="">value</rule> ) to add a single rule to FILTER-NAME.

Send a rules XML document (e.g. <rules>...</rules>) that wraps one or more <rule/> documents to batch add to FILTER-NAME.

For example: .../my[gnip]/publishers/PUBLISHER-NAME/filters/FILTER-NAME/rules.xml
Request Content-Type: application/xml

        Request Content-Encoding: gzip  (optional but recommended, our convenience libraries support this)

Response Content-Type: application/xml

Response Codes:

            200: if an individual rule is successfully added to the Filter.
                  if sending a <rules/>  document, a 200 will be returned if the document is successfully received, but the server processes a batch update asynchronously, so for <rules/>  a 200 does not imply that they have been added to the Filter. Use either the rules/ endpoint to search for a newly added rule or a GET on the whole Filter to ensure all rules in the batch were added.
            400: if adding a <rule/>  with a rule type that isn't supported by the publisher

DELETE:

Accepts requests of two forms:
Send a <rules/> XML document that wraps one or more <rule/> documents to batch delete from the FILTER-NAME.
A query string consisting of a single rule to be removed from the FILTER-NAME in the form ?type=actor&value=joe. It is strongly preferred to use the <rules/> document to delete rules.
For example: .../my[gnip]/publishers/PUBLISHER-NAME/filters/FILTER-NAME/rules.xml
Request Content-Type: application/xml

Response Content-Type: application/xml

Response Codes:
    200: If a single or batch of rules are successfully added to the Filter.
            If sending a <rules/> document, an HTTP 200 will be returned if the document is successfully received and well structured. The server then processes a batch update asynchronously. To ensure that a batch delete is successful end-to-end, request the Filter XML document and ensure that the rules are present.
    400: If an error occurs parsing the document, the Filter does not exist, or some other error is found in the document sent by the client.

Large Filters and the /rules endpoint

Over time, Filters on the Gnip Platform can grow "large" and consist of tens or hundreds of thousands of rules to match specific activities flowing through a single Publisher. It is strongly recommended that clients managing large rule sets in a Filter use the /rules endpoint to add and remove up to 5000 rules in a batch operation. Due to restrictions on the size of a POST sent to Gnip, it isn't possible to add this many rules to a Filter via the /filters endpoint. Instead, the preferred way to add 100,000 rules to a Filter is to send 20 HTTP requests to the /rules controller with a <rules/> document containing 5000 <rule/> elements.

The DELETE verb can also be used to remove up to 5000 rules from a Filter by specifying <rule/> elements in a <rules/> document.

Pushed Activities - POSTed to your endpoint

Filters can be subscribed to in such a way that any time activities are published that match the Filter criteria, those activities are pushed, via a HTTP POST, to a configurable URL. The URL to which activities will be pushed is specified by the postURL of the Filter XML. Any activities added to a Filter which has a <postURL/>, will automatically be POSTed to that URL. The body of these POSTs will be an Activities XML document (described above in the Activity Buckets section). Responses to these requests are ignored. PostURLs must be valid. Gnip does not HTTP Post gzip encoded today, we may change this in the future.

http://mysite.example/inbound-activity-handler.cgi

POST:

Pushes new activities, as an Activities XML document, to the subcriber to this collection.

Request Content-Type: application/xml

Activities XML

A set of activities are represented in XML documents that look like this. Full Gnip XML schema can be found here.

<?xml version="1.0" encoding="utf-8"?>10
0<activities publisher="gnip-sample">
    <activity>
        <at>2009-02-02T17:16:01.000Z</at>
        <action>post</action>
        <activityID>0.516958128858</activityID>
        <URL>http://gnipherald.com/stories/0.516958128858/jousting-gone-bad.xml</URL>
        <source>Vignette-Web-Publisher</source>
        <place>
            <point>38.2638886 -106.126131</point>
            <featurename>Chloride Mine</featurename>
        </place>
        <actor uid="12339948@4994" metaURL="http://gnipherald.com/profiles/jvaleski">jvaleski</actor>
        <destinationURL>http://gnipherald.com/stories/0.516958128858/jousting-gone-bad.html</destinationURL>
        <tag metaURL="http://gnipherald/about/tag/armor">armor</tag>
        <tag metaURL="http://gnipherald/about/tag/staff">staff</tag>
        <tag metaURL="http://gnipherald/about/tag/injury">injury</tag>
        <to metaURL="http://gnipherald.com/profiles/barinek">barinek</to>
        <regardingURL metaURL="http://apnewswire.com/jousting">http://apnewswire.com/jousting/accident.html</regardingURL>
        <payload>
            <title>Jousting Event Went Awry</title>
            <body> Two drunken guys, two jousting sticks, blindfolds and running full bore down a hallway at each other, lead to a night in hospital for two university students. One with chest injury the other unharmed. REST of BODY </body>
            <mediaURL>http://images.gnipherald.com/jousting-thumbnail.jpg</mediaURL>
            <raw>base64 encoded gzip compressed binary</raw>
        </payload>
    </activity>
</activities>

Publishers

Publishers are scoped to either be "system-wide" and publicly available (e.g. in the "gnip" namespace), or privately scoped in the "my" namespace. A current list of public Publishers in the Gnip system can be found here. They are owned by a single account and only that account can manipulate the publisher definition and publish activities to its activity stream. A Publisher can move their private "my" Publisher feed to a GNIP endorsed by contacting GNIP.

As a publisher you push your information to us in Gnip's standard format. See section Gnip Standard format for more details.

To publish an activity:

In three steps:
Step One: 
    you need to create a publisher through the API or create a publisher through the Gnip Web UI. 

 (click on "create one!")



Step Two
Define which tags you support so that data consumers can set up filters on your data.

Step Three
Using the API http POST an activity in Gnip schema format  to the below url: 

Note: A publisher only needs to publish activity.xml, Gnip will create another type of activity called notifications from this.

POST:
https://api-v21.gnip.com/my/PUBLISHER-NAME/activity.xml

Request Content-type: application/xml
Request Accept-Encoding: gzip   (optional but highly recommended for activity buckets over 100k, our convenience libraries support this)
Request Content-Encoding: gzip  (optional but highly recommended for activity buckets over 100k, our convenience libraries support this)

Response Content-Type: application/xml
Response Content-Encoding: gzip  (if gzip is set on request)
https://api-v21.gnip.com/my[gnip]/publishers.xml

Setting scope to 'my' - returns your publisher

Setting scope to 'gnip' - returns the list of endorsed Gnip publishers

Different actions can be done by doing a get, post or a delete to the URL above.

Retrieve a list of publishers:
GET:
Returns a Publisher XML document listing of current Publisher's in the system. Use "gnip" in the URL to return the list of endorsed publicly available publishers and "my" to return your list of publishers.
Response Content-Type: application/xml

To create a new publisher:
POST:

Accepts a Publisher XML document and creates a new publisher using that information. Post can only be done to the ".../my" scope. Only GNIP can create GNIP publishers.
Request Content-type: application/xml
Response Content-Type: application/xml

To delete a publisher you own:

DELETE:
       Accepts a query string consisting of a single pubisher to be removed using my for scope. Only GNIP can delete gnip publishers.
       For example: .../my/publishers/PUBLISHER-NAME
       Request Content-Type: application/xml
       Response Content-Type: *

Response Codes:

200: if an the activity was successfuly published, created or deleted
400: if adding an XML element or request is not understood.

Publisher XML

Publisher definitions are represented as XML documents similar to this. As a Publisher you can define which rules you support. Rule types today are: actor, tag, to, regarding and source. Below is an example of XML that has to, actor and regarding supported.

<?xml version="1.0" encoding="utf-8"?>

<type>actor</type>

<type>regarding</type>

</supportedRuleTypes>

</publisher>

Publisher XML document conform to the following schema. Full Gnip XML schema can be found here.

...

    <xs:element name="publisher">
        <xs:complexType>
            <xs:sequence>
                <xs:element name="supportedRuleTypes" minOccurs="1" maxOccurs="1">
                    <xs:complexType>
                        <xs:sequence>
                            <xs:element name="type" type="ruleType" minOccurs="1" maxOccurs="unbounded" />
                        </xs:sequence>
                    </xs:complexType>
                </xs:element>
            </xs:sequence>
            <xs:attribute name="name" type="uriSafeType" use="required" />
        </xs:complexType>
    </xs:element>
...

Examples

Examples of Gnip Activities

The examples below are broken into Data Consumer and Publisher sections.

Most users of Gnip are Data Consumers, but depending on the application, a user may be a Data Consumer, a Publisher, or both. Both Publishers and Data Consumers need to be familiar with Gnip activities which are individually represented by an <activity/> document and collections of <activity/> documents are wrapped in <activities/> documents. Below are examples of each; the first example represents a "notification"-style activity that does not contain the activity's "full-data (payload)". Second is a "full-data" <activities/> document example. Documents like these are emitted by both a Publisher's notification stream as well as a Filter's notification and activity streams.

Sample of a Gnip Activity XML for notifications (.../notification/...)

<activities publisher="gnip-sample">
    <activity>
        <at>2009-02-02T17:05:01.000Z</at>
        <action>post</action>
        <activityID>0.33783490478</activityID>
        <URL>http://gnipherald.com/stories/0.33783490478/jousting-gone-bad.xml</URL>
        <source>Vignette-Web-Publisher</source>
        <place>
            <point>38.2638886 -106.126131</point>
            <featurename>Chloride Mine</featurename>
        </place>
        <actor uid="12339948@4994" metaURL="http://gnipherald.com/profiles/jvaleski">jvaleski</actor>
        <destinationURL>http://gnipherald.com/stories/0.33783490478/jousting-gone-bad.html</destinationURL>
        <tag metaURL="http://gnipherald/about/tag/armor">armor</tag>
        <tag metaURL="http://gnipherald/about/tag/staff">staff</tag>
        <tag metaURL="http://gnipherald/about/tag/injury">injury</tag>
        <to metaURL="http://gnipherald.com/profiles/barinek">barinek</to>
        <regardingURL metaURL="http://apnewswire.com/jousting">http://apnewswire.com/jousting/accident.html</regardingURL>
    </activity>
</activities>

Sample of a Gnip Activity XML with full data enabled (.../activity/...)

Examples for Data Consumers

Retrieve recent Activities for a given Publisher

This requires having a prior knowledge of the Publisher ID, a Publisher ID is usually just a commonly known name for the Publisher such as Publisher ID for twitter is "twitter". (A current list of Publisher IDs can be found at https://api.gnip.com/publishers.html. In order to retrieve recent activities for a Publisher (e.g. "gnip-sample"), construct a URL for the Publisher of interest and perform a HTTP GET to retrieve the application/xml representation of that activity stream; for details go here. You can only access activities without full data, e.g. "notifications," for a Publisher's Public Timeline. If you want full data, you need to create a Filter and access it that way; for an example, go here. Publisher data is namespaced, that is, the list of publishers listed on the Gnip API site are scoped to "gnip". When you create a publisher, they will be scoped to "my".

===>
   GET /gnip/publishers/gnip-sample/notification/current.xml
   Accept: application/xml

<---
   200 OK
   Content-Type: application/xml

<activities publisher="gnip-sample">
    <activity>
        <at>2009-02-02T17:05:01.000Z</at>
        <action>post</action>
        <activityID>0.33783490478</activityID>
        <URL>http://gnipherald.com/stories/0.33783490478/jousting-gone-bad.xml</URL>
        <source>Vignette-Web-Publisher</source>
        <place>
            <point>38.2638886 -106.126131</point>
            <featurename>Chloride Mine</featurename>
        </place>
        <actor uid="12339948@4994" metaURL="http://gnipherald.com/profiles/jvaleski">jvaleski</actor>
        <destinationURL>http://gnipherald.com/stories/0.33783490478/jousting-gone-bad.html</destinationURL>
        <tag metaURL="http://gnipherald/about/tag/armor">armor</tag>
        <tag metaURL="http://gnipherald/about/tag/staff">staff</tag>
        <tag metaURL="http://gnipherald/about/tag/injury">injury</tag>
        <to metaURL="http://gnipherald.com/profiles/barinek">barinek</to>
        <regardingURL metaURL="http://apnewswire.com/jousting">http://apnewswire.com/jousting/accident.html</regardingURL>
    </activity>
</activities>

NOTE: /current.xml is only exposed to Gnip Standard Edition users and is not available in the Gnip Community edition. In Gnip Community edition you can retrieve current.xml based on up to 25,000 rules in a filter on a publisher.

Retrieve activity for a Publisher 'n' minutes ago (up to 60 minutes in the past)

If you are interested in a specific span of time in the past, relative to now, you will need to determine what the difference is between you local clock and Gnip’s clock, to minimize clock drift as a variable in making requests for activities; for details go here.

===>

HEAD /

<---

200 OK

Date: Mon, June 9 2008, 9:08:43

The returned Date header field can be used to figure the difference between Gnip's clock and the local one. Once that is known, the activity bucket ID for the time of interest can be calculated. Once the activity bucket ID is known its URL can be constructed using the same format as above, replacing "current" with an activity bucket ID (e.g. 200806090910), and the data can be retrieved. See the below section called "Activity Buckets". The activity bucket ID is represented in UTC time. Please note that the two buckets, current and 0-1 minutes old contain a variable amount of data, that is, data will continue to be added to the buckets until a full minute has passed. Buckets that are from 1-2 minutes and older contain a data that is fixed.

===>
 GET /gnip/publishers/gnip-sample/notification/200806090910.xml
 Accept: application/xml 
<---
 200 OK
 Content-Type: application/xml

<activities publisher="gnip-sample">
    <activity>
        <at>2009-02-02T17:05:01.000Z</at>
        <action>post</action>
        <activityID>0.33783490478</activityID>
        <URL>http://gnipherald.com/stories/0.33783490478/jousting-gone-bad.xml</URL>
        <source>Vignette-Web-Publisher</source>
        <place>
            <point>38.2638886 -106.126131</point>
            <featurename>Chloride Mine</featurename>
        </place>
        <actor uid="12339948@4994" metaURL="http://gnipherald.com/profiles/jvaleski">jvaleski</actor>
        <destinationURL>http://gnipherald.com/stories/0.33783490478/jousting-gone-bad.html</destinationURL>
        <tag metaURL="http://gnipherald/about/tag/armor">armor</tag>
        <tag metaURL="http://gnipherald/about/tag/staff">staff</tag>
        <tag metaURL="http://gnipherald/about/tag/injury">injury</tag>
        <to metaURL="http://gnipherald.com/profiles/barinek">barinek</to>
        <regardingURL metaURL="http://apnewswire.com/jousting">http://apnewswire.com/jousting/accident.html</regardingURL>
    </activity>
</activities>

Create a Filter and retrieve its Activity

To create a Filter you need one, or more, "rules." Each Publisher has a set of supported rule types which constrain the types of rules that can be specified for a Filter in a Publisher. For example, if a Publisher supports the rule types "Actor" and "To", a Filter can contain rules of using those types but cannot contain rules of type "RegardingURL", "Tag", and "Source". If a Filter is created with a rule type that a Publisher does not support, the server will return a response containing of an error message and error status code. Here is a list of Publishers. In order to find out what rule types a publisher supports you can look here.

Send an HTTP POST request to create a Filter. For details go here.

===>
 POST /gnip/publishers/gnip-sample/filters.xml
 Accept: application/xml
 Content-Type: application/xml
 
 <filter name="examplefilter" fullData="true">
    <rule type="actor">jvaleski</rule>
    <rule type="actor">ajackson</rule>
</filter>
<---
 200 OK
 Content-Type: application/xml

 <result>Success</result>

Once the Filter is created, accessing full-data activity associated with it is similar to accessing Publishers' notifications. Note that since fullData was specified as true in this example, the result will include the activity payload, and you'll access it via the ".../activity/..." URI. Trying to access the ".../notification/..." URI would generate a "Not Authorized" error.

===>
 GET /gnip/publishers/gnip-sample/filters/examplefilter/activity/current.xml
 Accept: application/xml
<---
 200 OK
 Content-Type: application/xml
 
<activities publisher="gnip-sample">
    <activity>
        <at>2009-02-02T17:16:01.000Z</at>
        <action>post</action>
        <activityID>0.516958128858</activityID>
        <URL>http://gnipherald.com/stories/0.516958128858/jousting-gone-bad.xml</URL>
        <source>Vignette-Web-Publisher</source>
        <place>
            <point>38.2638886 -106.126131</point>
            <featurename>Chloride Mine</featurename>
        </place>
        <actor uid="12339948@4994" metaURL="http://gnipherald.com/profiles/jvaleski">jvaleski</actor>
        <destinationURL>http://gnipherald.com/stories/0.516958128858/jousting-gone-bad.html</destinationURL>
        <tag metaURL="http://gnipherald/about/tag/armor">armor</tag>
        <tag metaURL="http://gnipherald/about/tag/staff">staff</tag>
        <tag metaURL="http://gnipherald/about/tag/injury">injury</tag>
        <to metaURL="http://gnipherald.com/profiles/barinek">barinek</to>
        <regardingURL metaURL="http://apnewswire.com/jousting">http://apnewswire.com/jousting/accident.html</regardingURL>
        <payload>
            <title>Jousting Event Went Awry</title>
            <body>Two drunken guys, two jousting sticks, blindfolds and running full bore down a hallway at each other, lead to a night in hospital for two university students.  One with chest injury the other unharmed. REST of BODY</body>
            <mediaURL>http://images.gnipherald.com/jousting-thumbnail.jpg</mediaURL>
            <raw>base64 encoded gzip compressed binary</raw>
        </payload>
    </activity>
</activities>

Create a Filter and have its Activity POSTed to a specified URL

Create a Filter as in the previous example and in addition specify a postURL to which activities should be HTTP POSTed. Gnip validates the URL by making a HEAD request to it, so ensure that the postURL responds successfully to an HTTP HEAD request. For more information go to the detailed filters section.

===>
 POST /gnip/publishers/gnip-sample/filters.xml
 Accept: application/xml
 Content-Type: application/xml

 Accept-Encoding: gzip   (optional but recommended, our convenience libraries support this)
 Content-Encoding: gzip  (optional but recommended, our convenience libraries support this)

 
 <filter name="examplefilter" fullData="true">
    <postURL>http://mysite.example/inbound-activity-handler.cgi</postURL>
    <rule type="actor">jvaleski</rule>
    <rule type="actor">ajackson</rule>
</filter>
<---
 200 OK
 Content-Type: application/xml

<result>Success</result>

Once the Filter is created, any activity that matches rules in the filter will be POSTed to the specified URL; see below for an example HTTP exchange that occurs when sending activities to a postURL. Gnip activity POSTing is currently fire-and-forget and does not interpret HTTP response codes from the POST.

===>
 POST http://mysite.example/inbound-activity-handler.cgi
 Content-Type: application/xml

 Accept-Encoding: gzip   (optional but recommended, our convenience libraries support this)
 Content-Encoding: gzip  (optional but recommended, our convenience libraries support this) 
<?xml version="1.0" encoding="utf-8"?>
<activities publisher="gnip-sample">
    <activity>
        <at>2009-02-02T17:16:01.000Z</at>
        <action>post</action>
        <activityID>0.516958128858</activityID>
        <URL>http://gnipherald.com/stories/0.516958128858/jousting-gone-bad.xml</URL>
        <source>Vignette-Web-Publisher</source>
        <place>
            <point>38.2638886 -106.126131</point>
            <featurename>Chloride Mine</featurename>
        </place>
        <actor uid="12339948@4994" metaURL="http://gnipherald.com/profiles/jvaleski">jvaleski</actor>
        <destinationURL>http://gnipherald.com/stories/0.516958128858/jousting-gone-bad.html</destinationURL>
        <tag metaURL="http://gnipherald/about/tag/armor">armor</tag>
        <tag metaURL="http://gnipherald/about/tag/staff">staff</tag>
        <tag metaURL="http://gnipherald/about/tag/injury">injury</tag>
        <to metaURL="http://gnipherald.com/profiles/barinek">barinek</to>
        <regardingURL metaURL="http://apnewswire.com/jousting">http://apnewswire.com/jousting/accident.html</regardingURL>
        <payload>
            <title>Jousting Event Went Awry</title>
            <body>Two drunken guys, two jousting sticks, blindfolds and running full bore down a hallway at each other, lead to a night in hospital for two university students.  One with chest injury the other unharmed. REST of BODY</body>
            <mediaURL>http://images.gnipherald.com/jousting-thumbnail.jpg</mediaURL>
            <raw>base64 encoded gzip compressed binary</raw>
        </payload>
    </activity>
</activities>
<---
 200 OK

A note on authentication: Gnip does not support a Filter owner specifying credentials to use on requests that post activities to a Filter's postURL. As an interim solution, a Filter owner can provide an opaque token on the postURL as part of the URL string. If this token is only known to Gnip and the postURL's server, it can be validated on your server when Gnip sends activities to a postURL to ensure that POST requests originate from Gnip.

Examples for Publishers

Publish Activities to an Activity Stream

If you are a Publisher interested in publishing Activities to an Activity Stream, POST application/xml to a Publisher created by you in the following example. For details go here. Required fields are:

<at>

<action>

If a payload is added you must also include (it can be empty):

<raw>

The supported Gnip rule types are:

actor

tag

regarding

source

If you support one or more of the above rule types, the field for that type is required.

===>

POST /my/publishers/PUBLISHER-NAME/activity.xml

Accept: application/xml

Content-Type: application/xml

   Accept-Encoding: gzip   (optional but recommended, our convenience libraries support this)
   Content-Encoding: gzip  (optional but recommended, our convenience libraries support this)

<activities publisher="gnip-sample">
    <activity>
        <at>2009-02-02T17:16:01.000Z</at>
        <action>post</action>
        <activityID>0.516958128858</activityID>
        <URL>http://gnipherald.com/stories/0.516958128858/jousting-gone-bad.xml</URL>
        <source>Vignette-Web-Publisher</source>
        <place>
            <point>38.2638886 -106.126131</point>
            <featurename>Chloride Mine</featurename>
        </place>
        <actor uid="12339948@4994" metaURL="http://gnipherald.com/profiles/jvaleski">jvaleski</actor>
        <destinationURL>http://gnipherald.com/stories/0.516958128858/jousting-gone-bad.html</destinationURL>
        <tag metaURL="http://gnipherald/about/tag/armor">armor</tag>
        <tag metaURL="http://gnipherald/about/tag/staff">staff</tag>
        <tag metaURL="http://gnipherald/about/tag/injury">injury</tag>
        <to metaURL="http://gnipherald.com/profiles/barinek">barinek</to>
        <regardingURL metaURL="http://apnewswire.com/jousting">http://apnewswire.com/jousting/accident.html</regardingURL>
        <payload>
            <title>Jousting Event Went Awry</title>
            <body> Two drunken guys, two jousting sticks, blindfolds and running full bore down a hallway at each other, lead to a night in hospital for two university students. One with chest injury the other unharmed. REST of BODY </body>
            <mediaURL>http://images.gnipherald.com/jousting-thumbnail.jpg</mediaURL>
            <raw>base64 encoded gzip compressed binary</raw>
        </payload>
    </activity>
</activities>

<---

200 OK

Content-Type: application/xml

<result>Success</result>

Using the Gnip REST API using Curl

This section is an overview of using CURL with the REST API. A full tutorial of using CURL on windows and on mac/linux with example files can be found based on their respective OS: windows and mac/linux.

It is easy to show how to use the API using curl without having to get into language specifics (Java, .Net, Perl, Ruby, PHP, etc). Curl gives us that neutral territory. Curl is native to Mac OSX and most distributions of linux have it installed. Windows users can download curl from http://curl.haxx.se/download.html. Scroll down to the Win32 section (a few pages down) and click on the bottom of the list, the libcurl version.

Go the appropriate section:

Windows

Mac/Unix/Linux

Windows

A full tutorial of using CURL with example files can be found at www.gnip.com/docs/tutorial_win.pdf

For windows either create this as a batch file (setenv.bat) or cut and paste and run this in the cmd.exe

Set gnipusername=yourgnipaccount@company.com

Set gnippassword=yourpassword

Set host=https://api-v21.gnip.com

This allow the curl statements to use these environment variables without you having to type them in again and again.

To delete a filter:

curl -k -X DELETE --user %gnipusername%:%password% "%host%/gnip/publishers/digg/filters/buzzfilter.xml" --header "Content-Type: application/xml"

To create a filter:

curl -k -X POST --user %gnipusername%:%password% "%host%/gnip/publishers/digg/filters" -T buzzfilter.xml --header "Content-Type: application/xml"

Where buzzfilter.xml contains the following:

    <rule type="tag">celebrity</rule>
    <rule type="tag">entertainment</rule>
    <rule type="tag">world news</rule>
    <rule type="tag">political & news</rule>
</filter>

To retrieve activity on the 'buzzfilter':

curl -k --user %gnipusername%:%password% "%host%/gnip/publishers/digg/filters/buzzfilter/activity/current.xml" --header "Content-Type: application/xml

Convenience Libraries

Convenience Libraries are available at http://github.com/gnip

They are available for the following languages:

Gnip Developer Site

At the Gnip Developer site, you can create publishers, filters, edit rules and view activity.

The most common task is to create your own filter on a Gnip endorsed publisher. Below is a walkthru of creating a filter on Digg that matches any chatter about Celebrities.

Configure Filters through the Gnip developer Site

Naturally you can create filters and update filter matching conditions such as actor and tag through the API. You can also create filters though the Gnip web UI. This allows your to get up and running quickly as you can to start seeing data immediately (provided the data matches the filter). It also allows you to understand how the system works.

Example of setting a filter on Digg

If you haven't already signed up for an account, signup for one here.

This will take you to the developer page:

Scroll down all the way down the page to the section below, which is called 'Active Publishers in the system', this is where you can create filters on a publisher, view filters and publisher details. Note: Filters are per publisher and not global across all publishers, this is by design as all publishers don't support all types of match condition such as tags, regarding and different people use different names commonly on different publishers/services.

Click on "[create filter] on digg will bring you to new filter page.

If you want full data then have this selected on the "full data" checkbox. If your just after notifications then have "Full Data" checkbox unselected.

Within this page it shows that publisher your making a filter on is 'digg'. The grayed out fields are fields not supported by the publisher. In this case digg supports Actor, Tag and two.

In this example, imagine we are wanting to retrieve all Dugg articles that are on Celebrities. In this case we set the name of the filter to be "celebritydigg", we set the tag to be "Celebrity" and then hit save. This is where the magic happens. From this point onwards any Dugg articles with celebrities will be either sent to us if you enter the website Post URL or put into buckets if you wish to poll for the information. In this example we did not set the Post URL.

To see if your filter is working then click on 'view my filter', then click on activities. (Since full data was selected activities can be selected, if full data was not selected then notifications could be selected)

Clicking on activities gives a list of activities and highlights which activities actually have data

From activity you can see that in the last 2-3 minutes there was 'Celebrity activity'. Clicking on this will allow you to see the xml activity data. (Do this in firefox and not Safari, in Safari you will need to view source to see the XML data more clearly.) In firefox it looks like:

Using the web interface for filters it is easy to help solve some of the problems with filters and feeds. When people are learning to use the system this tool becomes invaluable.

Developer Community

For any problems, updates to the Gnip service, new upcoming versions, convenience libraries and lots more, then go to http://groups.google.com/group/gnip-community

Gnip Schema Document

The full Gnip schema can be found at https://api-v21.gnip.com/schema/gnip.xsd

Miscellaneous

For Gnip publishers that are query based it is possible to submit advanced search parameters. if the rule includes additional parameters for language or geo-code then the rule must be formed properly as a query string. This is a BETA feature at this time and is not generally supported.

For example. When using the Twitter-Search publisher to filter on a keyword for the band "Devo" with only English results returned:

http://search.twitter.com/search.atom?q=devo&lang=en

Proper rule submission to Gnip: q=devo&lang=en
Improper rule submission to Gnip: lang=en&q=devo

Revision History

rev 20 - putting new diagrams on the REST API, changed pictures to meet the new Gnip style guide
rev 19 - made changes to the URL encoding section - we URL encode from the web, using the XML api you need to URL encode and make safe XML before sending to us
rev 18 - big changes to formatting, to links, to pictures. Changed the demo-v21 to api-v21
rev 17 - added the URL section for my and gnip URLS, moved schema details to the right section, added more links to the table of contents
rev 16 - add more important points to the GNIP information - UTC time, application/xml, clock drifts and add more points to HTTP Basic auth
rev 15 - added filter in the UI, added gzip performance documentation, added UI entering different special cases for words, added filtered terms we support, added publisher activity posting
rev 13 - updated a publisher filter with multiple support rule type tags, changed points to utf-16 to utf-8, updated the request for delete to have content: application/xml
rev 12 - Update to delete URL endpoint
rev 11 - Changed Publishers and Subscribers to Data Producers and Data Consumers. Fixed URLs, added publishing to an activity, did color changes.
rev 10 - changed XML examples from gnipherald publisher to gnip-sample
rev 9 - updated xml examples to include final schema, added delete publisher example
rev 8 - minor editing tweaks
rev 7 - some minor editing tweaks
rev 6 - new schema, version 2.1
rev 5 - added documentation for batch and search support in the rules/ endpoint. Added documentation to the Activities section of the samples. Rearranged some links and made some XML formatting changes.
rev 4 - updated for publisher capabilities, added paragraph on authentication for activity push, remove references to JIDs, document cleanup
rev 3 - updating publisher activity endpoint to reflect the correct one.
rev 2 - added verbiage to further clarify filter management.
rev 1 - moved bucket-ID API from five minute buckets to single minute buckets
rev 0 - initial cut of the 2.0 document