MetaFragments - draft

The idea behind MetaFragments is to use media fragments URIs and in-HTML meta-data to store timed data relating to audio and videos sequences – In the DOM, so web clients, crawler and javascript libraries can easily find and use this data without other prior knowledge than existing standards.

Why?

Our aim is to make web based audio and video first class citizens of the web of data.

Meta-data for video or audio is mostly non-existent in exposed, usable form.

When it exists, it’s segregated in various silos: external files in non-browser friendly formats, APIs, JavaScript code…

We believe that for timed meta-data to be successful, the format must be:

simple and fitting in already existing flows
based on web usages and standards
as near to the resource as possible

We also feel that Popcorn.js would benefit from a standard way to expose timed metadata in the web page, so that other JavaScript libraries or specific web clients (ie. search crawlers, mobile apps) could use the data without prior knowledge of the Popcorn.js app.

(as a reference, here are some prior notes on the state of web video (meta) data)

Scope

This is in-HTML, DOM addressable, timed meta-data format. We focus on data sharing and exposition, and we don’t address the presentation of the data or the interactivity.

It will reuse existing vocabularies and standards, like media fragments URIs, RDFa, Dublin Core, EmotionML sanctioned vocabularies…

interesting side effects of the format

the web site becomes a media API for video and audio, without the need for a dedicated entry point
you can copy and paste the HTML without loosing the reference to the sequences

Who can benefit from this standard?

A web developer who needs to manipulate metadata about time-based media (video or audio) in the browser.

Benefit for the developer:

A straightforward x-browser method of exposing media metadata and/or the ability to link, assemble and integrate media ‘fragments’ into their applications.

A journalist who needs to identify hidden stories and connections in the flow of video reporting, because she wants to go beyond just passing them.

Benefit for the journalist:

Both Professional and citizen journalists can easily and quickly make reference to and expose fragments of media and in addition more easily assemble ‘stories’ from various fragments of media.

An artist who

Benefit for the artist:

Artists can more easily integrate elements of media into their work, additionally they can take advantage of various meta data (especially that related to emotion) to imbibe meaning into their art.

A scientist that needs a web solution to exploit an interview corpus, because the teams she collaborate with are spread in various part of the world

Benefit for the scientist:

Scientists and academics can use MetaFragments to analyse media, correlations and references can be more easily made.

Basic structure

The basic structure looks like this - very simple:

<a rel=”dc:source” RDFa-META-DATA href=”MEDIA-FRAGMENT-URI”'>WORD-FROM-THE-TRANSCRIPT</a>

The A element is used, to reference the external media fragment we are describing.

Note: in HTML5, it is standard for A to contain any number of block level elements. For example we could have a full description with multiple P and IMG inside our A. This was used widely even before, but wasn’t standard.

We set the REL attribute to DC:SOURCE, as our html document is based on the content of the referenced media. A composite transcript will have different medias sources.

Other HTML5 attributes can/should be added if appropriate, like HREFLANG.

Note: we also have to declare the use of dublin core beforehand, <div xmlns:dc="http://purl.org/dc/elements/1.1/">…</div>

RDFa meta-data indicate that this text is a transcript, using PROPERTY and CONTENT.
In other case, RDFa can be used to indicate that a person, an emotion, a gesture, a facial expression, a place, is appearing in the sequence (see below for concrete examples)

the HREF attributes contain a media fragment URI that uniquely represent a specific portion of the media, in term of time and/or picture zone.

Note : The Media Fragment URI is used as a standardized, unique and non-equivocal identifier for any part of the media, and thus the URI is not necessarily referencing a specific file on the server. In simple cases, the media is the file. But in more complex case, the media exist in a variety of format.

As the media fragment URI is a valid URL, any web client (browser, search engine crawler, native app) should be allowed to use content negociation with the server to get the media in its prefered format – for example H264 on iOS, but webM in Firefox.

The server should respond to a request to this URL, in order to accommodate a web client without prior knowledge of the location of the various version of the media.

If we are writing a transcript, or subtitles, we then add the corresponding text for the fragment inside the A element. [might also be the place to add names, companies and other info that usually go to the lower third in videos]
The nature of the text – transcript, subtitle, translation, voice-over, etc. – is described by the RDFa meta-data. Any web client can make sense of it.

Vocabularies

Dublin Core

EmotionML sanctionned vocabularies

Use cases: from ad-hoc format to MetaFragments

Thanks to Samuel Huron, Mark Boas, Nicholas Doiron, we have a series of web apps and their actual format as use cases.

At the moment, these 3 web apps use various formats in the browser: an external HTML fragment; Cinelab, an independantly developed XML format; and a JSON format.

The use of the MetaFragments format for the front end allows interoperability, as any web client can make sense of the data.

The HTML page becomes the meta-data API of these three web apps.

Hyperaudio Pad by Mark Boas

«Hyperaudio Pad will allow people to quickly and easily assemble an audio or video program from hyperlinked word-time-aligned transcripts (hypertranscripts) by simply copying and pasting their text. Tightly coupled word-level media references remain intact.»

Example of the existing format for hypertranscripts

maybe,

you

know,

if

we

took--

Excerpt from this HTML fragment

Proposed basic format for the hypertranscript:

<a rel="dc:source" hreflang="en" href="http://hyper-audio.org/r/media#t=34160,34390”>maybe,</a>

The transcript code could be beautified with SECTION, P, B and I elements without interfering with the data, allowing the web page to be usable even without Javascript.

Benefit of using MetaFragments : As the transcript is purely HTML links, we can imagine an advanced user cutting and pasting the links directly in an HTML editor, and then back in Hyperaudio Pad. And Hyperaudio pad will immediately know what are the media referenced.

More advanced formating (with RDFa attributes in the mix)

<a rel=”dc:source” media=”audio” href="http://hyper-audio.org/r/media”'
property=”dc:type” content=”interview” >an interview</a>

<a rel=”dc:source” media=”audio” href="http://hyper-audio.org/r/media”' about=”http://www.geonames.org/2950159”
property=”place” content=”Berlin”>in Berlin</a>

<a rel=”dc:source” media=”audio” href="http://hyper-audio.org/r/media#t=34160,38970”
property=”host” content=”Krulwich, Robert” >Robert Krulwich:</a>

<a rel="dc:source" media=”audio” hreflang="en" href="http://hyper-audio.org/r/media#t=34160,34390”
property="transcript" >maybe,</a>

[we need to have vocabularies for the terms: transcript, subtitle, caption, comment etc? and interviewee, guest, host, etc?]

Visualization of audience feedback using Twitter - Polemic tweets by Samuel Huron

«Polemictweet.com is a webservice using social micropost stream (like twitter) to enhance event reporting, video browsing experience, and engaged user annotation with a really simple method. By aggregating the conversation this service help you to find the hotspot of the talk, and polemically position your comment. »

Example of an existing cinelab annotation serialized in JSON (not twitter one):

{

"begin": "1893800",

"end": 2196840,

"tags": [

{"id-ref": "5055cf7c-beea-11e0-9457-00145ea49a02" },

{"id-ref": "5055d3d2-beea-11e0-9457-00145ea49a02" },

{"id-ref": "5055d7ec-beea-11e0-9457-00145ea49a02" },

{"id-ref": "5055dbde-beea-11e0-9457-00145ea49a02" },

{"id-ref": "5055dfd0-beea-11e0-9457-00145ea49a02"},

{"id-ref": "5055e3cc-beea-11e0-9457-00145ea49a02"},

{"id-ref": "5055e82c-beea-11e0-9457-00145ea49a02"},

{"id-ref": "5055ec1e-beea-11e0-9457-00145ea49a02" }

"media": "9a493932-3053-11e0-862b-00145ea49a02",

"content": {

"mimetype": "application/x-ldt-structured",

"description": "synchronised crowded, catalyst, plausible promess, success, fail, Wikipedia, small and good, large and mediocre",

"title": "Question 1 : catalisation",

"color": "3355443",

"polemics": [Q],

"audio": {

"mimetype": "audio/mp3",

"src": "",

"href": null

}

"meta": {

"dc:contributor": "perso",

"id-ref": "c_E0FF6CF0-B8E9-8432-8B92-293EFFFFA827",

"dc:created": "2011-08-04T22:37:20.040839",

"dc:modified": "2011-08-04T22:37:20.040839",

"dc:creator": "perso"

"id": "s_E4C933C8-D4AB-7A24-B2D1-2A38EA18AB54"

Excerpt from a complete cinelab json file

It follows this data model [PDF] and exist in an xml serialization too.

Same data using MetaFragments

Adding meta-data on interviews with FollowFrost by Nicholas Doiron

«For online interviews, FollowFrost breaks the ice between viewers and the interviewee. Using Popcorn.js, you can read the speaker's profile, follow them on Twitter, and ask follow-ups. Video answers are broken down by question and added to the main video»

Example of the existing format for FollowFrost viewer

http://mapmeld.appspot.com/followfrost/view?id=932001

function connectPlugins(){
        people = [
                {person:'Beginning',start:0,end:0,twitter:"makemagazine"},
                {person:'Becky Stern, co-host of Make: Live',start:20,end:50,twitter:"bekathwia"},
                {person:'Matt Richardson, co-host of Make: Live',start:50,end:70,twitter:"MattRichardson"},
                {person:'Dale Dougherty, creator of Maker Faire',start:70,end:203,twitter:"dalepd"},
                {person:'Theatre Bizarre, illegal playwrights',start:203,end:273,twitter:"theatrebizarre"},
                {person:'Gon-KiRin, homemade dragon',start:273,end:597,twitter:"gonkirin"},
                {person:'William Gurstelle, author of Practical Pyromaniac',start:597,end:1025,twitter:"wmgurst"},
                {person:'Russ Wolfe, i3detroit Hackerspace Tour',start:1025,end:1121,twitter:"remakedetroit"},
                {person:'Jim Burke, PowerRacingSeries',start:1121,end:1484,twitter:"ppprs"},
                {person:'DigiKey',start:1484,end:1568,twitter:"digikey"},
                {person:'Backyard Brains Demo (warning: cockroaches)',start:1568,end:2116,twitter:"backyardbrains"}
        ];
        myPopcorn
        .tagthisperson({
                start:50,
                end:70,
                person:'Matt Richardson, co-host of Make: Live',
                image:'http://0.gravatar.com/avatar/8090f2411aa56ec040114f39fa56922e?s=212&d=http%3A%2F%2F0.gravatar.com%2Favatar%2Fad516503a11cd5ca435acc9bb6523536%3Fs%3D50&r=G',
                href:'http://twitter.com/mattrichardson',
                target: 'tagdiv'
        })
        .twitter({
                start:50,
                end:70,
                title:'Follow on Twitter',
                src:'@MattRichardson',
                target:'twitterdiv'
        })
        .tagthisperson({
                start:20,
                end:50,
                person:'Becky Stern, co-host of Make: Live',
                image:'http://sternlab.org/wp-content/uploads/2011/03/Screen-shot-2011-03-17-at-9.45.15-AM.png',
                href:'http://twitter.com/bekathwia',
                target: 'tagdiv'
        })
        .twitter({
                start:20,
                end:50,
                title:'Follow on Twitter',
                src:'@bekathwia',
                target:'twitterdiv'
        })
        .twitter({
                start:0,
                end:24,
                title:'Follow on Twitter',
                src:'@Makemagazine',
                target:'twitterdiv'
        })
        .tagthisperson({
                start:70,
                end:203,
                person:'Dale Dougherty, creator of Maker Faire (photo CC-BY James Duncan Davidson)',
                image:'http://upload.wikimedia.org/wikipedia/commons/thumb/5/5e/Etech05_Dale.jpg/220px-Etech05_Dale.jpg',
                href:'http://twitter.com/dalepd',
                target: 'tagdiv'
        })
        .twitter({
                start:80,
                end:203,
                title:'Follow on Twitter',
                src:'@dalepd',
                target:'twitterdiv'
        })