1 of 20

You Described, We Archived:

A Rich Audio Description Dataset

Charity Pitcher-Cooper, Manali Seth, Benjamin Kao

James M. Coughlan, Ilmi Yoon

​

2 of 20

Happy Birthday to YouDescribe!

Launched in 2013, Rebuilt in 2017, Turning 10 in 2023

2

3 of 20

What is YouDescribe?

  • It’s a free accessibility tool to add audio description to YouTube videos built especially for the BVI community

​

  • Viewers request videos they would like described through a wishlist

​

  • Volunteers create a script, record tracks, and edit track placement

​

  • The tracks and timing data is stored in a separate API and played together with YouTube video

​

​

​

3

4 of 20

What YouDescribe isn’t

​

  • YouDescribe isn’t a replacement for professional audio description. It’s a tool for what would be left undescribed due to:

​

    • Time
    • Cost
    • Importance

​

4

5 of 20

YouDescribed, We Archived (YuWA)

What’s a data paper and why write one?

    • A data paper is a detailed description of a data set
    • Datasets are time consuming to gather/archive and incredibly valuable with potential applications in many research fields
    • A data paper describes the metadata of a particular data set, and links to a repository that indexes and organizes that data so it can be accessed, refined, utilized
      • Metadata is “data about data”
      • Who, what, where, why, and how of the data collection
      • It draws no conclusion, it is a catalog or blueprint of the data
      • Descriptive metadata: data that describes information about a resource or a file. It is used to help with discovery and identification

5

6 of 20

What data do we collect?

(and don’t collect)

  • Our viewer and describer privacy are very important to us.

  • All data is anonymized

​

  • No data is collected for visitors to www.youdecribe.org that are under 18 years of age

​

  • We do not share email addresses, names, or any other identifying information

​

  • General visitor data is collected by google analytics (started in 2020)

​

6

7 of 20

The WHO of the YuWA data set

Our viewers are diverse

YouDescribe is first and foremost an accessibility tool for the BVI community. (but lots of other groups use it too)

Videos can be viewed from anywhere in the world with internet access and a computer

We also have an iOS app for apple phones and tablets.

Our describers are diverse

Over 3,000 volunteer describers

Over 7,000+ audio descriptions

Approximately 1,000+ ADs added in 2022, and on average 90+ videos described each month

More volunteers providing descriptions each year

7

YouDescribe.org has 12,000+ average annual visitors

They are global- But mostly the US.

8 of 20

Number of audio descriptions added yearly

YEAR- # of descriptions

​

2017 - 287

2018 - 579

2019 - 718

2020 - 1104

2021 - 1324

8

9 of 20

The Magic of Premium Data Sets

One video- so many descriptions

Over 80 descriptions for a single 21 second video

One describer club, 5 years of videos

HS age students, with identical training

9

10 of 20

Is the data any good?

Oxford University Group QuerYD

Dataset trainer for retrieval and event localization in videos

Compared to other training data sets content descriptions made with YouDescribe data:

  • more relevant than dialogue alone
  • more detailed than previous description attempts using crowd-sourced image descriptions

​

Our YuWA Dataset

Our data set is better

We have more metadata

We have twice the audio (both the video dialog and the audio descriptions)

We have extended descriptions

Better metadata means more research applications

10

11 of 20

You Described, We Archived Data Repository

URL : https://github.com/youdescribe-sfsu/You-Described-We-Archived

​

Steps to download the data

    • You will be asked to register
    • You will be given an API key
    • Run the provided Python file (download script)

​

​

You Described, We Archived ©️ 2022 by Smith-Kettlewell Eye Research Institute, San Francisco State University is licensed under CC BY-NC-ND 4.0.

​

11

12 of 20

Possible AI/ML applications: AI-Enhanced Video Accessibility for YouDescribe

Ilmi Yoon - Professor, Dept. Computer Science, San Francisco State University

  • YouDescribeX will support sighted volunteers to produce audio descriptions for YouTube videos - much faster and with much lower cognitive load than before

​

​

12

13 of 20

Data Statistics I

3100+ describers and

7,000+ audio descriptions (82,000+ audio clips ).

​

About 22% of the audio descriptions are rated out of which

59% are excellent,

25% are very good,

8% are good,

5% are fair, and

only 3% are rated as poor.

​

13

14 of 20

Video Categories

14

15 of 20

Data Statistics II – a typical YouDescribe video

Total Videos with High Quality AD: ~6300

Average duration of videos: 1,870,000 sec / 6300 => ~ 300 seconds ~5 minutes

​

Total audio descriptions: ~7000

Average duration of audio descriptions: ~7000000 sec / 7000 ~ => 100 seconds (or 1 minute 40 sec)

​

Total audio clips: ~82000

Average number of audio clips: 82000 / 7000 => ~ 12

Average duration of audio clips: ~ 8.5 seconds

Max duration of audio clip: 405.36 seconds ~ 6.75 minutes

​

Conclusion: 12 audio clips per description over 5 min videos, so about one audio clip per 25 seconds

​

15

16 of 20

Data Statistics III

16

17 of 20

Data Statistics IV

17

18 of 20

Multiple ADs per video

(total duration and counts of audio clips)

18

19 of 20

Multiple ADs per video

(Insertion points and description style decision)

19

20 of 20

Thank you for your attention!

A special thank you to YouDescribe inventor and mentor Dr Joshua Miele. There is no YouDescribe without you.

This work was supported by grants from:

  • NEI Department of Education
  • NIDLIRR- RERC on Blindness and Low Vision
  • Ability Central (formerly The Disabilities Communication Fund)

​

20