1 of 7

VIVA: An End-to-End System for Interactive Video Analytics​

Francisco Romero, Johann Hauswald, Aditi Partap,

Daniel Kang, Matei Zaharia, Christos Kozyrakis

1

2 of 7

Growing demand for video analytics

2

“How many cars passed by on Monday?”

“Find clips of Jake Tapper interviewing an angry Bernie Sanders”

“Show me Ronaldo’s headers when he played for Juventus”

3 of 7

Example – analyzing TV news

3

“Find clips of Jake Tapper interviewing an angry Bernie Sanders”

Not obvious which is best!

Train specialized model to detect Tapper and Sanders

Detect frames with Tapper and Sanders

Detect “angry Sanders” frames

Detect angry faces in frames

Detect Sanders from angry face frames

Detect Tapper in frames with “angry Sanders”

Query result

…

4 of 7

Existing work focuses on specific components

4

System

Component

Query

NoScope, Focus, …

Query optimizer/execution

Selection

MIRIS

Query execution

Tracking

BlazeIt

Query optimizer

Aggregation, limit

TASTI

Index

Proxy-based

Users today must manually combine these techniques/systems!

5 of 7

Video query challenges

Goal: enable videos to be searched like structured data

Challenges:

  • Ad-hoc, exploratory queries
  • Require expensive DNNs to extract information

Expensive hardware: A100 is $4.10/hr

Slow: as low as 3 frames per second

  • Optimization space spans structured and unstructured data

​

​

5

Spark SQL

6 of 7

VIVA

  • End-to-end interactive video analytics system
  • Enables interactive exploration of large video corpuses by:
    • Allowing users to describe domain-specific relations
    • Blending structured and unstructured data during query optimization
    • Co-optimizing storage and compute for efficient DNN execution

​

Under active development!

​

6

VIVA

Heterogeneous Serverless Backends

SELECT time_window FROM news_analysis

WHERE tapper_angry_sanders = TRUE

7 of 7

Open research directions

​

​

  • Specifying and incorporating domain knowledge
  • Determining when DNNs should be (re)-trained
  • Infrastructure for high throughput demands

​

​

7

VIVA

Query frontend

Relational hint explorer

Mixed-data optimizer

Accelerator-based execution engine

Structured table

Embedding cache

Serverless backends

Video file manager

Model

(re)-training service

Stop by my poster to learn more!