The Slow & Steady March for Generality in Vision

Tanmay Gupta | Research Scientist @ Allen AI

Enough of GPTs, we want GPVs!!

My computer can do 99 things, seeing well ain’t one ?!?!

General Purpose Systems

General Purpose Systems

Image

Class Label

Image

Caption

Image

Segmentation

Map

Image

Pose Keypoints

General Purpose Systems

Image

Output

Instruction

Classify this dog breed

Describe this image

Segment the cars, roads, and traffic signs

Localize the elbows and the knees

Class Label

Caption

Segmentation masks

Key points

ChatGPT �A General Purpose NLP System

In context learning

Describing an end task

Towards General Purpose Vision

Towards General Purpose Vision

GPV-1

Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture

Gupta et al. CVPR 2022

GPV-1

No more task specific branches

Use modality specific branches

GPV-1

No more task specific branches

Use modality specific branches

Visual input

Language input

Task instruction

GPV-1

No more task specific branches

Use modality specific branches

Cross modal fusion

GPV-1

No more task specific branches

Use modality specific branches

Bounding boxes + scores

Language output

Support tasks like

Captioning / VQA

Support tasks like

Loc / RefExp

Skill-Concept Generalization

Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture

Gupta et al. CVPR 2022

Skills

Concepts

GPV-1

Learn Skill

Learn Concepts

Model understands 80 concepts across 4 skills

How do we expand the concept vocab to 10K concepts?

Webly Supervised Concept Expansion for General Purpose Vision Models

Amita Kamath*, Chris Clark*, Tanmay Gupta*, Eric Kolve, Derek Hoiem, and Ani Kembhavi. ECCV 2022

GPV-2

Learn Skill

Learn Concepts

Use skill-concept generalization to transfer concepts from classification to other skills

Webly Supervised Concept Expansion for General Purpose Vision Models

Amita Kamath*, Chris Clark*, Tanmay Gupta*, Eric Kolve, Derek Hoiem, and Ani Kembhavi. ECCV 2022

Towards General Purpose Vision

Towards General Purpose Vision

Towards General Purpose Vision

Towards General Purpose Vision

Towards General Purpose Vision

Towards General Purpose Vision

More data

More parameters

More compute

Towards General Purpose Vision

More data

More parameters

More compute

Little/no new data

Little/no new learning

Uses existing models

Tag the characters on the TV show Big Bang Theory

1. detect faces

2. query knowledge

Penny

Leonard

Sheldon

Raj

Amy

Bernadette

Howard

3. matches faces and names

Tag the characters on the TV show Big Bang Theory

1. detect faces

2. query knowledge

4. display labeled bboxes

Penny

Leonard

Sheldon

Raj

Amy

Bernadette

Howard

3. matches faces and names

Tag the characters on the TV show Big Bang Theory

end-to-end

model

Unified-IO

end-to-end

model

Face detection

Knowledge retrieval

Image-Text matching

Image Processing

Plotting

Training

Funnel

end-to-end

model

Create a color pop of the first ladies and blur the rest

The left and right images contain a total of six people and two boats

False

Image classification

Object detection

Face recognition

Knowledge retrieval

Image generation

Pose estimation

Depth estimation

Semantic segmentation

Captioning

Question Answering

Surface normal estimation

Image processing

Scientific plotting

Text generation

Reading comprehension

Commonsense knowledge

Image Inpainting

Situation Recognition

Classification in context

Training

Funnel

Tag the characters on the TV show Big Bang Theory

An implementation for each step

The program that calls the functions in sequence

Execute the program on the given input

tag_characters.py

end-to-end

model

Image classification

Object detection

Face detection

Knowledge retrieval

Image generation

Pose estimation

Depth estimation

Semantic segmentation

Captioning

Question Answering

Surface normal estimation

Image processing

Scientific plotting

Text generation

Reading comprehension

Commonsense knowledge

Image Inpainting

Situation Recognition

Classification in context

Training

Funnel

image_classification ()

object_detection ()

face_detection ()

knowledge_retrieval ()

image_generation ()

pose_estimation ()

depth ()

semantic_segment ()

image_captioning ()

question_answering ()

image_text_matching ()

image_proc ()

image_plot ()

text_generation ()

reading_comp ()

commonsense ()

inpainting ()

situations ()

class_context ()

INVOKE

program.py

Visual Programming

Language description of a visual task

Program

Program

Generator

Output

Program

Interpreter

Task Inputs

Visual Programming

Language description of a visual task

Program

Program

Generator

Output

Task Inputs

In-context Learning in LLMs

Python + HTML

Program

Interpreter

VisProg Modules

Image Processing Libraries

Any program you like!

Neural Models

OBJ0 = Segment(image=IMAGE)

OBJ1 = Select(image=IMAGE, object=OBJ0, query='desert')

IMAGE0 = Replace(image=IMAGE, object=OBJ1, prompt='lush green grass')

FINAL_RESULT = Result(var=IMAGE0)

IMAGE

Replace the desert with lush green grass

OBJ0 = Segment(image=IMAGE)

OBJ1 = Select(image=IMAGE, object=OBJ0, query='desert')

IMAGE0 = Replace(image=IMAGE, object=OBJ1, prompt='lush green grass')

FINAL_RESULT = Result(var=IMAGE0)

Replace the desert with lush green grass

OBJ0 = Segment(image=IMAGE)

OBJ1 = Select(image=IMAGE, object=OBJ0, query='desert')

IMAGE0 = Replace(image=IMAGE, object=OBJ1, prompt='lush green grass')

FINAL_RESULT = Result(var=IMAGE0)

Replace the desert with lush green grass

OBJ0 = Segment(image=IMAGE)

OBJ1 = Select(image=IMAGE, object=OBJ0, query='desert')

IMAGE0 = Replace(image=IMAGE, object=OBJ1, prompt='lush green grass')

FINAL_RESULT = Result(var=IMAGE0)

Replace the desert with lush green grass

OBJ0 = Segment(image=IMAGE)

OBJ1 = Select(image=IMAGE, object=OBJ0, query='desert')

IMAGE0 = Replace(image=IMAGE, object=OBJ1, prompt='lush green grass')

FINAL_RESULT = Result(var=IMAGE0)

Replace the desert with lush green grass

OBJ0 = Segment(image=IMAGE)

OBJ1 = Select(image=IMAGE, object=OBJ0, query='desert')

IMAGE0 = Replace(image=IMAGE, object=OBJ1, prompt='lush green grass')

FINAL_RESULT = Result(var=IMAGE0)

FINAL_RESULT

Replace the desert with lush green grass

Information about the task and tools available for the task

Instruction: Replace the BMW with an Audi and cloudy sky with a clear sky

Program:

GPT3

Program Generator

Instruction: Hide the face of Nicole Kidman with :p

Program:

OBJ0=Facedet(image=IMAGE)

OBJ1=Select(image=IMAGE, object=OBJ0, query=‘Nicole Kidman’)

IMAGE0=Emoji(image=IMAGE, object=OBJ1, emoji=‘face_with_tongue’)

RESULT=IMAGE0

​

Instruction: Create a color pop of the white Audi

Program:

OBJ0=Seg(image=IMAGE)

OBJ1=Select(image=IMAGE, object=OBJ0, query=‘white Audi’)

IMAGE0=ColorPop(image=IMAGE, object=OBJ1)

RESULT=IMAGE0

​

Instruction: Replace the red car with a blue car

Program:

OBJ0=Seg(image=IMAGE)

OBJ1=Select(image=IMAGE, object=OBJ0, query=‘red car’)

IMAGE0=Replace(image=IMAGE, object=OBJ1, prompt=‘blue car’)

RESULT=IMAGE0

Instruction: Replace the BMW with an Audi and cloudy sky with a clear sky

Program:

GPT3

Program Generator

Performance with varying # context examples

Tag these famous computer vision researchers from Berkeley and Stanford

OBJ0=FACEDET(image=IMAGE)

LIST0=LIST(query='famous computer vision researchers

from Berkeley and Stanford’, max=20) OBJ1=CLASSIFY(image=IMAGE, object=OBJ0, categories=LIST0)

IMAGE0=TAG(image=IMAGE, object=OBJ1)

FINAL_RESULT=RESULT(var=IMAGE0)

Tag these famous computer vision researchers from Berkeley and Stanford

Replace Jitendra with George Clooney wearing sunglasses and Fei-Fei with Aishwarya Rai wearing sunglasses

OBJ0=FACEDET(image=IMAGE)

OBJ1=SELECT(image=IMAGE,object=OBJ0,query='Fei-Fei',category=None) IMAGE0=REPLACE(image=IMAGE,object=OBJ1,prompt=‘Aishwarya Rai wearing sunglasses') OBJ2=SELECT(image=IMAGE0,object=OBJ0,query='Jitendra',category=None) IMAGE1=REPLACE(image=IMAGE0,object=OBJ2,prompt='George Clooney wearing sunglasses’)

FINAL_RESULT=RESULT(var=IMAGE1)

Visual Rationales

Sources of Error

Recent Works

Image Generation

Cho et al. Visual Programming for Text-to-Image Generation and Evaluation. NeurIPS 2023

Cho et al. Visual Programming for Text-to-Image Generation and Evaluation. NeurIPS 2023

Image Generation

Simplifying Prompts

Hsieh et al. Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models.  Arxiv 2023

Program of Thoughts

Chen, Wenhu et al. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. TMLR 2023

ViperGPT

Suris et al. ViperGPT: Visual Inference via Python Execution for Reasoning. ICCV 2023

ReAct

Shunyu et al. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023

ReAct

Shunyu et al. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023

Multimodal ReAct

Yang et al. MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action. Arxiv 2023

Gao et al. AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn. Arxiv 2023

We’ve barely scratched the surface …

• Error detection and correction

​

• Multi-step and multimodal program generation

​

• Scaling the number of tasks and modules

​

• Image and video editing/generation with VisProg

​

• Learning VisProg modules & programs from training data / feedback

Training language instruction following robots for the real world

w/o RL

w/o real-world training

On-policy RL

• Act using the current policy to collect episodes with rewards
• Train on those episodes to update policy to maximize rewards

​

​

On-policy RL -> Behavior Cloning

• Act using an expert policy to collect episodes
• Train on those episodes to imitate expert actions

On-policy RL -> Behavior Cloning

• Act using an expert policy to collect episodes
• Train on those episodes to imitate expert actions

Supervised Learning!

On-policy RL -> Behavior Cloning

• Act using an expert policy to collect episodes
• Train on those episodes to imitate expert actions

Large and Diverse training data? Expert?

Large and Diverse Training Data for BC

ProcTHOR (NeurIPS 2022 Best Paper)

Large and Diverse Training Data for BC

Objaverse (CVPR 2023)

Large and Diverse Training Data for BC

Generate virtually infinite home simulation environments.

Populate these environments with diverse 3D assets

Experts for BC

Shortest-path plans using privileged information such as object locations, floor plan, and reachable locations.

Experts for BC

“Fetch a bowl”

Training Data

Training Data

Model

Goal-conditioned visual encoder

Action Decoder

Learned Behaviors

Emergence of exploration from shortest-path plans!

Task-specific behaviors

Recent advances in procedural generation of simulation environments, strong visual encoders, and the ubiquitous Transformers have allowed us to develop Robotic systems using good ol’ supervised learning in simulators.

​

Combined with widely available low cost hardware (Stretch, Mobile ALOHA) in the near future -

​

​

Significantly reduced barrier to training real-world robots

Thank You!

http://tanmaygupta.info/

https://prior.allenai.org/