2 of 26

Timeline

Neural Modular Network(NMN), which is our main focus:

paper 1. Deep Compositional Question Answering with Neural Module Networks

NMN: Improvement/variant (paper 2,3,4):

paper 2. Inferring and Executing Programs for Visual Reasoning

paper 3. Learning to Reason: End-to-End Module Networks for Visual Question Answering

paper 4. FiLM: Visual Reasoning with a General Conditioning Layer

Conclusion

3 of 26

Visual Question Answering, VQA

What is VQA？

VQA is answering questions about image (using AI).

example: ChatGPT

4 of 26

What is VQA?

With the development of AI, the VQA form task was first proposed in 2015.

It can be defined as a 3-tuple(w,x,y), where:

w is a natural-language question
x is an image
y is an answer

We want: A model encodes a predictive distribution p(y |w,x;\theta)

5 of 26

Two types of models for VQA

Monolithic Network : using CNN, RNN construct an fixed network
Neural Modular Network(NMN): sets of modules are dynamically assembled into a NN, based on a specific problem.

6 of 26

Deep Compositional Question Answering with Neural Module Networks

https://arxiv.org/pdf/1511.02799.pdf

7 of 26

NMN:motivation

There is no single best networks for all tasks. (type1:Monolithic Network)

Though different networks are used for different purposes, it is commonplace to initialize systems for many of vision tasks with a prefix of a network trained for classification

8 of 26

Main ideas of NMN

Semantic parser
Combination of modules for specific tasks
Usage of an LSTM to read the question
Combine the output of NMN and LSTM

NMN structure

9 of 26

NMN: modules

TYPE[INSTANCE](ARG1)

TYPE: high-level module(Attention, Classification)

INSTANCE:Particular instance of the model under consideration

10 of 26

Different type of modules

attention module

image → convolve position with a weight vector → heatmap or unnormalized attention

Re-attention module

Shift the attention region

MLP + non-linear activation

Mapping the attention to another.

11 of 26

Different type of modules

combine[except]: stack of active and inactive attention region

classify[where]: Mapping the image and attention to an distribution

measurement module

ex: evaluate the existance

12 of 26

From Strings to Network:Parsing

Parsing the question with a Stanford Parser to obtain a universal dependency representation.(understanding the question)

Grammatical relationships between parts of a sentence

ex: what color is the truck -> color(truck)

Basic lemmatization, reduce the sparsity

ex: “kites” to “kite”

“were” to “be”

13 of 26

From Strings to Network: Layout

Based on specific task : symbolic representation → modular network

What color is his tie?

-> color(tie) #parsing

root leaf

How to build a predictive model based on that understanding?

Mapping:

All leaves -> attend modules
All internal nodes -> re-attend or combine
root nodes -> measure or classify

Layouts, attentions, and answers are real predictions made by the model

assignments of modules

14 of 26

From Strings to Network: structure summary statistic

Structure summary statistics for neural module networks used in the paper NMN.

ex:

a network with depth 4 and size 5

15 of 26

NMN: Prediction

LTSM + NMN, jointly train

LSTM : model underlying syntactic rules

and capture semantic regularities

ex:

What is flying,

What are flying

parser: is&are →be : what(fly)

answer: “kite” & “kites”

LSTM & NMN module both make prediction over the answer distribution. The final prediction is the mean of these two prob. distribution.

16 of 26

NMN: experiments

Experiments show that NMN outperforme the previous works

17 of 26

NMN: examples of output

18 of 26

NMN: limitation

The NMN model takes a question as input and parses it into a combination of different modules using NLP rules. This combination of modules is often referred to as a "program".�However, for complex questions, the effectiveness of rule-based parsing is limited.

�E.g. Are these people most likely experiencing a work day?

praser -> be(people, likely), but desired analysis is is(people, work).

19 of 26

NMN: Improvement/variant (paper 2,3,4)

NMN-Based Models:

Inferring and Executing Programs for Visual Reasoning
Learning to Reason: End-to-End Module Networks for Visual Question Answering

Feature Modulation Approaches Model:

FiLM: Visual Reasoning with a General Conditioning Layer

20 of 26

Inferring and Executing Programs for Visual Reasoning

https://arxiv.org/pdf/1705.03633.pdf

22 of 26

Learning to Reason: End-to-End Module Networks for Visual Question Answering

https://arxiv.org/pdf/1704.05526.pdf

24 of 26

FiLM: Visual Reasoning with a General Conditioning Layer

https://arxiv.org/pdf/1709.07871.pdf

25 of 26

FiLM learns to adaptively influence the output of a neural network by applying an affine transformation to the network’s intermediate features(F_i,c), based on some input.

learns directly from visual and textual input!

26 of 26

Conclusions

Parser Rules vs. Deep Network Learning

Modularity vs. Feature Modulation

Dynamic Network Construction:

Film don’t have the need for specific modular designs or program generation logic.

1 of 26