Text matters?!

Statistical analysis on COCO-Text dataset

overview

• Motivation
• Experiment Design
• Methodology: Ablation Study
• Data partition
• Comprehensive Experiment Results
• Case Study on False Correlation
• Problem and challenge
• Q & A

Motivation

• You would think in-image text shall tell you a lot about the image semantics...
• Civil instead of wild setting
• Street name, billboards, stop sign -> salience highlight of the image
• We want to know how or how much in-image texts matter to image semantics, particularly how are they interpreted by captioning networks
• Perhaps we can use the more sophisticated OCR techs to improve captioning performance

Experiment Design (Methodology)

• Apply ablation on selected parts of the image.
• Compare captions of ablated images with the original’s
• Ablation methods:
• gaussian, black-out, median, background black-out, background median.

​

​

Experimental Design (Data Partition)

• Split the COCO Dataset into following groups:
• Group A: images without text instances annotated.
• Group B: images with only irrelevant text instances.
• Group C: images with some relevant text instances.�
• We called a text instance “relevant” if it appears in captions from the training set.

​

​

Captions:

• a stop sign at an intersection, with "hammer time" printed on it
• a stop sign with graffiti reading "hammer time."
• street sign on post with graffiti near green shrubs.
• a stop sign where someone wrote "hammer time."
• a stop sign at a corner with the words hammer time below stop.

Captions:

• a boy is swinging at a ball during a baseball game.
• a boy that is hitting a ball with a baseball bat.
• a person with a bat and helmet hits a ball.
• a boy in a yellow and white uniform hitting a baseball
• a young baseball player is hitting the ball.

Experiment Design (Group B)

• Contains all images with only irrelevant text instances.
• Ablation Methods:
• Gaussian
• Black-out
• Median
• Metrics:
• Jaccard Index
• Tree LSTM

Experiment Design (Group C)

• Contains images with relevant text instance(s)
• Ablation Methods:
• Gaussian
• Black-out
• Median
• Metrics:
• Jaccard Index
• Tree LSTM

Experiment Design (Group B, C)

• Hypothesis�
• Group C (img with rel. text) should experience more caption change than Group B (img without rel. text)�
• Black-out should cause most disrupted changes, while median filter should cause the least.�

​

Experiment Result (Group B)

• Disruptive Degree: Blackout > Gaussian > Median
• Most doesn’t change

Experiment Result (Group C)

• Disruptive Degree: Blackout > Gaussian > Median
• Most doesn’t change

Experiment Result (Group B + C)

• Group C experiences more changes than Group B
• Difference isn’t very significant.

Table: Data of median ablation on both Group B and Group C

Table: Data of blackout ablation on both Group B and Group C

Experiment Result (Group B + C)

• Jaccard indexes distributions are discrete.
• Tree LSTM similarity scores are more continuous.
• Eg. Data for Group B with gaussian ablation. �Normalized Jaccard indexes (left) v.s. Tree LSTM scores of (right).

​

Experiment Result (Group B + C)

• Distribution of Tree LSTM similarity scores is highly similar to that of jaccard score.
• Eg. Data for Group C with median ablation�Norm. Jaccard(left); Norm. Tree LSTM scores (middle); Diff. (right)

Experimental Design (Group A)

​

• Contains all images without text instances
• Ablation methods:
• Background blackout
• Background median
• Hypothesis:
• Background should contribute to Object-scene context; as a result, removing the scene completely will create high caption-semantic disruption.
• Semantic Distance Metrics:
• Jaccard Index
• Tree-LSTM Similarity Score
• Bleu Score Difference

Experiment Result (Group A)

Interesting examples

False Correlation Experiment (From this week)

• In previous anecdotal experiments, we noticed that with the background blackout ablation, the captioning network will produce stable word pair even with minimal visual cues, eg. “toilet and sink”, “red rose”.�
• Maybe the RNN does not see the images at all (or extract very limited information from image contexts)?�
• Hint at a way to improve captioning accuracy.

Hypothesis

�

​

• Word context dominates image context during caption generation.

​

procedures

1. Calculate word-word conditional probabilities from training captions.
2. Categorical studies on how ablation affects appearance of the highly probable words.
1. We choose “toilet” category.
2. Highest probable words are “BATHROOM”, “SINK”, “WHITE”, “SITTING”, and “NEXT”
3. Extreme ablation: Only leaves a toilet.
1. Interesting both experimentally and aesthetically.

Results

• The co-appearing words generally have high priors
• Over 10%, with mean probability over all word-pairs being 0.0003%.
• Captions highly follow prior probability distribution.
• Some examples:
• Toilet->[BATHROOM, SINK, WHITE, SITTING, NEXT]
• Rose -> [VASE, PINK, RED, TABLE, WHITE]
• Clock ->[TOWER, BUILDING, LARGE, TOP, TALL]

​

• Caveat: The ones we spotted are not the highest in ranking.

Results -- Ablation

• Total # of ‘toilet’ images: 2318������
• Decrease in col.3 might be due to random guessing of the network: Since some images after ablation is hardly recognizable.

Results -- Toilet only

• Median width = 71, total number of image = 1135�����
• The size of the vocabulary of the original + ablated captions is merely 374, compared to the entire training vocab of 22644.

Results -- Toilet only

​

������

​

• Saturated conditional probabilities due to shrunk vocabulary size

Further Research ideas

• Adversarial performance metric for captioning network�
• Separately train RNN on richer corpus to gain a stronger language model�
• Try to use simple mechanisms to reproduce neuraltalk benchmark
• Start with a word, then discard image context altogether and rely solely on word context.

Q & A