1 of 25

VisToT: Vision-Augmented Table-to-Text Generation

https://vl2g.github.io/projects/vistot/

AI for India @ AI-ML Systems’23

Prajwal Gatti, Anand Mishra, Manish Gupta, Mithun Das Gupta

2 of 25

Can we combine information in tables and images to generate rich text descriptions?

2

3 of 25

Table-to-Text Generation

3

Conventional Table-to-Text Method:

“Lough Leane is a 4700 acre estate in Killarney, County Kerry, Ireland.”

[Lebret et al. EMNLP’16; Novikova et al. SIGDIAL’17; Parikh et al. EMNLP’20]

4 of 25

Vision-Augmented Table-to-Text Generation

4

Conventional Table-to-Text Method:

“Lough Leane is a 4700 acre estate in Killarney, County Kerry, Ireland.”

​

​

Vision-augmented Table-to-Text (Our work):

“Lough Leane is a large lake in Killarney, County Kerry, Ireland.”

5 of 25

Vision-Augmented Table-to-Text Generation

5

Conventional Table-to-Text Method:

“Lough Leane is a 4700 acre estate in Killarney, County Kerry, Ireland.”

​

​

Vision-augmented Table-to-Text (Our work):

“Lough Leane is a large lake in Killarney, County Kerry, Ireland.”

​

Ground Truth:

“Lough Leane is the largest of the three lakes in Killarney, County Kerry, Ireland.”

6 of 25

Vision-Augmented Table-to-Text Generation

6

Given a table T and an image I about an entity

Generate a text summary S describing the entity using �T and I as the source context.

7 of 25

We introduce WikiLandmarks

A new dataset containing tables and images for� 73K world landmarks

7

8 of 25

Image

Table

Text summary

8

Name

Amitabha Drukpa

Country

Nepal

Location

Kathmandu

Dedicated To

Amitabha

“Amitabha Monastery is a Tibetan Buddhist Monastery

in Nepal”

8

8

9 of 25

Image

Table

Text summary

Name

Michigan Stadium

Location

1201 South Main Street

Ann Arbor, Michigan

Owner

University of Michigan

Nickname

The Big House

“Michigan Stadium, nicknamed The Big House, is the football stadium for the University of Michigan in

Ann Arbor, Michigan”

9

9

10 of 25

Image

Table

Text summary

Name

Niesen

Elevation

2,632 m

Prominence

407 m

Location

Canton of Bern, Switzerland

Parent Range

Bernese Alps

“The Niesen is a mountain peak of the Bernese Alps in the Canton of Bern, Switzerland”.

10

10

11 of 25

11

Colosseum, Italy

Castelão, Brazil

Mysore Palace, India

St. Louis Cathedral, USA

WikiLandmarks is geodiverse!

Belukha Mountain, Russia

12 of 25

12

  • 73K+ unique world landmarks�
  • Curated from Wikipedia and Google Landmarks Dataset (Weyand et al. 2020)�
  • ~10 natural images available per landmark�
  • Includes churches, mountains peaks, castles, historical sites, statues, and more.

Dataset Statistics

13 of 25

We introduce VT3

A multimodal transformer for the VisToT problem

13

14 of 25

VT3: Visual-Tabular Data-to-Text Transformer

Key

Value

Name

Lough Leane

Location

Killarney, County Kerry

Coordinates

58°2’30’’N 9°33’0’’W

Basin countries

Ireland

Surface Area

4,700 acres

Islands

Innisfallen

Table

Image

BART

Encoder

[NAME] Lough Leane [Location] Killar…

Swin

Transformer

…

Tokenize

BART

Decoder

“Lough Leane is a large lake in Killarney, …”

“[bos] Lough Leane is a large lake ”

Generated Sentence

14

15 of 25

15

VT3: Visual-Tabular Data-to-Text Transformer

We introduce three pre-training objectives:

16 of 25

16

Benchmarking on WikiLandmarks

VT3 outperforms both SoTA table-to-text and image captioning methods

17 of 25

17

Effect of Visual Encoders

Swin yields superior results than other ViTs as well as an object-centric vision encoder (FRCNN)

18 of 25

Capitol Park Historic District

Capital Park Detroit MI

Capitol Park, from the north

Detroit, Michigan, U.S.

46.64611°’N 7.6525°’W

1877

Albert Kahn Associates et al.

Italianate, Romanesque Revival

March 18, 1999

Michigan State Historic Site

Name

Image

Caption

Location

Coordinates

Built

Architect

Architecture

Added

Designated

18

Result

VT3 (w/o vision):

Capitol Park is a park in Downtown Detroit, Michigan, United States.

�VT3:

Capitol Park Historic District is a commercial historic district located in downtown Detroit, Michigan. ��Ground Truth:

The Capitol Park Historic District is a historic district located in downtown Detroit, Michigan.

19 of 25

19

Thank You

Visit our project page for Code and Dataset

https://vl2g.github.io/projects/vistot/

Supported by Microsoft Academic Partnership Grant 2022

20 of 25

Appendix

20

21 of 25

Appendix

21

22 of 25

Appendix

22

23 of 25

Appendix

23

24 of 25

Appendix

24

25 of 25

Appendix

25