1 of 50

On Text Localization in End-to-End OCR-Free Document Understanding Transformer without Text Localization Supervision

Geewook Kim^{1 * †}, Shuhei Yokoo^{2 *}, Sukmin Seo¹, Atsuki Osanai², Yamato Okamoto² and Youngmin Baek¹^�

^{* Equal Contribution † gwkim.rsrch@gmail.com}

2 of 50

Agenda

Introduction: � Visual Document Understanding (VDU) &� Document Understanding Transformer (Donut 🍩)
Research Motivation & Problems to Solve
Proposal: �A Bag of Tricks for Text Localization in Donut (Free Donut)
Experiments and Analysis
Conclusion

3 of 50

Introduction: VDU & Donut 🍩

Introduction Part is from Donut ECCV-22 Slide

4 of 50

Visual Document Understanding (VDU)

VDU aims to extract useful information from the document image. For example,

VDU Model

Useful Information

Introduction Part is from Donut ECCV-22 Slide

5 of 50

Example 1: Document Classification

A document classifier aims to extract a category information from the image.

VDU Model

{ "class": "receipt" }

Introduction Part is from Donut ECCV-22 Slide

6 of 50

Example 2: Document Parsing

For another example, a document parser aims to get a data in a format,

such as, JSON or XML, that contains full information.

VDU Model

{ "menu": [

{

"nm": "3002-Kyoto Choco Mochi",

"unitprice": "14.000",

"cnt": "x2",

"price": "28.000"

}, … }

Introduction Part is from Donut ECCV-22 Slide

7 of 50

Conventional VDU Model

Here, we show a representative pipeline of visual document parsing.

Input

Output

{ "items": [

{

"name": "3002-Kyoto Choco Mochi",

"count": 2,

"priceInfo": {

"unitPrice": 14000,

"price": 28000

}

}, {

"name": "1001 - Choco Bun",

"count": 1,

"priceInfo": {

"unitPrice": 22000

"price": 22000

}

}, ...

],

"total": [ {

"menuqty_cnt": 4,

"total_price": 50000

}

]

}

≈

8 of 50

Conventional VDU Model

Most conventional VDU methods share a similar pipeline.

…

Input

Output

{ "items": [

{

"name": "3002-Kyoto Choco Mochi",

"count": 2,

"priceInfo": {

"unitPrice": 14000,

"price": 28000

}

}, {

"name": "1001 - Choco Bun",

"count": 1,

"priceInfo": {

"unitPrice": 22000

"price": 22000

}

}, ...

],

"total": [ {

"menuqty_cnt": 4,

"total_price": 50000

}

]

}

≈

9 of 50

Conventional VDU Model

First, a text detector finds all text boxes.

Detection!

10 of 50

Conventional VDU Model

And then, a text recognizer reads all texts in the extracted boxes.

Detection! Recognition!

{ "words": [ {

"id": 1,

"bbox":[[360,2048],...,[355,2127]],

"text": "3002-Kyoto"

}, {

"id": 2,

"bbox":[[801,2074],...,[801,2139]],

"text": "Choco"

}, {

"id": 3,

"bbox":[[1035,2074],...,[1035,2147]],

"text": "Mochi"

}, {

"id": 4,

"bbox":[[761,2172],...,[761,2253]],

"text": "14.000"

}, …, {

"id": 22,

"bbox":[[1573,3030],...,[1571,3126]],

"text": "50.000"

}

]

}

11 of 50

Conventional VDU Model

This two parts are also called as Optical Character Recognition (OCR).

Detection! Recognition!

{ "words": [ {

"id": 1,

"bbox":[[360,2048],...,[355,2127]],

"text": "3002-Kyoto"

}, {

"id": 2,

"bbox":[[801,2074],...,[801,2139]],

"text": "Choco"

}, {

"id": 3,

"bbox":[[1035,2074],...,[1035,2147]],

"text": "Mochi"

}, {

"id": 4,

"bbox":[[761,2172],...,[761,2253]],

"text": "14.000"

}, …, {

"id": 22,

"bbox":[[1573,3030],...,[1571,3126]],

"text": "50.000"

}

]

}

OCR

12 of 50

Conventional VDU Model

Finally, the OCR results are fed to a following module �to get full information of the document.

{ "items": [

{

"name": "3002-Kyoto Choco Mochi",

"count": 2,

"priceInfo": {

"unitPrice": 14000,

"price": 28000

}

}, {

"name": "1001 - Choco Bun",

"count": 1,

"priceInfo": {

"unitPrice": 22000

"price": 22000

}

}, ...

],

"total": [ {

"menuqty_cnt": 4,

"total_price": 50000

}

]

}

≈

{ "words": [ {

"id": 1,

"bbox":[[360,2048],...,[355,2127]],

"text": "3002-Kyoto"

}, {

"id": 2,

"bbox":[[801,2074],...,[801,2139]],

"text": "Choco"

}, {

"id": 3,

"bbox":[[1035,2074],...,[1035,2147]],

"text": "Mochi"

}, {

"id": 4,

"bbox":[[761,2172],...,[761,2253]],

"text": "14.000"

}, …, {

"id": 22,

"bbox":[[1573,3030],...,[1571,3126]],

"text": "50.000"

}

]

}

Detection! Recognition! Parsing!

OCR

13 of 50

Conventional VDU Model:

Details of the Parsing Stage

For example, in most methods, BIO-tags are predicted by a backbone.

… 3002-Kyoto Choco Mochi 14, 000 …

B-name I-name I-name B-price I-price

Transformer Backbone�(BERT, LayoutLM, …)

(Off-the-shelf)�OCR Engine

14 of 50

Conventional VDU Model:

Details of the Parsing Stage

Then, the tag sequence is converted into a final data format (e.g., JSON).

… 3002-Kyoto Choco Mochi 14, 000 …

B-name I-name I-name B-price I-price

Transformer Backbone�(BERT, LayoutLM, …)

(Off-the-shelf)�OCR Engine

{ "menu": [

{

"nm": "3002-Kyoto Choco Mochi",

"unitprice": "14.000",

"cnt": "x2",

"price": "28.000"

}, … }

15 of 50

Conventional VDU Model: Overview

OCR induces several negative ramifications to the subsequent processes.

Input Image

(Off-the-shelf)�OCR Engine

Backbone

(BERT-like)

BIO-Tags / Answer Token Span / etc

Output

high computational costs
inflexibility of OCR on languages or document type
OCR error propagation

16 of 50

New Approach: OCR-free Donut

Donut 🍩�(End-to-end Model)

Token Sequence

Output

Input Image

17 of 50

Pipeline Comparison

Donut 🍩�(End-to-end Model)

Token Sequence

Output

Input Image

(Off-the-shelf)�OCR Engine

Backbone

(BERT-like)

BIO-Tags / Answer Token Span / etc

Output

18 of 50

Donut Model Details

This is the overview of Donut.

<vqa><question>what is the price

of choco mochi?</question><answer>

Converted JSON

transformer encoder

Input Image and Prompt

transformer decoder

Donut 🍩

<class>receipt</class>

</classification>

14,000</answer></vqa>

<item><name>3002-Kyoto Choco Mochi</name>・・・ </parsing>

{ "items": [{"name": "3002-Kyoto Choco Mochi",

"count": 2,

"unitprice": 14000, …}], … }

Output Sequence

{ "class":"receipt" }

{ "question": "what is the price of choco mochi?",

"answer": "14,000" }

19 of 50

Donut Model Details

The visual encoder maps the input image into a set of embeddings.

<vqa><question>what is the price

of choco mochi?</question><answer>

Converted JSON

transformer encoder

Input Image and Prompt

transformer decoder

Donut 🍩

<class>receipt</class>

</classification>

14,000</answer></vqa>

<item><name>3002-Kyoto Choco Mochi</name>・・・ </parsing>

{ "items": [{"name": "3002-Kyoto Choco Mochi",

"count": 2,

"unitprice": 14000, …}], … }

Output Sequence

{ "class":"receipt" }

{ "question": "what is the price of choco mochi?",

"answer": "14,000" }

20 of 50

Donut Model Details

The textual decoder processes the image embeddings and prompt tokens.

<vqa><question>what is the price

of choco mochi?</question><answer>

Converted JSON

transformer encoder

Input Image and Prompt

transformer decoder

Donut 🍩

<class>receipt</class>

</classification>

14,000</answer></vqa>

<item><name>3002-Kyoto Choco Mochi</name>・・・ </parsing>

{ "items": [{"name": "3002-Kyoto Choco Mochi",

"count": 2,

"unitprice": 14000, …}], … }

Output Sequence

{ "class":"receipt" }

{ "question": "what is the price of choco mochi?",

"answer": "14,000" }

21 of 50

Donut Model Details

Then, the decoder outputs token sequences,

<vqa><question>what is the price

of choco mochi?</question><answer>

Converted JSON

transformer encoder

Input Image and Prompt

transformer decoder

Donut 🍩

<class>receipt</class>

</classification>

14,000</answer></vqa>

<item><name>3002-Kyoto Choco Mochi</name>・・・ </parsing>

{ "items": [{"name": "3002-Kyoto Choco Mochi",

"count": 2,

"unitprice": 14000, …}], … }

Output Sequence

{ "class":"receipt" }

{ "question": "what is the price of choco mochi?",

"answer": "14,000" }

22 of 50

Donut Model Details

that can be converted into a desired data format, such as, a JSON format.

<vqa><question>what is the price

of choco mochi?</question><answer>

Converted JSON

transformer encoder

Input Image and Prompt

transformer decoder

Donut 🍩

<class>receipt</class>

</classification>

14,000</answer></vqa>

<item><name>3002-Kyoto Choco Mochi</name>・・・ </parsing>

{ "items": [{"name": "3002-Kyoto Choco Mochi",

"count": 2,

"unitprice": 14000, …}], … }

Output Sequence

{ "class":"receipt" }

{ "question": "what is the price of choco mochi?",

"answer": "14,000" }

23 of 50

Donut Model Details

Swin Transformer and BART are used as an encoder and decoder, respectively.�More details can also be found in the manuscript.

<vqa><question>what is the price

of choco mochi?</question><answer>

Converted JSON

transformer encoder

Input Image and Prompt

transformer decoder

Donut 🍩

<class>receipt</class>

</classification>

14,000</answer></vqa>

<item><name>3002-Kyoto Choco Mochi</name>・・・ </parsing>

{ "items": [{"name": "3002-Kyoto Choco Mochi",

"count": 2,

"unitprice": 14000, …}], … }

Output Sequence

{ "class":"receipt" }

{ "question": "what is the price of choco mochi?",

"answer": "14,000" }

24 of 50

Donut showed promising results, and…

Document parsing benchmark scores from the Donut paper (ECCV 22).

Table from https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136880493.pdf

25 of 50

Donut has many application, but…

Screenshot from https://huggingface.co/spaces/naver-clova-ix/donut-base-finetuned-cord-v2

26 of 50

Research Motivation & Problems to Solve

27 of 50

How can we obtain the bounding boxes?

Screenshot from https://clova.ai/ocr/en

Our Question

28 of 50

Option 1: Should we revert to using OCR + Parser?

29 of 50

Option 2: Should we incorporate a Pix2Seq-like approach into Donut?

Images are from https://arxiv.org/abs/2109.10852

30 of 50

Research Question: Can we entirely eliminate the need for the localization annotation?

31 of 50

Intuition: Using the cross-attention map as a heat map!

We decided to explore this possibility.

Images are from OCR-Free Document Understanding Transformers (Donut), ECCV-22.

32 of 50

However, a straightforward visualization produces a blurred heatmap.

33 of 50

Proposal:� A bag of tricks for text localization in Donut (Free Donut)

34 of 50

Our idea is derived from three separate analyses:

How can we merge multi-head cross-attention maps into one singular map?
Does the tokenization level impact the maps?
Can we gain any advantages from using an unsupervised text blob detector?

35 of 50

Q: How to merge heatmaps? A: Use Variance.

36 of 50

Q: Do tokenization affects the maps? A: Yes.

37 of 50

Q: Can we be benefit from a (weak) blob detector? A: Yes.

Otsu’s binarization + Watershed labeling algorithm

38 of 50

In general, many non-text blobs are included.

39 of 50

However, a simple IoU-based blob matching is robust to non-text blobs.

40 of 50

As a result, here are some examples from the CORD test set.

41 of 50

Experiments and Analysis

42 of 50

We test two datasets: CORD and JP Business Card.

43 of 50

CORD.

JP Business Card.

44 of 50

Word-level Eval.

Field-level Eval.

45 of 50

The conventional method based on OCR still appears to be superior in text localization.

CORD.

JP Business Card.

46 of 50

The conventional method based on OCR still appears to be superior in text localization.
However, when evaluating models in an end-to-end setting, our proposed model demonstrates promising outcomes. This can be attributed to Donut's high accuracy in text recognition and parsing.

CORD.

JP Business Card.

47 of 50

Lastly, we analyzed the contribution of each component within Free Donut.

48 of 50

Lastly, we analyzed the contribution of each component within Free Donut.

Interestingly, character-level tokenization enhances text localization but reduces the overall end-to-end score.
We hypothesize that this decrease may be a result of the compromised language modeling capability of the character-level model.

49 of 50

Conclusion

We have introduced a series of techniques to improve localization performance in OCR-Free Document Understanding Transformers.
We believe our work contributes towards the advancement of effective OCR-free document understanding models.

50 of 50

If you have any questions please feel free to contact me :)�Thank you!

Thank you!

^{Contact: gwkim.rsrch@gmail.com}