1 of 10

1 / 10

Evolving Edison Papers

Week 7

Partha

Thomas A. Edison Papers, Rutgers University

​

July 9, 2026

2 of 10

Where Today’s Work Fits

2 / 10

IIIF Ingest

(Phase 1 – today)

OCR / HTR /

VLM / Hybrid draft

Structured JSON output

Human review

Clean transcription

→ DB

edison-papers: piloted which transcription method works best (Stages 2–5).

edison-pipeline (this repo): builds the production ingest that feeds Stage 2 at scale — the subject of this talk.

3 of 10

Recap: Transcription Method Benchmark

3 / 10

From edison-papers — 8 methods, 26 documents, Volume 9 letters/telegrams.

Method

Type

Avg CER

Avg WER

hybrid-paddle-qwen

Hybrid

0.199

0.270

hybrid-doctr-qwen

Hybrid

0.219

0.279

qwen3.6-27b

VLM

0.225

0.285

paddleocr-vl

OCR-VL

0.244

0.320

mistral-small-3.2

VLM

0.452

0.503

doctr

OCR

0.516

0.823

Winner

hybrid-paddle-qwen (PaddleOCR-VL layout → Qwen3.6 refinement) is the best overall method — this is the model slated for the transcription stage.

4 of 10

Recap: Where Each Method Struggles

4 / 10

docTR (local, cheap) is weak everywhere, especially crossouts (0.69) and handwriting (0.53).

hybrid-paddle-qwen wins or ties on 5/7 challenge types — typed text (0.13), long documents (0.05), handwriting (0.22).

5 of 10

Phase 1 Scope: Extraction Ingest

5 / 10

Goal: given a CSV of Document IDs, fetch each document’s IIIF manifest, download its page images, and record metadata + image locations in a SQLite extraction table that the transcription stage will read from.

In scope

IIIF manifest fetch (v2 & v3)

Metadata parsing (title, author, date, place, recipient, type, . . . )

Page image download → bucket storage Idempotent SQLite extraction table

Out of scope (this phase) Transcription (OCR/HTR/VLM) “Create new document” workflow

“Add images to existing document” workflow Provisioning a real S3/GCS bucket

6 of 10

How It Works

Read document_ids from an input CSV.

Fetch the IIIF Presentation manifest for each document from

edisondigital.rutgers.edu.

Parse metadata (title, author, date, place, recipient, mentioned entities, document type,

. . . ) and page/image URLs — handles both IIIF v2 (sequences/canvases/images) and v3 (items/items/items) shapes.

Download each page image into bucket storage (fsspec: local disk today, s3:// is a config change, not a code change) and upsert the documents/pages rows in SQLite.

6 / 10

1

2

3

4

Idempotent by design

Re-running the same command skips documents already marked fetched; a single failed page never fails the whole document.

7 of 10

Live Run

7 / 10

A real ingest of document CL207AAA against the live edisondigital.rutgers.edu IIIF endpoint.

8 of 10

Live Run: The Extraction Table

8 / 10

Metadata lands in documents; per-page bucket locations land in pages — exactly what the transcription stage will read.

9 of 10

9 / 10

Live Run: Bucket Storage

10 of 10

Status & Next Step

10 / 10

Phase 1 (extraction ingest) is implemented and tested: manifest parsing (v2/v3), bucket round-trips, and upsert idempotency all covered by pytest.

Confirmed working end-to-end against the live Edison Digital archive.

Next: feed the extraction table’s images into the hybrid-paddle-qwen transcription stage validated in edison-papers (CER 0.199), then route drafts into the human-review workbench.

​

Questions?