From digital content to users participation: a journey with IIIF (and AI)
The OHOS project
Jean-Philippe Moreux
Cooperation department/Gallica scientific advisor
© art Buck/Flickr
he general picture
T
Digital heritage collections
and reuse; sharing; public participation
GALLICA�Digital Repositories�Catalogs
TNA�Digital Repositories�Catalogs
BL
Heritage �Holdings
ow
N
For quite a while, our users extract and reuse heritage digital content (CHO, cultural heritage object) with APIs, curated datasets, dumps…
APIs, dumps
GALLICA�Digital Repositories�Catalogs
TNA�Digital Repositories�Catalogs
BL
Heritage �Holdings
ow
N
Some of them also act as CHO producers:
which means our content becomes alive (enriched, transcribed, commented…)
APIs, dumps
GALLICA�Digital Repositories�Catalogs
TNA�Digital Repositories�Catalogs
BL
Heritage �Holdings
ow
N
…which leads us to few challenges:
Technical challenges:
APIs, dumps
GALLICA�Digital Repositories�Catalogs
TNA�Digital Repositories�Catalogs
IIF?
I
Can IIIF meet these challenges? Some �of them?
All institutions can share CHOs with IIIF.
CHOs can be “enriched” by end users.
End users communities can share their data.
Heritage institutions may wish to benefit from this “enrichment” work (orange arrow).
https://iiif.io
International Image �Interoperability �Framework
DH Team #3
Heritage �Holdings
H
(Is it painful to implement?)
ow does it works?
H
(It’s not!)
https://gallica.bnf.fr/iiif/ark:/12148/bpt6k258926j/f5/3315,3922,707,470/1000,/0/native.jpg
�https://api.bnf.fr/fr/api-iiif-de-recuperation-des-images-de-gallica
Image API: access to the images
https://iiif.io/api/image/3.0
ow does it works?
Gallica IIIF �image server
H
(It’s not!)
https://gallica.bnf.fr/iiif/ark:/12148/bpt6k258926j/manifest.json
Presentation API: access to the digital document
https://iiif.io/api/presentation/3.0/
ow does it works?
Gallica IIIF �manifests server
IIIF toolbox
IIIF data model
Where are �the illustrations?
W
x,y,w,h
Canvas,�Region
hat kind of CHOs?
Manuscripts
Newspapers
Magazines
Audio, video: APIs version 3
3D: work in progress
t1, t2
Where are �the tracks ?
What is the CHO content? (transcription, translation, tagging, comment…). Annotations are embedded or referenced (as static files or delivered by a server).
Tagging: Hair�x3,y3,w3,h3
Tagging: Person, Woman�x1,y1,w1,h1
Vogue, 1936
W
hat kind of CHOs?
Supplementing: Caption text �x2,y2,w2,h2
List of �annotations
he services: real life applications
T
Universal access to images and documents
Annotations
Collections of documents
Dissemination of textual resources
Image info�(Image API)
A
ccess to images
gallicapix.bnf.fr
https://iiif.wellcomecollection.org/image/L0009291.jpg/1,1,1594,1256/1000,/0/default.jpg
https://gallica.bnf.fr/iiif/ark:/12148/btv1b85945334/f19/1,1,2835,4081/�1000,/90/native.jpg
https://github.com/altomator/Image_Retrieval
Multi-collection content-based �image retrieval use case
Manifests�(Presentation API)
A
ccess to documents
Mirador
Drag & Drop�feature!
W3C annotations�(Presentation API)
A
nnotations: use cases
https://training.iiif.io/iiif-online-workshop/day-four/
Human or machine produced annotations: �IIIF from end to end!
Gallica repo �& APIs
GallicaPix�database (BaseX)
Deep learning processing�(IBM Watson, Google Cloud Vision, Google OCR, Yolo, �locally trained models)
GallicaPix�web app
IIIF annotations server
Gallica
W3C annotations�(Presentation API)
A
nnotations: use cases
(British Library)
Crowdsourcing �heritage collections (e.g. TNA or BL projects)
BL �repo
Annotations�database
Crowdsourcing�(Zooniverse)
Internal �uses
Annotations server
QA
https://docs.libcrowds.com/data/api/
Collections
(Presentation API)
Vogue French edition title in Mirador
C
ollections
http://codh.rois.ac.jp/icp/index.html.en
Curation is a IIIF extension aimed to curate document regions.
Collection of periodical titles
IIIF collections are intended to organise documents (manifests) �in a structured way (a tree).
See also feature targeting a XML OCR file
(BnF implementation of version 3.0 Presentation API,
Winter 2022)
A
ccess to textual resources
Text (OCR, transcription) can be disseminated �as a link (to a resource) or as annotations
See also�(Presentation API)
Text overlay plugin in Mirador
OCR + named entities as annotations �(OtherContent, Presentation API v2, NewsEye EU project, https://platform.newseye.eu/)
A
ccess to textual resources
Text (OCR, transcription) as annotations.
Annotations�(Presentation API)
ToC and index as Ranges organised in a navigational structure (BnF implementation of version 3.0 Presentation API, winter 2022)
A
ccess to the document structure
ToC, lists, index can be exposed as �a structure of Ranges.
Range�(Presentation API)
National Library of Wales, newspapers�digital collection
he OHOS project
T
Is IIIF really for us?
Tools of interest (from the IIIF community)
Related projects
IIF implementation?
I
IIIF is flexible: you may want to implement the Image API alone �(3 levels of implementation), the Presentation API, the Search API…
(But to take advantage of the W3C annotation model (on IIIF documents and web pages), you need the Presentation API.)
Multi-resolution feature may �be implement in the form of tiled �pyramidal files or through�native multi-resolution�formats (tifp, jpg2000)
https://iiif.io/guides/guide_for_implementers/
s IIIF really universal?
I
IIIF will not help you to work on born- digital CHOs relying on formats like PDF or HTML.
It is unlikely that the producers targeted by the project will implement the IIIF framework in the near future...
Nevertheless, the data that the OHOS project is committed to provide could be proxied (even PDFs, as sequence of images) by the project and disseminated by IIIF.
OHOS platform
Producer�#1
Producer�#2
Still images
Producer�#3
HTML
OHOS IIIF server
OHOS Sparql end-point
From lightweight viewers (OpenSeaDragon…) to mainstream IIIF viewers (Mirador, UV)
T
he tools: viewers
Annona (Javascript library)
https://gallica.bnf.fr/iiif/ark:/12148/btv1b531454753/f1/info.json
Leaflet-IIIF (Javascript library)
https://ncsu-libraries.github.io/annona/tools/#/display?url=https%3A%2F%2Faltomator.github.io%2Fannonatate%2Fannotations%2F12148-btv1b531454753-f1-list.json&viewtype=iiif-storyboard&manifesturl=&css=&settings=%7B%22fullpage%22%3Atrue%7D&props=
Mirador+plugin; Annonatate; SimpleAnnotationServer, Miiify (Gitlab)…
T
he tools: annotating
Annonatate (stores annotations in Github)
Mirador public instance + plugin
Mirador+plugin+bindings to a in-house production app
https://raw.githubusercontent.com/altomator/IIIF/main/annotations/vdc_100022589080_annot2.json
https://api.bl.uk/metadata/iiif/ark:/81055/vdc_100022589080.0x000002/manifest.json
static storage on Github
local storage
local database
Many solutions exist to avoid the look&feel of heavy viewers.
T
he tools: exhibition-like
Yith (React component library)
Wax (framework)
Some of those tools are for storytelling (CanvasPanel, Digirati / Storiiies, Coggap+BnF / Exhibit, Mnemoscene)
T
he tools: storytelling
https://exhibit.so
NewsEye EU project, https://platform.newseye.eu
PanelTruck (Boston Public Library, Norman B. Leventhal�Map & Education center): IIIF manifests or images, static images, tiled maps…
T
he tools: geographical data?
ther projects dealing with various content
O
Recogito (Pelagios Network): ingest a wide variety of formats (txt, xml, iiif, csv…), let users annotate them all, export the data (rdf, geo-json, csv…)
IMG
IIIF
TXT
ResearchSpace (British Museum): �ingest (RDF) data, documents, leverage ontologies, create/curate/annotate data �give access to data (search, browse), �write narratives…
ther projects dealing with various content
O
PIA (‘Participatory Images Archives’, Switzerland, 2021-2025): a citizen science initiative focusing on three collections in the photo archives of the Swiss society of folk traditions (SSTP). Planning to use Linked Art (CIDOC-CRM profil), IIIF, W3C annotation model, AI for cataloguing and indexing images.
brother project
A
o wrap up
T
IIIF implementation
Architecture
Demo
HOS architecture for the content flow?
O
OHOS �Graph DB
CGDCs
OHOS Sparql end-point
Ingestion�pipelines
OHOS �platform
Enrichment�pipelines
OHOS IIIF server
Images
Web
Still images
Text flow
Metadata
Text in images
OHOS �Graph DB
CGDCs
OHOS Sparql end-point
Ingestion�pipeline
OHOS �platform
Enrichment�pipeline
OHOS IIIF server
Still �images
Web
Images
Text
From this step, most of the CHOs could be handled thanks to a � unified IIIF+W3C framework
Metadata
HOS architecture for the content flow?
O
IIIF as an enabler: the sooner, the better!
sers?
U
OHOS �Graph DB
OHOS Sparql end-point
OHOS Curation�(ResearchSpace �like?)
OHOS IIIF server
Storrytelling,�narratives
Annotation, transcription, correction
Search, �browse
Submit�CGDC
…
emos: PDF to IIIF pipeline
D
Internet �Archive�API�(PDF eBooks)
Book as �JP2 files
JP2 images served by iipsrv – no tilling�(Image API)
Manifest generation with Javascript & Deno�(Presentation API)
OCR generation (Tesseract)
Solr indexation �(binded to IIIF Search)
emos: Web images to IIIF (toy) pipeline
D
Test bed: People’s collection (National Library of Wales)
Web site
Scraping images�(scrapy, �splash)
OCR �generation
Image analysis�(zero shot classification or search with CLIP)
Manifest generation
John Moore’s pipeline
Zero shot classification: photograph/printed document/poster (CLIP or trained model)
emos: Web images to IIIF
D
1. Images scrapping of the ‘Women for life on Earth’ corpora: 34 images, some �of them are written documents
169 items: PDF, pictures, video
?
Scrapy
34 images
1. Images scrapping of the ‘Women for life on Earth’ corpora: 34 images, some �of them are written documents
169 items: PDF, pictures, video
?
Scrapy
34 images
emos: Web images to IIIF
D
2. This part of the pipeline is in charge of the image conversion and the manifest production: this can be done with libraries (e.g. iiif-prezi), �a templating system…
Manifest
Manifest template
Pyramidal images�(static, jp2, tifp)
Classification output �from AI models
Local Mirador 3�for test
Collection of manifests
Batch of �images
…
libvips, �kakadu…
}
Project�server
emos: Web images to IIIF
D
3. CLIP (OpenAI) is a text to image model trained on the web (hundred of M of image-caption pairs). In a document types classification scenario, we have to figure out the right captions (“prompt engineering”) regarding our targeted classes.
similarity between the text/image features are computed
See on https://colab.research.google
top probabilities
emos: Web images to IIIF
D
CLIP (OpenAI) is a text to image model trained on the web (hundred of M of image-caption pairs). In a document types classification scenario, we have to figure out the right captions (“prompt engineering”) regarding our targeted classes.
similarity between the text/image features are computed
See on https://colab.research.google
top probabilities
100% recall
emos: Web images to IIIF
D
And what about photographs of documents? �(We would like to apply an OCR on these.)
emos: Web images to IIIF
D
Central Midwives Board - Register of Cases-5
Frame of two Welsh poems-6
Guessing from metadata?
4. Content analysis of the photographs: the same zero shot capacity can be leveraged. Using our prior knowledge of the WPM corpora, we can classified the image content relatively to our vocabulary:
emos: Web images to IIIF
D
demonstration
‘Women Archives of Wales’ corpora: �830 images, greater variety of content
916 items
?
830 images
emos: Web images to IIIF
D
OCR/HTR
emos: Web images to IIIF
D
OCR
emos: Web images to IIIF
D
Query: woman in the trenches
CLIP can be used to produce searchable labels, but a better scenario is to implement its ‘query by prompt’ ability as a end user functionality
https://imgs.ai
Try it on heritage collections!
emos: Web images to IIIF
D
emos: Web images to IIIF
D
Output from CLIP (document type classification): �label attached to �Canvas #1
Finally, we put it all together in a IIIF manifest + a linked annotations list.
emos: Web images to IIIF
D
Output from CLIP �(content classification) and human curation: �list of annotations linked �to the manifest on�canvas #1
MERCI !
jean-philippe.moreux@bnf.fr
gallica.bnf.fr
gallicapix.bnf.fr
data.bnf.fr
api.bnf.fr
www.bnf.fr/fr/bnf-datalab
IIIF at the BnF
Time Machine Academy training session on IIIF, September 2021
The Gallica technical stack, presentation at TNA, November 2019
Gallica user statistics:
Unique visitors per day: ≈55k / day
Calls to the Image API: ≈300k / day
2021
2020
Resources
My Stylo log book during the secondment
GallicaPix
gallicapix.bnf.fr
https://github.com/altomator/Image_Retrieval
IIIF
iiif.io
https://github.com/IIIF/awesome-iiif
https://github.com/kirschbombe/iiif-for-scholars
https://www.ghentcdh.ugent.be/projects/madoc-iiif-annotation-and-crowdsource-platform
https://www.cogapp.com/storiiies
https://github.com/altomator/IIIF
Annotation
https://www.w3.org/TR/annotation-model/
https://training.iiif.io/iiif-online-workshop/day-four/
https://github.com/glenrobson/SimpleAnnotationServer
https://dhcnc.org/project/open-source-annotations-on-iiif-resources-an-interview-with-niqui-oneill/
OCR (open source)
Tesseract
Kraken
PERO OCR
…
HTR (handwritten text recognition)
https://readcoop.eu/transkribus -> experiments with the McGill Univ. Library
eScriptorium.fr (based on Kraken OCR engine)
AI in the heritage sector
ai4lam.org <- open to all!
IFLA Artificial Intelligence Special Interest Group
CENL AI in libraries network group