1 of 21

Image Recognition for Archaeological Research

Claudia Engel and

Justine Issavi

SUL AI Studio Experiments

Jan 23, 2019

Many thanks to Chris Chute, Peter Mangiafico, Scott Haddow, Jochen Kumm

2 of 21

Project Aim:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt

The competition:

Lorem ipsum
Dolor sit amet

Apply machine learning techniques to enhance the metadata of Çatalhöyük Research Project’s (ÇRP) image repository.

3 of 21

Today ÇRP has accumulated close to 5TB of data, including:

Image repository with a total of ~150,000 images.

~49,000 images have inconsistent or very incomplete metadata.

4 of 21

Flinders Petrie behind a camera at excavations in Abydos (1899).

Courtesy © Petrie Museum of Egyptian Archaeology (UCL)

Archaeology is a destructive science.

Photography has played an essential role in recording the excavation process since the very beginnings of the discipline.

While the site of Çatalhöyük + ÇRP are quite exceptional in many ways, this problem is not exceptional and exists for most archaeological sites and projects.

Archaeology is a destructive science.

As we dig, we destroy the very contexts that are essential to archaeological research and interpretation, so one of the principle aims of archaeological fieldwork is the detailed and accurate recording of the excavation process, making photographs a vital visual records for archaeological research.

Where machine learning has been applied in archaeology, it has typically focused on single objects and patterns to support researchers in their assessment and classification of individual finds (such as lithics, etc.)

Here, we would like to go beyond identifying archaeological objects and also focus on the contexts that these objects are found in. (i.e. relation among artifacts and excavator).

This is especially exciting because beyond basic identification, this type of contextual information is not usually contained in the metadata.

So with machine learning, we finally have techniques that can explore the untapped analytical potential of archaeological photographs.

5 of 21

Desired output:

Experiments:

To label ~49,000 images that lack valuable metadata using:

•A subset of already labeled images in the database

•A subset of images labeled manually

•A subset of images that were taken with a whiteboard containing information about the object and photograph.

Ultimately, we would like to query images for “burial hole with skeleton” or “bone with stone artifacts,” we also plan to identify particular archaeological objects (e.g. figurines, bucrania, obsidian blades, etc.)

Detect images with whiteboards.
Extract textual information + parse the text.
Annotate the whiteboards to isolate and machine read the handwritten text on whiteboards.

6 of 21

Tagging untagged images

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt

The competition:

Lorem ipsum
Dolor sit amet

7 of 21

Tagging untagged images

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt

The competition:

Lorem ipsum
Dolor sit amet

49023

8 of 21

Object recognition

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt

The competition:

Lorem ipsum
Dolor sit amet

9 of 21

Object recognition

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt

The competition:

Lorem ipsum
Dolor sit amet

22068

10 of 21

Existing Models

n = 766

11 of 21

Existing Models

“soil”

Google Vision API

12 of 21

Existing Models

“soil”

Clarifai Predict API

13 of 21

Existing Models

“soil”

Google Vision API

Clarifai Predict API

14 of 21

Existing Models

Histo with how many images share labels broken down by number of labels

15 of 21

Existing Models

“predictor agreement”

16 of 21

Accuracy

In order to better assess the usefulness of the labels provided by these generic models we are going back to our own metadata, our own labeling - even though it is limited - and try to see how our own labels line up with the predicted ones.
Out of the images we labeled ‘wall’ Google caught about 20% and Clarifai caught about 60% and correctly labeled them ‘wall’.
To have a point of comparison I am showing here also the proportion out of ALL 766 images that google/clarifai labeled “wall”, which is lower in both cases, which seems to indicate that the predictors perhaps did pick up on something and are not just arbitrarily labeling a certain percentage of all images as ’wall’.
However if we look at the confidence scores, while the scores for the “true” wall images are slightly higher they can hardly be called significant, and Google again is less confident overall.

17 of 21

Thank you

?

https://cengel.github.io/Catal-Vision-API

19 of 21

20 of 21

m agic-vision.herokuapp.com�(created by Peter Mangiafico)

cengel.shinyapps.io/ClarifaiClass

21 of 21

Object recognition

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt

The competition:

Lorem ipsum
Dolor sit amet

1 of 21

2 of 21

3 of 21

4 of 21

5 of 21

6 of 21

7 of 21

8 of 21

9 of 21

10 of 21

11 of 21

12 of 21

13 of 21

14 of 21

15 of 21

16 of 21

17 of 21

18 of 21

19 of 21

20 of 21

21 of 21