Introduction to Fedora
Before We Begin
* Only if you’re mildly comfortable on the command line and want to run some test scripts in terminal - this is optional. I will demo it.
Introduction to Fedora
Overview
Arran Griffith, Program Manager, LYRASIS
Hands on: Claim a sandbox instance
Navigate to your sandbox instance
NOTE to sandbox users: replace “localhost” with your sandbox IP address when copying example commands.
Or you can use Docker!
docker run -d –rm -p8080:8080 -p61616:61616 -p8181:8181 --name=fcrepo6 fcrepo/fcrepo:latest�docker exec -it fcrepo6 bash
Learning Outcomes
Understand what Fedora can do for you
Understand the key features of the software
Gain hands-on experience with Fedora
LYRASIS supports enduring access to our shared academic, scientific and cultural heritage through leadership in open technologies, content services, digital solutions and collaboration with archives, libraries, museums and knowledge communities worldwide.
Stores, preserves, and provides access to digital objects
Supports flexible content models for objects
Supports semantic relationships between objects
Supports millions of objects, both large and small
Interoperates with other applications and services
Why use Fedora?
Fedora is flexible: it can handle both simple and complex use cases
Content in Fedora is durable: Fedora supports long-term preservation
Fedora is standards-based
Fedora powers successful digital repository and DAM applications
Fedora is backed by a thriving community
What’s new with Fedora 6?
Enhanced digital preservation capabilities
Robust migration tooling
Built-in search
Improved performance and scale
Metrics collection and reporting
Fedora Front-Ends
Fedora is middleware
You can build a custom framework, or join a broader community:
Fedora Examples
Institutional Repository
https://arminda.whitman.edu
Research Data
https://deepblue.lib.umich.edu/data
Newspapers
https://www.lib.umd.edu/univarchives/student-newspapers
Special Collections
Fundamental Concepts
COMPONENT of ECOSYSTEM
Fedora
Binaries
Sources: https://www.w3.org/RDF/icons/
https://pixabay.com/en/icon-communication-sender-antenna-157359/
HTTP- API
IIIF
Server?
Site
?
Profiles
?
Data Export Applications
Research
Data
?
Binaries
Camel Toolbox
Triple
store
SOLR
Your
HTTP
Service
Web Resources
Everything is a web resource with an identifier (URI)
Resources can have properties expressed as RDF triples
Resources can contain other resources
Structure
Book Example: Hierarchical
Book Collection
Book 1
Book 2
Page 2
Page 1
Page 2
Page 1
Page1.jpg
Page1.tiff
Page2.tiff
Page2.jpg
Page2.tiff
Page2.jpg
Page1.tiff
Page1.jpg
Container
Binary
Example URL:
https://host/fedora/BookCollection/Book1/Page1/Page1.jpg
Book Example: Flat
Book Collection
Page 1
Book 2
Page 2
Book 1
Page 2
Page 1
Page1.jpg
Page1.tiff
Page2.tiff
Page2.jpg
Page2.tiff
Page2.jpg
Page1.tiff
Page1.jpg
Container
Binary
pcdm:hasMember
Example URL:
https://host/fedora/Page1/Page1.jpg
RDF Properties
Core Features
Core Services and Standards
Versioning
02
Create/Read/Update/Delete
01
Service
Standard
Authorization
03
Fixity
04
Persistence
05
Messaging
06
Oxford Common File Layout
A simple, non-proprietary, specified, open-standards approach to the layout of preservation persistence.
OCFL Offers...
Benefits of Fedora + OCFL
Application-independent persistence
Fewer migrations in the future
Standards-based pattern for creating and managing resources
Basic tools for modeling content
HTML Interface Cheatsheet
PATCH
Slug
{
POST
GET
DELETE
Resource URI
Hands on: Claim a sandbox instance
Navigate to your sandbox instance
NOTE to sandbox users: replace “localhost” with your sandbox IP address when copying example commands.
Or you can use Docker!
docker run -d –rm -p8080:8080 -p61616:61616 -p8181:8181 --name=fcrepo6 fcrepo/fcrepo:latest�docker exec -it fcrepo6 bash
Step 1a: Resource Creation (POST)
$ curl -i -X POST -ufedoraAdmin:fedoraAdmin -H "Slug: image-collection" http://localhost:8080/fcrepo/rest
Step 1b: Resource Creation (POST)
$ curl -i -X POST -ufedoraAdmin:fedoraAdmin -H "Slug: andromeda-galaxy" http://localhost:8080/fcrepo/rest/image-collection
Step 2: Resource Retrieval (GET)
$ curl -i -H "Accept:text/turtle" -ufedoraAdmin:fedoraAdmin http://localhost:8080/fcrepo/rest/image-collection/andromeda-galaxy
Binaries
We’ll use an image for (most) examples of binaries
Download from: https://go.nasa.gov/2Hc15jG
(Save as andromeda.jpg)
Andromeda Galaxy
Credit: NASA
Step 3: Binary Resource Creation (POST)
$ curl -i -X POST -ufedoraAdmin:fedoraAdmin -H"Content-Type: image/jpeg" -H"Slug: andromeda.jpg" --data-binary "@andromeda.jpg" http://localhost:8080/fcrepo/rest/image-collection/andromeda-galaxy
Step 4: Binary Resource Retrieval (GET)
# the metadata
$ curl -i -u fedoraAdmin:fedoraAdmin http://localhost:8080/fcrepo/rest/image-collection/andromeda-galaxy/andromeda.jpg/fcr:metadata
# the binary itself
$ curl -i -u fedoraAdmin:fedoraAdmin http://localhost:8080/fcrepo/rest/image-collection/andromeda-galaxy/andromeda.jgp
Step 5: Update RDF (PATCH)
DELETE {}�INSERT { <> dc:title "Andromeda Galaxy"}�WHERE {}
b. Press “Update”
$ curl -i -XPATCH -H"Content-Type:application/sparql-update" -ufedoraAdmin:fedoraAdmin -d "INSERT DATA {<> <http://purl.org/dc/elements/1.1/title> 'Andromeda Galaxy'}" http://localhost:8080/fcrepo/rest/image-collection/andromeda-galaxy/andromeda.jpg/fcr:metadata
Our updated RDF Properties from step 5.
Last step: Delete a resource (DELETE)
What do you see?
$ curl -i -XDELETE -ufedoraAdmin:fedoraAdmin http://localhost:8080/fcrepo/rest/image-collection/andromeda-galaxy/andromeda.jpg
Departed
Fedora creates tombstone resources at “original/path/fcr:tombstone” URL, in this case
“image-collection/andromeda-galaxy/andromeda.jpg/fcr:tombstone”
To recreate a resource at that same PATH you need to delete the tombstone placeholder first
Discovered tombstone resource at /image-collection/andromeda-galaxy/andromeda.jpg, departed: 2019-08-21T17:16:17.047Z
Versioning
Versioning
Versions: retrieved via Memento protocol
Versions: created automatically or on demand via the REST-API
What is a “version”?
Icon made by ultimatearm from Flaticon
Key Memento Concepts
Creating a Version
Create a resource in Fedora:
Download Old Californa Map and create a binary resource in the image-collection.
http://localhost:8080/fcrepo/rest/image-collection/map
Modify the Resource
Update the metadata of http://localhost:8080/fcrepo/rest/image-collection/map
DELETE {}
INSERT { <> dc:title "Map of California" }
WHERE {}
View the Version History
View versions via HTML Interface
$ curl -i -X POST -ufedoraAdmin:fedoraAdmin http://localhost:8080/fcrepo/rest/image-collection/map/fcr:versions
Advanced Versioning
You can do many more things with versions:
Bundle multiple changes into a single version
With Transactions
$ curl -i -u fedoraAdmin:fedoraAdmin -X POST"http://localhost:8080/fcrepo/rest/fcr:tx"
$ curl -H"Atomic-ID: <Location>" -X PUT "http://localhost:8080/fcrepo/rest/test1234" -u fedoraAdmin:fedoraAdmin
$ curl <Location> -X PUT -u fedoraAdmin:fedoraAdmin
Fixity
Fixity continued...
On ingest, Fedora can verify a user-provided digest against the calculated value�
On demand, A digest can be recalculated and compared at any time via a REST-API request
Limitations of Fixity Checking
Fixity: Hands On
Check the fixity of the California map
$ curl -i -u fedoraAdmin:fedoraAdmin -X GET http://localhost:8080/fcrepo/rest/image-collection/map/fcr:fixity��$ curl -I -H "Want-Digest: MD5" -u fedoraAdmin:fedoraAdmin -X GET http://localhost:8080/fcrepo/rest/image-collection/map��
Simple Search
Simple Search
New resources are automatically indexed
Search using the following fields:
* Wildcards (*) supported
Simple Search
Stats
Stats API
curl -u fedoraAdmin:fedoraAdmin http://localhost:8080/rest/fcr:stats
curl -u fedoraAdmin:fedoraAdmin http://localhost:8080/rest/fcr:stats/binaries
curl -u fedoraAdmin:fedoraAdmin http://localhost:8080/rest/fcr:stats | jq
curl -u fedoraAdmin:fedoraAdmin http://localhost:8080/fcr:stats/binaries | jq
curl -u fedoraAdmin:fedoraAdmin http://localhost:8080/rest/fcr:stats/rdf-types | jq
Metrics
External Services Overview
COMPONENT of ECOSYSTEM
Fedora
Binaries
Sources: https://www.w3.org/RDF/icons/
https://pixabay.com/en/icon-communication-sender-antenna-157359/
HTTP- API
IIIF
Server?
Site
?
Profiles
?
Data Export Applications
Research
Data
?
Binaries
Camel Toolbox
Triple
store
SOLR
Your
HTTP
Service
Message Based Integrations
External services listen for messages from Fedora
Relevant messages trigger services
This system is scalable and fault-tolerant
External - Indexing
Index repository content for search
Indexing is configurable - could be based on any property
Solr and Elasticsearch have been tested
The Fedora & The Camel Toolbox
What is Camel Toolbox?
Camel is an a platform for easily implementing a host of Enterprise Messaging Patterns.
CTB is a swiss-army knife of Fedora related services
What can Camel Toolbox do?
Demo: Solr and Triplestore Indexing
$ git clone git@github.com:fcrepo-exts/fcrepo-camel-toolbox.git
$ cd fcrepo-camel-toolbox/docker-compose
$ docker compose up -d�# in separate terminal windows - view output from each container
$ docker logs -f docker-compose_fcrepo_1
$ docker logs -f docker-compose_fuseki_1
$ docker logs -f docker-compose_solr_1
$ docker logs -f docker-compose_camel-toolbox_1
Camel Toolbox Demo Video
Migrating to Fedora 6
IMLS Grant-funded Work
Fedora Migration Paths and Tools: A Pilot Project
Resources
https://github.com/fcrepo-exts/migration-utils
https://github.com/fcrepo-exts/fcrepo-migration-validator
https://docs.google.com/document/d/1AM8TMb0H0Q5RMCp96YHMXOROx--KpWDV8Q8rOEf1z-Q/edit
https://wiki.lyrasis.org/display/FEDORA6x/Migrate+to+Fedora+6
Migration-utils
Migration Validator
Metadata Remediation Guide
Migration Guide
Migration Routes to Fedora 6
F6
F3
F4
F5
Migration-utils
migration-utils takes command-line arguments
For an initial test migration:
Migration-utils - Common Configuration Options
7. --exported-dir: directory of exported Fedora 3 content
8. --resume: resume from last migrated object
9. --continue-on-error: skip objects with errors
10. --username: Username to associate with resources
Migration-utils - Common Configuration Options
11. --pid-file: a list of PIDs to migrate
12. --extensions: add file extensions to migrated files
13. --f3hostname: hostname to replace placeholders
14. --algorithm: Checksum algorithm for OCFL objects
15. --no-checksum-validation: Disable datastream checksum validation
16. --enable-metrics: Enable Prometheus metrics
Example test migration
java -jar migration-utils-6.2.0-SNAPSHOT-driver.jar
--source-type=legacy
--datastreams-dir=brown/source/brown-subset/repoarchive/datastreams_2019/2019
--objects-dir=brown/source/brown-subset/repostore/data_2019/objects/2019
-- target-dir=brown/fedora_home
-- working-dir=brown/work
Example test migration
java -jar migration-utils-6.2.0-SNAPSHOT-driver.jar -t legacy -d brown/source/brown-subset/repoarchive/datastreams_2019/2019 -o brown/source/brown-subset/repostore/data_2019/objects/2019 -a brown/fedora_home -i brown/work
Source Type
Datastreams Directory
Objects Directory
Target Directory
Working Directory
Migrate a sample data set
Migration Sample Data: Migration Example
Migration Documentation: https://wiki.lyrasis.org/display/FEDORA6x/Migrate+to+Fedora+6
Migration Validator
Validations include:
Migration Validator - Example
General usage:
java -jar fcrepo-migration-validator-1.0.0-driver.jar [cli options | --help]
Example Migration Validation
java -jar fcrepo-migration-validator-1.1.0-driver.jar
--source-type=legacy
--datastreams-dir=brown/source/brown-subset/repoarchive/datastreams_2019/2019
--objects-dir=brown/source/brown-subset/repostore/data_2019/objects/2019
-- ocfl-root-dir=brown/fedora_home/data/ocfl-root
-- results-dir=/validation-results
Easy as that!
Thank You!
Arran Griffith - Fedora Program Manager, LYRASIS
Ways to Get Involved:
Resources & Helpful Links
About: https://wiki.lyrasis.org/display/FF/
Communication Channels: https://wiki.lyrasis.org/display/FF/Mailing+Lists+etc
IMLS Grant Details: https://wiki.lyrasis.org/display/FF/Fedora+Migration+Paths+and+Tools
Oxford Common File Layout Community: https://ocfl.io
Get Fedora:
Download the latest version: https://github.com/fcrepo/fcrepo/releases
Fedora User Docs: https://wiki.lyrasis.org/display/FEDORA6x
Migration Support: https://wiki.lyrasis.org/display/FEDORA6x/Migrate+to+Fedora+6
Camel Toolbox: https://github.com/fcrepo-exts/fcrepo-camel-toolbox
Wrap Up - Questions?