1 of 25

Stick with it?!?

ActiveFedora and Fedora 6

Chris Colvard, Indiana University

Samvera Connect 2023

2 of 25

Samvera means “togetherness”

Geekscruff has a posse

I have a posse

You have a posse

Growing open technologies, together

I want to acknowledge that it has been 4 years since we’ve met in person and a lot has happened. While not everyone can be here in Philly there are two that I’d like to remember who have passed on since the last non-virtual Samvera Connect in St. Louis in 2019. Julie Allinson and from the Avalon team, Michael Johnson. It is has been a hard four years for many people and I’m so glad that you’re all here whether I’ve known you for a long time or haven’t yet met you. You have a place here. This community is special and thrives when we show up and are engaged and support each other. That is what I take from the Julie Allinson stickers that say “Geekscruff has a posse”. We support each on a professional level and a personal level. Geekscruff has a posse. I have a posse. You have a posse. Thanks for being here. It means a lot to me.

3 of 25

Avalon Media System

Avalon is a Samvera solution bundle for audio and video access.

Avalon 1.0 was in 2012 using Fedora 3.6 with ActiveFedora 3.3.0

4 of 25

How does Avalon use Fedora?

Models: Collections, Works, Files, Derivatives
Metadata

Work descriptive metadata in a MODS xml document
Technical metadata, access control in RDF
Structural metadata in a custom xml document

Small files (Thumbnail, poster, waveform data, etc.)
Not storing original or derivative files
No versioning
Ordered file relationships (ordered_aggregation)

5 of 25

Performance Issues

Fedora/ActiveFedora performance is passable in many Avalon use cases, except:

Works with many files (ordered_aggregation in ActiveFedora)

A small percentage of our content that causes a lot of pain
99.9% of works have less than 30 files

Reindexing

Tree walking fedora to gather ids of objects to index is impossible because root node has too many objects
Hours to return streaming GET request of root node

6 of 25

Migration to Fedora 4

ActiveFedora made a hard cut over to Fedora 4 in 2015 (v9.0)

Avalon migrated to Fedora 4 in 2017 (v6.0)

Avalon created custom migration tools including dashboard

Took a fair amount of coding effort.

Running the migration was slow.

There was a long tail to clean up problem items.

7 of 25

Why migrate to Fedora 6?

Fedora 4 - unsupported, modeshape

Fedora 6 - supported, OCFL

Will Fedora 6 provide any relief for our performance issues?

8 of 25

Which path to Fedora 6 to take?

Hyrax

Valkyrie

ActiveFedora

Mike W Hallett / Ashwell Road fork / CC BY-SA 2.0

9 of 25

Possible Pathway - Hyrax

Hyrax 5 comes with Valkyrie

Avalon did say we’d do this at SC2017 but had to divert attention

Would still be a really large effort: 1+ devoted years

Data migration

Reassessing long-term roadmap of Avalon so not ready to commit

10 of 25

Possible Pathway - Valkyrie

Valkyrie would provide Fedora 6 and more options

Would be large effort? Maybe we could use Wings?

Data migration

Possible issues: MODS xml files with OM? Callbacks?

11 of 25

Possible Pathway - ActiveFedora

☑️ About to be deprecated

☑️ Performance issues

BUT…it’s what we’re using right now.

Maybe it could be a temporary solution until we figure out our long-range plan?

12 of 25

Let’s experiment: ActiveFedora

Prior work by others in branch for Fedora 5 support

Experiments 2 years ago looked promising with most tests passing

Let’s experiment some more and see if it’d work without investing too much time.

Can we avoid slow app-specific data migration and tooling?

Will our pain points be any better?

13 of 25

Steps

Get ActiveFedora tests passing with Fedora 6.4
Get gems avalon uses working with branch of ActiveFedora
Get Avalon tests passing with Fedora 6.4
Manual testing of greenfield instance
Migrate small avalon instance
Manual testing of migrated data

14 of 25

Changes to ActiveFedora

https://github.com/samvera/active_fedora/compare/main...fedora6-cjcolvar-rebase

2019: moved types form direct/indirect container classes into headers (Trey Pendragon)

2020: changed handling of external files (Stuart Kenny)

version handling updated, dropped ability to restore versions (Stuart Kenny)

2021, 2023: Fix tests, upgrade faraday to 2.0, test with fcrepo 6.4 (Chris Colvard)

15 of 25

Changes to other gems

active_fedora-datastreams: https://github.com/samvera-labs/active_fedora-datastreams/compare/main...fedora6_rebase

Use the ActiveFedora branch

Pin rdf-rdfa

Test with fcrepo 6.4

16 of 25

Changes to Avalon

https://github.com/avalonmediasystem/avalon/pull/5363

Pair-tree ids: Fedora no longer creates pair-tree ids by default so had to override ActiveFedora to force app to mint ids
ordered_aggregation: has_part relation triple on parent not being created so doing it manually in IndirectlyContainsAssociation#insert_record and code in ActiveFedora::ChangeSet#changes to detect this change so it gets persisted

17 of 25

Test migration

Setup test instance with data from https://archivomesoamericano.org

5 collections, ~550 works, ~400 files, ~1100 derivatives

~4300 ActiveFedora::Base objects total

Migration process: https://wiki.lyrasis.org/display/FEDORA6x/Migrate+to+Fedora+6#MigratetoFedora6-TheFedora4%E2%86%926MigrationPath

Export from Fedora 4 (3 min 42 sec)
Upgrade data to 5 (1 min 13 sec)
Upgrade data to 6 (1 min 4 sec)

Start up Fedora 6 on migrated OCFL filesystem and it just works!

18 of 25

Changes to migration tooling

Skip special handling of ACL and ebucore:filename predicates so they flow through since Avalon and Hydra-Head are using them differently than Fedora expects.

Diff: https://gist.github.com/cjcolvar/39f1290323373499bb86de0981ba6f27

19 of 25

Neat! It seems to work?

�But what about our performance pain points?

20 of 25

Real World Performance Testing

Batch ingest a 100 section item into Avalon:

Fedora 4 - 32 minutes 13 seconds

Fedora 6 - 44 minutes 6 seconds

Add another section to that item:

Fedora 4 - 85 seconds

Fedora 6 - 504 Gateway timeout - 35 minutes 55 seconds (left object broken?)

Publish the item (simple save):

Fedora 4 - 12 seconds

Fedora 6 - 6 seconds

21 of 25

Performance Testing: many works in a collection

Benchmark.benchmark { |x|100.times { x.report { MediaObject.create!(collection: c) } } }

22 of 25

Performance Testing: many files in a work

Benchmark.benchmark { |x|100.times { x.report { MasterFile.create(workflow_name: 'avalon', media_object: mo) } } }

23 of 25

Avoid the ordered_aggregation!

So what exactly is ordered_aggregation?

Order through a linked list of proxy nodes using ore:aggregation

How to store order differently?

Serialized JSON array of ids

Like how Valkyrie does it

Would require a data migration but might be fairly narrow and simple?

Could this be the performance problem for pre-Valkyrized Hyrax as well?

https://github.com/samvera/hydra-pcdm/blob/main/lib/hydra/pcdm/models/concerns/pcdm_behavior.rb#L27-L35

Changes to models for testing: https://gist.github.com/cjcolvar/785f723942707653bfaa442a77305317

Changes in testing script including explicitly saving twice:

mf = MasterFile.create!(workflow_name: 'avalon')

mf.media_object = mo

mf.save

24 of 25

Conclusions and Future Explorations

Took <1 month of work
Slower in our pain point
More testing
Try fully removing ordered_aggregation and write migration
Contribute code back to ActiveFedora, gems, migration tools
Investigate cost of implementing Valkyrie and migration

25 of 25

Questions or comments…reach out!

#avalon channel in Samvera Slack
avalon-discuss-l@iu.edu list
more at avalonmediasystem.org/connect