1 of 32

Hydra-in-a-Box�Final Project Report

CNI Forum 2017 Washington, DC

Tom Cramer & Hannah Frost�Stanford Libraries

Michele Kimpton�DPLA

Debra Hanken Kurtz

DuraSpace

2 of 32

Fledgling DPLA looking for more robust solutions for its Hubs
Hydra project thriving organically (but still a framework, more than an app)
Opportunity for easy-to-deploy, easy-to-maintain, cloud-ready, high-quality repository platform

Flashback to 2014

First, let’s review just what this project is about, in case you haven’t heard about it before, or if you need a refresher:

About 4 years ago, the Digital Public Library of America, DuraSpace, and Stanford recognized together that Hydra -- the growing open source community and framework focused around digital repository solutions -- needed to develop a packaged solution to make it easier for institutions of all sizes, even those with limited technical development capacity and limited budgets, to adopt the Hydra repository platform and the open, next-generation web technologies associated with it, and further to provide an on-ramp to participation in DPLA through the repository.

<click>

So the three partners sought and were successfully awarded a National Leadership Grant from the IMLS, one focused on fostering a national digital platform through community-based repository infrastructure, components, and open source stewardship.

3 of 32

HyBox Objectives

Create a turnkey, Hydra-based application
Bolster DPLA via modern interfaces and metadata aggregation capabilities
Get cloud-ready
Do this in concert with the Hydra & DPLA Hub communities

4 of 32

Hydra-in-a-Box

5 of 32

Work Accomplished

Survey & Product Design
Turnkey & Cloud Engineering
Hyrax Feature Development
Hyku Hosted Service Pilot
Community Engagement

6 of 32

7 of 32

Easy to Install

8 of 32

0:01/5:19

2:49/5:19

See full video: https://youtu.be/hC0_PB31ko8

2:50 - 3:38/5:19

9 of 32

Easy to Configure

10 of 32

�

11 of 32

�

12 of 32

Easy to Deposit

13 of 32

�

14 of 32

Easy to Manage

15 of 32

�

16 of 32

17 of 32

�

18 of 32

19 of 32

Easy to Share

20 of 32

21 of 32

To this list of features we can add another major development in our project: an implementation of ResourceSync as an alternative, or next generation, approach to harvesting repository metadata by aggregators.

ResourceSync is a framework developed by LANL in 2013, and was formalized as an NISO/OAI specification earlier this year. It takes a web-centric approach to solving the problem of keeping web-published resources in sync as inevitable changes occur. Nothing special is required beyond publication of a site map and change lists, leveraging timestamps to indicate that changes have been published and when. It is likely to be an improvement over OAI-PMH and it has the added advantage of facilitating search engine optimization at the same time.

We tested our implementation with DPLA and wrote up the results in a blog post:

http://hydrainabox.samvera.org/2017/06/22/resourcesync.html

22 of 32

Major Deliverables

New solution bundle with fresh design, better UX
Installation by Docker or AWS CloudFormation
Multitenancy for hosted services
ResourceSync for aggregators
Features: Rightsstatements.org, IIIF API,�Workflow, Internationalization, User Dashboard

See Detailed List of Deliverables

23 of 32

Recognizing Community

�

24 of 32

Recognizing Contributors

�

25 of 32

Tracking Progress

�

26 of 32

Development Priorities Ahead

Bulk import/export
Bulk metadata review and editing
Analytics engines for reporting on discovery & use
More content types
Improved group/role management
Institutional authentication support

�See HykuDirect Gap Assessment

Our original scope was pretty big -- and we knew that it would require flexibility and there existed the possibility of scaling back dependent on the level and consistency of development resources sustained over the course of the project.

We have heard that while our support for IR content is pretty strong -- because it is based on the Hyrax repository code designed for IR use cases and has been in production for four years – there is room for growth in the way that digital library collections are managed and presented in the application.

This list of development priorities reflects work we need to do – and in some cases is already underway in terms of UX design, specification, and implementation – in order to solidify this repository for the broad common set of needs by the community.

But we know we are the right track. Some piloters are productively using the application already.

27 of 32

What is the “aggregator in a box”?

Different than initially envisioned
Needed tool built for speed and scale
To ingest 10 of millions of records
Ingestion 3 born
Continuing the work this year to build out the remaining tool set working with our hub network

28 of 32

Spark OAI Harvester

Harvests records and sets from OAI-PMH repositories to Spark SQL
Technical documentation: https://digitalpubliclibraryofamerica.atlassian.net/wiki/spaces/TECH/pages/87658172/Spark+OAI+Harvester
Github repo: https://github.com/dpla/ingestion3

�

29 of 32

Cloud Work

CNI Fall 2017| @debra

Cloud work

The Hyku application is multi-tenant, which allows for sharing infrastructure while also ensuring organizational accounts and content are distinct. This is handled at every level within the system, from the Hyku application, to the database, the search engine, and Fedora.�
Multitenancy is ideal for consortia, anyone who wants to host for more than one group, far more capable of scale, auto-scaling allows adjust to demand more quickly and easily. Multi-tenancy allows for significant reduction of overall cost and complexity as the capacity of the system grows�
Cloud formation templates are used for building the stacks, they document all that is needed to do to hook parts together for AWS. Eliminating this piece of infrastructure ==> entire setup within 20 minutes, infrastructure as code, make pull requests against infrastructure set up so we know exactly what changes are made in Git. �
Overview of architecture�- Deployment of the system is managed using CloudFormation in AWS; this is a template-based deployment method which codifies each step of the deployment process, ensuring that it is consistent and repeatable� �- Several managed AWS services are used to reduce management overhead and simplify operations such as storage, job processing, database management used by Hyku and Fedora� �- All applications are deployed within a private subnet in a VPC (Virtual Private Cloud), ensuring that most communication between components occurs within a private network and that access to external networks is tightly regulated� �- The Hyku and Fedora applications are both deployed using Elastic Beanstalk, which provides management of load balancing, auto-scaling, as well as the process of performing deployment of new application versions to ensure no application downtime� �- The Solr indexing engine, along with Apache Zookeeper, which is utilized for Solr cluster management, are also deployed using Elastic Beanstalk, with custom configuration that allows Solr and Zookeeper to communicate directly within the cluster while other applications communicate through load balancers� �Design decisions�- Chose to use AWS, as it provides the greatest depth and breadth of any IaaS (Infrastructure as a Service) vendor, with a continually growing list of services and features. The growth of AWS is continually accelerating, providing new opportunities for efficiency.� �- Chose to make use of AWS services where possible, allowing us to off-load operating servers when AWS offered a service to handle those operations for us. This reduces overall stack complexity and operations load� � �Why this is a good path� �- Operations becomes less ad-hoc and significantly more consistent and repeatable� �- Operational approaches and techniques have become code which can be shared with the community and improved for mutual benefit� �- Even those not running completely in AWS can take advantage of specific AWS services where needed to improve local operations, and all of the configuration for how we configure these services is captured in code that can be reviewed�
Costs - Costs more to run the full stack, multi-tenancy, than for one account. One instance - multi-tenant is over-kill. Operations - much better approach because it allows consistency, reproducible - tactics necessary to maintain system over time�

30 of 32

Interest by adopters and interest from vendors�

New Hampshire Digital Project - Green Field

CNI Fall 2017| @debra

40+ institutions of all sizes from around the world have contacted us to indicate interest in hosted Hyku including consortia. �
Interest in Hyku has come from institutions that want to combine their IR and cultural heritage materials, as well as from institutions and consortia that are specifically interested in a multi-tenant IR option�
Institutions with cultural heritage materials want to see some further development especially with regard to metadata workflow features.�
Our six diverse pilot institutions were interested in hosted Hyku and provided valuable feedback, and two PALCI and Az State Archives indicated interest in continuing to pilot as further development work has been done.�
Most pilot institutions indicated that they would consider a move to Hyku if the gap assessment items were addressed and tooling developed for migration from existing systems

31 of 32

Looking forward

https://github.com/samvera-labs/hyku

DuraSpace Registered Service Provider Program: �etripp@duraspace.org

CNI Fall 2017| @debra

Where to get it: download and go - Git (link)
Is it ready to be adopted? - Depends on use case. For institutional repository use cases, the software underlying Hyku has been used in production by numerous institutions since 2013. For digital collections management use cases, Hyku might be a better fit for greenfield applications, allowing you to get your institution's assets in place. If you're migrating from another repository system, it is currently ‘good enough’ to capture basic content -- work on feature parity with other repository systems such as DSpace and CONTENTdm continues. Hyku's UI is more current & user-friendly than competing repository systems, in general.
How can I contribute? - Developers can jump in on tickets, install the software and use it! File bug reports, update documentation, if looking to build single tenant and are willing to share setup templates
I’m a vendor; can I resell/help/host? Contact DuraSpace! Service Provider Program

32 of 32

Stay Informed

Visit hydrainabox.org

Read Hyku wiki�� Follow @HydraInABox and @HykuRepo

Reach hyku-contact@googlegroups.com

Download Hyku

Thank you