1 of 25

Maintaining Healthy Databases

Kelli Babcock

Archeion Coordinator, Archives Association of Ontario

2023.05.11 - 11:15 - 12:00

�

2 of 25

So hello, my name is Kelli. I am presenting today as the AAO’s Archeion Coordinator - a role I took on in January of this year
Around ten years ago, as a relatively fresh new archives grad, I was also on the AAO Board, taking on the Vice President role in 2012 just as the 1.7 million dollar National Archival Development Program funding pool was cancelled by the federal government during heritage sector funding cuts
It is impossible to do this presentation without positioning myself as someone who graduated in times of fiscal restraint, fiscal restraint we are again re-experiencing a decade on, who knows what it is like to go from contract to contract on precarious employment, because that experience informs all of my opinions about keeping healthy databases - you can’t have healthy databases without staff to care for them, and this is directly tied to the funding available to archives and archivist jobs
Healthy archival databases - and websites - are essential to reinforcing existing archives advocacy efforts: we need to continue and continue and continue to say the same things over and over again to our funders and administrators. From my perspective, the 3 core issues are:

1) in most cases, digital infrastructure in our organizations is more ephemeral than the physical records in an archives’ care. We should not over invest in our database infrastructure but also must not underestimate their costs
2) we need to carefully and strategically fund and resource our digital infrastructure as it meets our institutions’ priorities and our users’ needs and do so responsibly in our roles as memory and information institutions - in most cases, the databases we run are intended to provide access to information in perpetuity in line with our mandates to care for the physical records in perpetuity
3) as information professionals, we also have to know our own limits and capacities for keeping our digital infrastructure healthy. We need to work very hard to implement these boundaries when we are often asked to do more with less

This presentation is intended to help you set those boundaries and justify why you need to set them, and think strategically about where to invest time to keep your database healthy - even if you’re not a database administrator, even if you don’t work at all with digital infrastructure at your institution, you still have a role to play in these strategies - and probably great feedback and important insight on all of the various aspects of labour and maintenance tasks required that I’m inevitably missing here because I don’t do archival processing, I work solely on the database side of archival discovery
The presentation is intended to outline practical and hands on strategies - maybe some of this will seem simple, but they remain important to surface and highlight
I also want to position my personal perspective and context: I am white, able-bodied, and the ancestor of settlers. I am working on the traditional lands of the Huron-Wendat, the Seneca, and the Mississaugas of the Credit.
I want to recognize that I am also privileged to have full time and permanent employment, in addition to being the Archeion Coordinator, as Digital Initiatives Librarian at the University of Toronto Libraries where part of my role involves managing our Discover Archives AtoM service. This job affords me to bring over 10 years of knowledge in maintaining databases to the AAO’s Archeion program, but it also enables me to have security in sharing my perspectives honestly and openly
I want to recognize a lot of the points in this presentation are also the result of many years of collaboration with archivists and technology staff at the University of Toronto Libraries and U of T archives; if I remembered the name of the person who shared a tip or an article with me - I put it in the slide notes. I am grateful for the many years of Slack and Teams chats with the folks I work with - I always have lots of questions and ideas and they are always patient and collaborative and I learn so much from them

3 of 25

Agenda

Archival databases
Database wellness + health plan activities
Our databases & our patrons

https://bit.ly/aao-2023-healthy-databases

4 of 25

Section 1: Archival databases

What do I mean by databases? Definitely talking about archival databases today - the slides are mostly recommendations from my experience managing relational databases for archival description metadata, like AtoM
For many decades now, databases have been essential in supporting archivists in organizing, describing, and, when possible, making archival records accessible online and are also essential in many digital preservation work flows for many institutions
Running any database platform is a significant investment of labour - whether you self-host the database but also even when you use a shared database, like Archeion, or pay a vendor to host - it requires a lot of planning, policies, and resourcing
We all use different tools, some examples are on this slide, but the point is: we all have a database web presence of some sort and the suggestions in this presentation are applicable to most - though others at this conference likely have other tips from experience with specific types of datasets or specific database work flows
Healthy databases are something all AAO conference attendees will likely need to consider at some point in their careers

5 of 25

https://www.archeion.ca/

https://aao-archivists.ca/

Archeion, as most of you likely know, is an example of an archival database - it runs on the Access to Memory, or AtoM, platform
The maintenance and management of Archeion is funded by AAO member institution fees as well as a provincial operating grant
The AAO does not host it - it is hosted by Artefactual Systems, the company that maintains the AtoM platform
The Archeion database is a free service for Institutional Members of the AAO - any institutional member of the AAO can use Archeion and add their archival descriptions to the database and get free support from the Archeion Coordinator for onboarding and using the system
In the slides I have linked to a page about Archeion’s history on the AAO website if you are interested in learning more about how the Archeion service started

Link: Read more about Archeion’s history

It is a really valuable database, a public good, and it is actually probably older than most of the databases at our institutions - it launched in 2000 so it is 23 years old and has been running on the Access to Memory platform since 2011.
I am going to use Archeion in some examples but I do want to highlight its age and how well cared for it has been over the years; any database requires maintenance but health issues, like with humans, increase as a database ages
I think it’s ok to be public about maintenance issues because that’s how we acknowledge and improve database health and also helps communicate why we need staff to maintain our databases long term.

Top-right image: building logo from Laurentian University Archives Archeion Institution page: https://www.archeion.ca/laurentian-university-archives
Bottom left postcard image: https://www.archeion.ca/bolton-ont-looking-east
Bottom middle image - finding aid from Archeion: https://www.archeion.ca/vttv

6 of 25

Section 2: Database wellness

How do we always ensure we

keep our databases 100% healthy?

You can’t.

Because of “technical debt”.

I guess I want to start by saying that your database will never be 100% healthy - no matter what you do and how much funding you have or the size of your institution.
That’s because of a thing called technical debt - if you have a database, you automatically begin accruing technical debt that will inevitably impact the health of your database
In the slide notes I’ve linked to a favourite 2022 article about archives and technical debt from The American Archivist
It was initially shared with me in an archives and technology research group that I am a part of
The article provides a history to the term “technical debt” and helpfully places the phrase in an archival context by explaining:

First… a technical debt metaphor for archivists is not about software development, per se; it's about stewardship of their digital collections. Second, archivists manage collections with enduring or long-term value with technologies and tools that are comparatively ephemeral. The notion of perpetuity emphasizes the long-term consequences of decisions made in managing archives.

They go on to note:

Technological advances over the last two decades have driven and enabled an explosion of [digital infrastructure] within libraries, archives, and museums… While archivists continue to perform their traditional duties around collection development, appraisal, processing, description, reference, and records management, many of us must also incorporate vocabulary and skills from the technological and business sectors to meet reasonable—yet ever increasing—user expectations for transparent digital access and services.

from: Déirdre Joyce, Laurel McPhee, Rita Johnston, Julia Corrin, Rebecca Hirsch; Toward a Conceptual Framework for Technical Debt in Archives. The American Archivist 1 March 2022; 85 (1): 104–125. doi: https://doi-org.myaccess.library.utoronto.ca/10.17723/2327-9702-85.1.104

Even if your database is functioning well - if everything appears to be working - your database accrues technical debt just by existing and this adds an additional workload area to your institution’s staff…

7 of 25

Technical debt health plan activities

You can monitor the technical debt of your database by becoming involved in the software community of the platform you use - for example, if you use Archeion, then your platform is AtoM, and you can keep track of AtoM’s Release Notes and software issues while also signing up for their Google Group to get email updates.
This can help in developing plans and timelines for when you will need to work on your database - for example, the image on the slide shows the AtoM issues tracker where you can see what issues AtoM developers are resolving for the next public release and you can begin to plan capacity to have staff time to learn about and respond to this next release when it is made public in 2023
It’s important to be aware of the release cycles of your database platform - new releases always have the potential to introduce new bugs into the database since the codebase can change, introduce new user interface changes that will impact your users, or result in new features and new work flows that will impact anyone in your institution using the database

8 of 25

Other database software may have different methods for keeping track of their roadmap and new releases
One of the tools I love at my other job at U of T, because we run AtoM but also many different technology applications, is a website called https://endoflife.date - the link is in the slide notes
It shows release dates, the end of security support date, and the latest release date for many different applications heavily used in GLAM technology
Maybe some of you use Drupal or Islandora - this website shows you that if you use Drupal 9 then you can expect to have Drupal 9 end of life impact your work flows imminently
This site can be really handy to use during annual planning - even if you’re not a database administrator, new releases and version end of life is part of planning for the technical debt that your database accrues and it will impact you in some way
You can also see the history of Drupal version frequency - this helps to understand how often you will be confronted with version updates if you choose to use Drupal or Drupal-adjacent tools in your archives; this can help with resource planning or be a consideration during database platform selection

9 of 25

10 of 25

Section 2: Database wellness

Data integrity, reliability, and loss

11 of 25

12 of 25

13 of 25

Using the search in the database is actually so helpful to check for data integrity issues - you can run certain queries to check that your data has the expected results
For example, in Archeion, we want all descriptions to be tied to an institutional record - there should not be any orphan descriptions
But data errors happen, and anyone using the AtoM platform can use search strategies to check for data integrity issues - like where an institution has not been attached to a top-level fonds or collection description
In the slides, I’ve linked to an example of how to check for this through the Archeion search interface - I’m using search documentation for AtoM and the public search bar to return results for description where description records do not have a repository - which, sadly, there are about 3000 in Archeion, so there is some data clean up ahead of us

Example Archeion query to find orphan descriptions: https://www.archeion.ca/informationobject/browse?topLod=0&sort=relevance&query=%21_exists_%3Arepository.i18n.en.authorizedFormOfName&repos=

Related Artefactual Systems documentation: https://www.accesstomemory.org/en/docs/2.7/user-manual/access-content/advanced-search/#searching-for-empty-fields

You can also check data integrity by validating your data against external data validators for a given standard.

This is difficult to do with RAD, but in AtoM you can export data to EAD and then run the EAD xml file through public and free EAD validators - I’ve linked to one in the slides

Link to validator: https://saa-sdt.github.io/eas-validator/
Additional validation and data checking tools are listed on https://www2.archivists.org/groups/encoded-archival-standards-section/recommended-tools-and-resources

14 of 25

For internal database staff, we know that, often, data integrity issues like missing repository information in archival description records are just a data entry mistake
But these small and easy mistakes do impact the public perception of the reliability of our databases
A user might land on this description without a repository, not be able to find who to contact with questions about the file or how to access the records, and immediately write off the database as unhelpful
Archival description can be confusing enough - we need to help guide users to the holding institution so they know who to contact for help
So data integrity and reliability issues around orphan description data in Archeion can be very impactful
The 2022 article I referenced earlier about technical debt describes the end user impact of unreliable data a bit more.

They explain: “Internal stakeholders may have a more sophisticated appreciation for the impact of quality on features such as search and retrieval; for example, users may not know what they are missing due to poor-quality metadata, while an archivist with intimate knowledge of a collection [does].”

They go on to identify other data reliability issues such as inaccurate “lengthy descriptive notes”

“for example, misidentifying individuals [in photographs]. Subject terms in one photographic collection regularly told the viewer if men, women, or children were present, but neglected broader terms that could let the viewer know what the photograph was about, such as farming or a nature park. Library of Congress Subject Headings were applied across many of the larger collections, but their interpretation, application, and entry into fields were inconsistent.”

from: Déirdre Joyce, Laurel McPhee, Rita Johnston, Julia Corrin, Rebecca Hirsch; Toward a Conceptual Framework for Technical Debt in Archives. The American Archivist 1 March 2022; 85 (1): 104–125. doi: https://doi-org.myaccess.library.utoronto.ca/10.17723/2327-9702-85.1.104

We know our data but our end users are relying on us to ensure the data they view is understandable and reliable and we are checking for data integrity, reliability, and loss

15 of 25

Broken links health plan activity

16 of 25

BROKEN LINKS IMPACT SEO

Broken links are so important to consider because they impact our database health in terms of search engine optimization - or SEO - which is “the practice of increasing both the quality and quantity of website traffic… through non-paid search engine results.”
SEO is how our databases and websites are discovered by people when they search in Google or Bing or Duck Duck Go or any other search engine.
Broken links are known to reduce a database or website’s ranking in search engines results - meaning our database or website will not surface high enough for a user to see it.
We can expect 79% to 90% of traffic to our public databases and websites to come from search engines so both fixing broken links and learning more about SEO can be really important for keeping our databases healthy and relevant

Reference: see page 75 of https://www.oclc.org/content/dam/oclc/reports/tipping-point/215133-tipping-point.pdf and https://www.smashingmagazine.com/smashing-guide-search-engine-optimization/

For some users, if our web presence doesn’t surface in search engines - we don’t exist

Links to learn more about SEO:

17 of 25

Section 2: Database wellness

User management

18 of 25

User management health plan activity

19 of 25

Section 2: Database wellness

Security

Security is an increasing concern with our databases
To help mitigate risk for security attacks as we discussed with technical debt, you always want to ensure that your database is kept up to date - you should not be running end of life software.
You will also want to have data security and data back ups in place
Resourcing for true and reliable database security has been the biggest revelation in my role working with systems staff in the U of T ITS department
They use security advisories and tools like Sentry to help manage security - links to these in the slides that you can explore as a Security health plan activity
It is a huge amount of invisibile labour and staff time that is almost never identified in grant proposals or digital infrastructure plans and should be discussed as to who at your institution is managing security and how
This is an area of database health I have very little expertise in but have watched for over a decade the significant labour that goes into protecting a database by the developer and systems staff I work alongside - shout out to Sunny, both Andies, Chris, Graham, Steve, Hitesh, Amaz, and Bilal and all the dev and systems staff who will never see this presentation because they’re too busy keeping things running

Links:

20 of 25

Section 2: Database wellness

Institutional priorities

21 of 25

Institutional priorities health plan activity

Your health plan for your database should actually start with institutional priorities and documentation about the database - this helps set your internal boundaries and priorities around how much resourcing you are going to give to the database and the specific value you want to derive from it
The health plan activity here is to clearly define the institution’s goals of the database:

What institutional priorities is the database relevant to?
Are there data limits? Terabytes/Gigabytes? File formats? Copyright? Access restrictions? Document these.
Set limits on yourself > what capacity do you have to ensure the health of the database? Maybe you don’t want to run your own infrastructure.
Recognize that institutional priorities will change and reviewing these priorities must be iterative - I’ve not talked at all about accessibility and redescription issues many of our databases need resources allocation for - but these are examples of why we need to review our institutional priorities against our database resourcing

In the slide notes I link to an excellent report recently shared with me by my colleague, Emily Sommers, that talks about “Stewardship capacity”

Link: https://www.oclc.org/content/dam/research/publications/2021/oclcresearch-total-cost-of-stewardship.pdf

This is a really helpful resource for estimating time for maintenance, data entry, including processing time for your database
If you do not have time for stewardship of the database and its data, maybe it is not going to meet your priorities and goals
The Total Cost of Stewardship’s related Tool Suite, linked in the slides, can help you do this work

Link: https://www.oclc.org/research/publications/2021/oclcresearch-total-cost-of-stewardship-tools-suite.html

If database is for digital assets and tied to digitization, another handy tool to reference when setting institutional capacity is the Digitization Cost Calculator

Link: https://dashboard.diglib.org/

Reference: Weber, Chela Scott, Martha O’Hara Conway, Nicholas Martin, Gioia Stevens, and Brigette Kamsler. 2021. Total Cost of Stewardship: Responsible Collection Building in Archives and Special Collections. Dublin, OH: OCLC Research. https://doi.org/10.25333/zbh0-a044

22 of 25

Institutional priorities health plan activity

A grant proposal to digitize a rare collection of slides at [the low rate] of 150 dpi would never make it through review.

A digital preservation plan consisting of CD-ROMS in a closet would be returned for revision.

Yet, every year, institutions develop, reviewers approve, and funders support dozens of term positions that are part-time, under-classified, or without benefits: the labour equivalent of these digital strategies.

Tillman et al. 2019.

Collective Responsibility: Seeking Equity for Contingent Labor in Libraries, Archives, and Museums. https://osf.io/m6gn2

On the topic of institutional priorities, if you are trying to create a database with one time only funding and no expectation of further resourcing you are essentially setting fire to that money and also the database
It will eventually degrade and die without resourcing - which makes the act of creating it for information management, discovery, and access a relatively pointless activity
In the slides I link to a number of reports on staffing and precarious labour from folks with more expertise who discuss this issue - but it is a common problem I see often within organisations and funding agencies. We need to reconsider how we staff our databases to keep them healthy
This quote from a White Paper on the Collective Responsibility for Seeking Equity for Contingent Labor in Libraries, Archives, and Museums really captures the issue of staffing to maintain healthy databases:

“A grant proposal to digitize a rare collection of slides at [the low rate] of 150 dpi would never make it through review. A digital preservation plan consisting of CD-ROMS in a closet would be returned for revision. Yet, every year, institutions develop, reviewers approve, and funders support dozens of term positions that are part-time, under-classified, or without benefits: the labour equivalent of these digital strategies.”

Links:

Tillman et al. 2019. Collective Responsibility: Seeking Equiting for Contingent Labor in Libraries, Archives, and Museums. https://osf.io/m6gn2
Weber, Chela Scott. 2017. Research and Learning Agenda for Archives, Special, and Distinctive Collections in Research Libraries. Dublin, OH: OCLC Research. https://doi.org/10.25333/C3C34F
https://udspace.udel.edu/server/api/core/bitstreams/ae2c27a3-09e0-4ddc-8a74-ddbe49f86b0c/content

23 of 25

Section 3: Our Databases & Our Patrons

Let’s talk briefly about our databases and our patrons.
We owe healthy databases to our users
It is part of our professional Code of Ethics and Professional Conduct as defined by the ACA - especially the sections on Access and Risk Management presented as images here and present in the link in the slides:

We must make records available to the widest possible audience in a manner consistent with their content, source, and the statutory obligations that govern the jurisdiction in which we work > our databases are 100% a part of how we do this
We must manage the risk of the database technologies we use to create, use, and manage records and users’ access to them - ensuring we are actively monitoring our database health helps us mitigate risk

Link: Posted 17 October 2017 - https://archivists.ca/resources/Documents/Governance%20and%20Structure%20/20171018_aca_code_of_ethics_final.pdf

24 of 25

Section 3: Our Databases & Our Patrons

Finally, in an online environment, it’s hard to know our users. There’s a risk if we don’t keep healthy databases - with reliable data - because this can negatively impact our users and we never even see it
Our websites, in most cases, interact with more users than any public-facing staff member will. This is a consideration my colleague at U of T Gordon Belray surfaced recently. In most cases, we will never get the opportunity for a human interaction with a large percentage of the people who access our web presence.
Concerningly, data collected for some of our users groups indicates that they are conducting information retrieval and research under conditions of stress
For example, data collected about students at higher education institutions indicates that 46% of student respondents felt more than average stress, with 15% reporting a tremendous amount of stress; 88% of respondents reported feeling overwhelmed in conducting their student activities

Link to stats: https://www.provost.utoronto.ca/wp-content/uploads/sites/155/2018/03/Report-on-Student-Health-Well-Being.pdf

Creating welcoming, reliable database, with an ease of navigation and contact point, can support our users in their research and personal study
I was very interested to hear Wendy speak during the “Archives Rx: Caring, Wellness and Relationships in the Age of Reckoning” session about kindness in developing healthy relationships with our users and would love someone to do a bit of research on what it means to create a “kind” database, with considerations around user-centered design of archival discovery databases and how we welcome and support our users in virtual relationship-based access

25 of 25

Thank you!

archeion@aao-archivists.ca

Question during session regarding how to learn more about working with digital infrastructure and databases:

Depends on what you want to learn:

Identify the platforms, tools, or applications you want to learn
Learn by doing/experimenting with a platform
Read existing documentation from the platform and/or its community
Attend conferences
Talk to colleagues

Training spots:

https://aao-archivists.ca/Digital-Resource-Hub (AAO Members only)
https://www.codecademy.com/catalog/language/sql (and other tutorials)
https://librarycarpentry.org/lessons/
Not directly related to archival databases, but a U of T colleague recommended the Linux Foundation’s free courses for information about open source software management, such as https://training.linuxfoundation.org/training/fundamentals-of-professional-open-source-management-lfc110/ - this may help you with system administration or working with system administrators