1 of 24

Who are you calling generic?: �Using Generic XML normalization rules for ETDs in Primo VE

Elliot Williams, Metadata Strategist

University of Texas at San Antonio

Ex Libris Southcentral Users Group (ELSUG), Dec 6, 2024

Hi everyone, my name is Elliot Williams, I use he/him pronouns, and I am the metadata strategist at the University of Texas at San Antonio Libraries and Museums. I’m here today to talk about using generic XML normalization rules to import a collection of electronic theses and dissertations in Primo VE.

I do want to say at the outset that because of the nature of this presentation, it is somewhat technical, and does assume some knowledge of XML metadata and Alma discovery normalization rules. I’ve tried my best to not make it overwhelmingly technical, but I apologize in advance for the amount of XML I’ve included in the slides.

I realized while I was writing this presentation that in some ways, it’s as much about DSpace as it is about Primo, but I’ve tried to focus on the Primo aspects of this process. And I hope it will be useful as a case study of how you can work with more uncommon metadata formats in Primo VE, even if your institution doesn’t use DSpace.

2 of 24

Background

UTSA has been using Alma and Primo VE since 2018
We use DSpace as our institutional repository, hosted by Texas Digital Library
Started adding electronic theses and dissertations (ETDs) to our IR in 2024
DSpace started being added to Primo VE recently, in 2023, so we were starting with a relatively clean slate of how to manage the process

For a bit of background, UTSA Libraries has been using Alma and Primo VE since 2018. For most of that time, we have been importing external records into Primo, so that materials we hold in other repository systems can be searchable in our primary discovery interface. Until fairly recently, though, that was limited to digital collections in our special collections CONTENTdm system. Like a lot of university libraries, we also have an institutional repository where we provide access to research outputs from faculty, students, and staff. For us, that system is DSpace, which is hosted for us by the Texas Digital Library consortium.

Early this year, we began including electronic theses and dissertations (or ETDs) in our DSpace repository. This was a big initiative for us, because previously ETDs had only been available through ProQuest and we were excited about having a local copy of our graduates’ scholarship that we could make available ourselves, and not be totally reliant on a commercial vendor like ProQuest. We absolutely knew that we wanted to make our new ETD collection searchable in Primo, and since this was a pretty high value collection for us, I was willing to put in as much work as necessary to make sure it looked as good as possible in Primo.

I mentioned that we’ve been using Primo VE’s ability to import external records for years, but it was only about a year ago that we started importing records from DSpace into Primo VE. That meant that A) I had recent experience with how that functionality works, and B) we had a relatively clean slate and I felt comfortable experimenting with how we brought in the new ETD records.

3 of 24

Quick Overview of Harvesting External Records in Primo VE

Typically used for harvesting metadata from an external system into Primo.
The external system provides XML metadata via OAI-PMH, then Alma imports that metadata and transforms it for display & searching in the Primo environment.

External System

Discovery Import Profile harvests records

Normalization Rules transform records to Primo structure

Records are searchable in Primo

4 of 24

Quick Overview of Harvesting External Records in Primo VE

Discovery Import profile – Defines the actual import of records (OAI-PMH endpoint, metadata format to use, etc.)

Metadata source format can be Dublin Core, Generic XML, or MARC

Normalization rules (Discovery) – Defines how to map the metadata from the OAI-PMH feed to Primo’s metadata format

Depending on the source format, these may be simple or complex
Normalization rules have to be added to a normalization process, which is then applied to a discovery import profile.

External System

Discovery Import Profile harvests records

Normalization Rules transform records to Primo structure

Records are searchable in Primo

The way it works on the backend in Alma is through Discovery Import profiles. Every external record source has a discovery import profile, which can be set to run on a defined schedule. The import profile defines the import of records – what the OAI-PMH endpoint is (i.e. where to get the records), what metadata format to use, what the schedule for imports is, where to find the link back to the object, etc. Relevant to this presentation, in the discovery import profile, you specify what metadata format to use; the options are Dublin Core, generic XML, or MARC (either MARC 21 or Unimarc).

In addition to the import profile itself, you also have to create normalization rules, which define how to map the XML metadata that Alma gets from the external system into Primo’s metadata format. An important note is that these are discovery normalization rules, which are slightly different from the normalization rules you use for manipulating records within Alma – the syntax is similar, but they are different, and are accessed in a different part of the Metadata Editor. Depending on the source format, these can be simple or complex – as I’ll talk more about in a minute.

And then, normalization rules have to be added to a normalization process, and that process is what you select when you’re defining the import profile.

5 of 24

Discovery import profile

6 of 24

Dublin Core vs Generic XML Normalization

DUBLIN CORE

Mapping is pre-defined by Ex Libris
Can only be used with DC metadata
Limited ways you can manipulate data
Often, all you have to do is move data from one field to another and include resource type mappings

GENERIC XML

No default mapping – you have to specify everything
Can be used with any XML metadata (DC or not)
More ability to manipulate data using XPath functions
Can only map XML elements to Dublin Core fields or local Primo fields, not “standard” Primo fields

To talk a little bit more about metadata formats and normalization rule types: As I mentioned, when you create a discovery import profile, you can set the source metadata format as Dublin Core, generic XML, or MARC. In my case, I had to choose between either Dublin Core or generic XML.

I want to acknowledge Lori Strethers, whose great ELUNA Learns presentation on a similar topic earlier this fall provided the inspiration for this slide.

If you select Dublin Core, Alma will apply the default Primo VE Dublin Core mapping. Ex Libris has defined this mapping, and you don’t have to change it (in fact, you have limited options for changing it). Since Dublin Core is pretty simple, the mapping is generally pretty straight-forward: dc:title maps to Title, dc:creator maps to Creator, etc. In a lot of cases, the only thing that you need to include in the normalization rules for Dublin Core is a resource type mapping, but you can write normalization rules to map from Dublin Core to local display fields or to move data around between Dublin Core fields.

The other option is generic XML. If you specify that the metadata is “generic XML”, you have to create the metadata mapping from scratch – the normalization rules have to specify every element in the source XML metadata that you want Alma to look for, and what Dublin Core or local field to map that element to. You do have to be very thorough and specific – if the normalization rules don’t include an element, that element won’t be included in Primo. With this option, you have the ability to process any kind of XML, and it can even be used with Dublin Core data, if you want to have more flexibility than the default DC mapping allows.

One thing that is important to note about generic XML rules, which took me a bit of time to wrap my mind around, is that, except for local fields, the normalization rules are mapping from the source metadata format to Dublin Core, and then Alma is still using the established DC mappings to map that to Primo fields. So essentially, the generic XML normalization rules are another layer happing on top of the normalization that happens if you are just importing Dublin Core metadata. This also means that you can’t use generic XML normalization to map to standard Primo fields that aren’t part of the DC mapping. For example, the Primo Genre field is not mapped from Dublin Core, so you can’t use generic XML normalization rules to put a value in that field. I used to work at a Primo Back Office institution, and this is very different from the kind of normalization rules that Back Office uses, so I wanted to mention it here.

7 of 24

DSpace in Primo

When we started adding non-ETD metadata from DSpace, we used the standard Dublin Core normalization rules.
However, DSpace doesn’t include all of the thesis-specific metadata fields in its Dublin Core output. And for Dublin Core fields, the OAI-PMH metadata doesn’t include all of the qualifiers that DSpace uses for DC fields.

For example, dc.date.available and dc.date.issued both just become dc.date.

DSpace also provides metadata in XOAI format, which includes all elements, including qualifiers
Two advantages to XOAI and Generic XML:

More, and more granular, metadata from DSpace
Greater ability to manipulate the metadata in Primo

I mentioned that we started importing our DSpace metadata into Primo about a year ago. At that time, we decided to use the standard Dublin Core normalization rules, since those worked well enough for the fairly standard scholarly research materials we had in the repository at the time.

However, even at the time, I had some frustrations with the way that DSpace provides metadata in its OAI-PMH feed, and that became even more clear as I was working on the ETD materials. We use several ETD-specific metadata fields, such as degree name and degree level (masters, PhD), and those fields just aren’t included in the Dublin Core metadata. DSpace also makes a lot of use of qualifiers with Dublin Core fields, and that specificity is lost in the DC metadata. For example, dc.date.available and dc.date.issued (which mean very different things in our DSpace data) both just get put into dc.date.

But luckily, in addition to various flavors of Dublin Core, DSpace also provides metadata through its OAI-PMH feed in a metadata format called “XOAI”. Now, I’ll be honest with you, I haven’t been able to find out much about this metadata format, but it is standard in DSpace repositories. And it provides all metadata fields and qualifiers, in a much richer format, which meant that there is a lot more metadata there for me to extract and harvest into Primo.

For our use case with the ETDs, I did some testing using both the standard Dublin Core mapping and Generic XML normalization, and ultimately decided to go with Generic XML (as you know, since I’m giving this presentation). There were two reasons why it made sense to go with generic XML: First, we could get richer and more granular metadata from DSpace if we used a metadata format other than Dublin Core. And second, using generic XML gave us greater ability to manipulate the metadata in Primo, to make sure that these ETDs looked and functioned as well as possible for our users. I’ll give some examples of this in a minute, but some of the things that generic XML allowed us to do was to separate values into multiple fields appropriate, to combine elements into one display field in Primo, and to add display constants and other explanatory text to help make it clear what the metadata meant when it was viewed in Primo.

8 of 24

DSpace XOAI metadata

<element name="contributor">� <element name="author">� <element name="none">� <field name="value">Bonner, Robert Lee</field>� <field name="authority">56c48883-c40b-43c3-a50b-bf382f611437</field>� </element>� </element>� <element name="committeeMember">� <element name="none">� <field name="value">Lengnick-Hall, Cynthia A.</field>� <field name="value">Rudy, Bruce C.</field>� <field name="value">Keating, Jerome P.</field>� </element>� </element>�</element>

I apologize for putting a bunch of XML on a slide, but I wanted to give an example of what that XOAI metadata looks like from DSpace. This is the XML for the contributor.author and contributor.committeeMember fields – so for an ETD, this is the author of the thesis or dissertation, and the faculty members who were on the committee. Each element is just called “element”, and then has an attribute that tells you what the actual name of that element is. As you can see, the XML is very verbose – it has a lot of elements and sub-elements, which means that the normalization rules for it also have to be pretty long and complex. But it includes all of the information from DSpace, including the custom qualifiers, so we can know which of these contributors are the author, and which are just committee members.

9 of 24

Example: Thesis Advisors & Committee Members

In Dublin Core, ETD advisors and committee members are all put in <dc.contributor>. In XOAI, each role has a demarcated subelement.

<dc:contributor>Ahmed, Sara</dc:contributor>

<dc:contributor>Jin, Yufang</dc:contributor>

<dc:contributor>Cao, Yongcan</dc:contributor>

<dc:contributor>Alamaniotis, Miltos</dc:contributor>

<dc:contributor>Ramasubramanian, Deepak</dc:contributor>

<element name="contributor">� <element name="advisor">� <element name="none">� <field name="value">Ahmed, Sara</field>� </element>� </element>� <element name="committeeMember">

<field name="value">Jin, Yufang</field>

<field name="value">Cao, Yongcan</field>

<field name="value">Alamaniotis, Miltos</field>

<field name="value">Ramasubramanian, Deepak</field> </element>�</element>

I wanted to walk through a couple of examples of how we used the generic XML normalization rules to solve some of the problems that I mentioned with using just the Dublin Core mapping, so you can see why we bothered with all of this.

Continuing with the example of advisors and committee members, we wanted to be able to provide some more information in Primo about who these people are and what their relationship is to the item, rather than have them just display as generic “contributors”. As I just mentioned the XOAI metadata provides the qualifier to dc:contributor that specifies what their role was. So on the left is the metadata we would get using Dublin Core, which has all 5 names just as dc.contributor, but on the right, in the XOAI metadata, you can see that Sara Ahmed is the advisor, and the other four are committee members.

10 of 24

Example: Thesis Advisors & Committee Members

Generic XML normalization rules:

Look for elements with nested <element name=“contributor”><element name=“advisor”>
Extract the value of that element
Concatenate “Advisor: ” to the beginning of that string
Add the concatenated value to discovery.local3 field (a local display field that displays as “General Note”)

Then do the same thing for Committee Members.

(The normalization rules couldn’t loop through all of the advisor and committee member elements, so I had to add 8 rules for committee member and 2 for advisor.)

11 of 24

Example: Thesis Advisors & Committee Members

// ADVISORS & COMMITTEE MEMBERS

// Concatenate with display label, and add to local display field 3 (General Note).

// Need to have one rule for each possible instance of each field (up to 8 cmte members and 2 advisors). Rules are in inverse order here, so they will display in same order as DSpace.

rule "advisor note 2"

when

exist "//*[local-name()='record']/*[local-name()='metadata']/*[local-name()='metadata']/*[local-name()='element']/*[local-name()='element'][@name='contributor']/*[local-name()='element'][@name='advisor']/*[local-name()='element']/*[local-name()='field'][@name='value'][2]"

then

copy "concat('Advisor: ',//*[local-name()='record']/*[local-name()='metadata']/*[local-name()='metadata']/*[local-name()='element']/*[local-name()='element'][@name='contributor']/*[local-name()='element'][@name='advisor']/*[local-name()='element']/*[local-name()='field'][@name='value'][2])" to "discovery"."local3"

end

Looks for “advisor” element that is under “contributor” element

This rule is for the 2nd instance of this element – there are similar rules for the first instance.

Uses “concat” to add a string to the value of advsitor, and adds that to local field 3

12 of 24

Example: Thesis Advisors & Committee Members

Using Dublin Core:

Using Generic XML:

13 of 24

Example: Degree information

ETD-specific fields (e.g. degree type, department) are not included in the OAI-DC feed. Using the XOAI metadata meant we could include those fields, and normalization rules could be used to combine them into a single field for display.

<element name="thesis">� <element name="degree">� <element name="department">� <element name="none">� <field name="value">Electrical and Computer Engineering</field>� </element>� </element>� </element>� <element name="name">� <element name="none">� <field name="value">Doctor of Philosophy</field>� </element>� </element>

</element>

Dissertation (Doctor of Philosophy) – University of Texas at San Antonio, Electrical and Computer Engineering, 2023

14 of 24

Example: Degree Information

// DISSERTATION NOTE
// Concatenate from thesis.degree fields, and add to local display field 3 (General Note)
// Format should be: Dissertation/Thesis ({degree.level}) -- {degree.grantor}, Department of {degree.department}, {date.issued}

rule "Dissertation note - Doctoral"
when
"//*[local-name()='record']/*[local-name()='metadata']/*[local-name()='metadata']/*[local-name()='element']/*[local-name()='element'][@name='degree']/*[local-name()='element'][@name='level']/*[local-name()='element']/*[local-name()='field'][@name='value']" equals "Doctoral"
then
copy "concat('Dissertation (',//*[local-name()='record']/*[local-name()='metadata']/*[local-name()='metadata']/*[local-name()='element']/*[local-name()='element'][@name='degree']/*[local-name()='element'][@name='name']/*[local-name()='element']/*[local-name()='field'][@name='value'],') -- ',//*[local-name()='record']/*[local-name()='metadata']/*[local-name()='metadata']/*[local-name()='element']/*[local-name()='element'][@name='degree']/*[local-name()='element'][@name='grantor']/*[local-name()='element']/*[local-name()='field'][@name='value'],', Department of ',//*[local-name()='record']/*[local-name()='metadata']/*[local-name()='metadata']/*[local-name()='element']/*[local-name()='element'][@name='degree']/*[local-name()='element'][@name='department']/*[local-name()='element']/*[local-name()='field'][@name='value'],', ',//*[local-name()='record']/*[local-name()='metadata']/*[local-name()='metadata']/*[local-name()='element']/*[local-name()='element'][@name='date']/*[local-name()='element'][@name='issued']/*[local-name()='element']/*[local-name()='field'][@name='value'])" to "discovery"."local3"
end

Looks for degree.level element with a value “Doctoral”

Combines degree.name, degree.grantor, degree.department, and date.issued elements into one string

15 of 24

Challenges and Trade-offs

Norm rules are very long, because the XOAI metadata structure is very verbose.

I tried to compensate for this by adding comments to each rule in the normalization rules.
Additional limitation: comments cannot be the first line in a normalization rule set.

You can’t use XML normalization rules to map to specific Primo display fields, only to Dublin Core fields or local display fields.
Adding complexity means adding risk, and likely adding more work down the road.

I have some other examples of how we used generic normalization rules for these materials, but for time purposes I’m not going to walk through them now. But they are in my slides, which will be shared after the conference, if anyone is interested in them.

I did want to talk a little bit about challenges and trade-offs with this approach. As I’ve mentioned a few times, and you’ve seen in the previous slides, the normalization rules get very long and difficult to read, because of how this XML format is structured. It definitely made writing and trouble-shooting them a challenge, and I worry that will be an on-going challenge. As much as possible, I tried to compensate for that by adding lots of comments in the normalization rule set – explicitly stating what each rule is doing and why, so it’s a little bit easier to go back and look at them. I did discover an additional challenge, though, which I haven’t seen documented anywhere in the Ex Libris documentation: the first line of a normalization rule set cannot be a comment. I wanted to add some general comments at the top of the rule set, but that line just kept disappearing when I saved the rules. Not a huge problem (I added those comments at the end, instead), but something to be aware of.

Again, as I mentioned earlier, you can’t use XML normalization rules to map to specific Primo display fields. This meant that I couldn’t do a few things that I wanted to do, like include a Genre field, but ultimately not a huge obstacle (although I imagine it might be for other types of material with more complex metadata).

The biggest trade-off, though, is that this method is quite a bit more complex than using the standard, built-in Dublin Core mapping that Alma offers, and adding complexity means adding risk and very likely adding work down the road. There’s always the worry that if I leave this position, it will be harder for someone else to understand what I’ve done and the choices that I made. There’s also the possibility that Ex Libris will change some of this functionality down the road, and we’ll have to re-do some or all of this process. Additionally, if Ex Libris makes new functionality available for DC records and external records, it will likely be designed for the standard Dublin Core mapping, and we could have to do extra work to take advantage of that.

All of those are real concerns that I discussed with the other stakeholders at my library. In the end, we decided that the benefits of having richer, more complete, and more functional metadata for this ETD collection were worth it, and we tried to do what we could to minimize those trade-offs, by creating a lot of documentation and not adding additional complexity when we could help it.

16 of 24

Epilogue: Where we are now

Ultimately, a lot of this work ended up not resulting in records that display to users.
We also have MARC records for most of our ETDs that are in ProQuest. We figured out how to get those records to dedup with the records being imported from DSpace (hooray!), but the MARC record tends to get precedence, so that is what displays to users in Primo.
However, the lessons learned from writing these normalization rules for ETDs led me to also re-write our normalization rules for other DSpace collections. So all of our DSpace records that are imported to Primo now use generic XML normalization rules, and have richer metadata in Primo because of it.

17 of 24

Useful documentation & examples

18 of 24

Thank you!�elliot.williams@utsa.edu

19 of 24

More Examples of Normalization Rules

20 of 24

Example: Open Access indicator

We wanted to include Primo’s OA indicator, but only on ETDs that were truly open – not those that were under embargo or restricted to UTSA users.

// OPEN ACCESS INDICATOR

// Only apply when access-status="open.access" (this excludes restricted and embargoed items)

rule "add OA indication"

when

"//*[local-name()='record']/*[local-name()='metadata']/*[local-name()='metadata']/*[local-name()='element'][@name='others']/*[local-name()='element'][@name='access-status']/*[local-name()='field'][@name='value']" equals "open.access"

then

set "Unrestricted online access" in "dcterms"."accessRights"

end

Looks for “access.status” element with value “open.access”

Writes a pre-defined value into the dcterms.accessRights field

Another issue that we were able to solve with the generic XML normalization rules and XOAI metadata had to do with Primo’s Open Access indicator. One of the main goals of our institutional repository is to make research as openly available as possible, so we really wanted to highlight that many of these ETDs are OA. However, because of various factors with how we added ETDs to the repository, not all of them are open – some are restricted to signed-in UTSA users only, and some are currently under embargo. So we wanted to be able to include the OA indicator only on the items that are truly open.

Fortunately, the XOAI metadata includes an element called “access-status” that has a pretty small number of possible values. So I wrote a normalization rule that looks for the open.access value in that element, and if it is found, then it creates a dcterms.accessRights element in the Primo record with the value of “Unrestricted online access”. And having that value in accessRights triggers Primo to add the OA indicator to the record.

This is also a good example of that idea that generic XML normalization rules are actually mapping from the source XML format to Dublin Core, and then Primo maps from Dublin Core to its own metadata format.

21 of 24

Example: ETD Publication Date

In Dublin Core metadata, multiple date fields are included in <dc:date>, including the date the item was added to DSpace and the date the embargo will expire. In XOAI metadata, those dates have their qualifiers included.
For Primo, we only wanted to include the date the ETD was published.

<dc:date> 2024-01-25T19:32:03Z </dc:date>

<dc:date> 2025-05-24 </dc:date>�<dc:date>2018</dc:date>

22 of 24

Example: ETD Publication Date

// DATE
// Only using dc.date.issued for dc.date
rule "dc.date"
when
exist "//*[local-name()='record']/*[local-name()='metadata']/*[local-name()='metadata']/*[local-name()='element']/*[local-name()='element'][@name='date']/*[local-name()='element'][@name='issued']"
then
copy "//*[local-name()='record']/*[local-name()='metadata']/*[local-name()='metadata']/*[local-name()='element']/*[local-name()='element'][@name='date']/*[local-name()='element'][@name='issued']/*[local-name()='element']/*[local-name()='field'][@name='value']" to "dc"."date"
end

23 of 24

Example: Abstract vs Description

Using XOAI and XML normalization rules, we could put the abstract in one field, and other descriptive notes in a local “General Note” field. Because of the structure of the XML, we can tell if the dc.description element has a qualifier based on the number of sub-elements.

<field name="value"> This item is available only to currently enrolled UTSA students, faculty or staff. </field>

</element>� <element name="abstract">� <element name="none">

<field name="value">Abstract goes here. <field>

</element>� </element>

</element>�

Non-abstract description fields are only one level below the main <element> tag.

Abstract fields are two levels below the main <element> tag.

24 of 24

Example: Abstract vs Description

// DESCRIPTION

// Add any dc.description fields that have a qualifier to dc.description. Currently, this is dc.description.abstract and dc.description.department

rule "dc.description - qualifier"

when

exist "//*[local-name()='record']/*[local-name()='metadata']/*[local-name()='metadata']/*[local-name()='element']/*[local-name()='element'][@name='description']/*[local-name()='element']/*[local-name()='element']"

then

copy "//*[local-name()='record']/*[local-name()='metadata']/*[local-name()='metadata']/*[local-name()='element']/*[local-name()='element'][@name='description']/*[local-name()='element']/*[local-name()='element']/*[local-name()='field'][@name='value']" to "dc"."description"

end

Norm rule looks for two elements below the “description” tag, which is how we know that element has a qualifier (like “abstract”).