ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
Canadian PID evaluation and selection matrix: user guide
2
Purpose of the matrix
3
The purpose of this matrix is to capture key criteria in a structured way, enabling consistent comparisons of persistent identifiers. This will support informed decision-making about which PIDs to prioritize as part of the national PID roadmap, and will bring transparency to the selection process. This shared record of the assessment of each candidate PID will also act as a form of 'organizational memory' for the national PID roadmap initiative.
4
Scope of the matrix
5
The scope of this matrix is to match the internal characteristics of each PID to assessment criteria drawn from consultation with Canadian research community stakeholders. It is designed to be equally applicable to newly-emerging and well-established PIDs, and focuses on a range of features of both the technology underpinning the PID service, and the organizations providing those services. Persistence is a feature of organizations or initiatives, rather than technology, so the matrix covers the governance, community engagement, and published policies of the parent organization in some detail.
6
Note that this matrix provides a snapshot of the status of a PID at the time of evaluation, and is intended to be revisited to map changes or progress over time, and to support updated comparisons as new PIDs emerge, enabling a record of the evolution of priorities and PIDs alike over time. It should, therefore, be treated as a living document, open to expansion or amendment as community needs or priorities evolve.
7
External factors, such as integration levels, adoption in specific communities, interoperability with other PIDs, policies relating to PIDs etc. are not included in this matrix. These factors are highly changeable, and are a component of the context for strategic activities, rather than a feature of any given PID itself. They should be investigated as part of the roadmap design process.
8
Information sources used in developing this matrix
9
SourceURL or PID of resource
10
ODIN project 'trusted PID' criteriahttp://doi.org/10.6084/M9.FIGSHARE.824314
11
European Open Science Cloud PID policyhttps://doi.org/10.2777/926037
12
Principles of Open Scholarly Infrastructurehttps://openscholarlyinfrastructure.org/
13
PIDwijzer PID selection toolhttps://www.pidwijzer.nl/en
14
The Canadian PID strategy vision, mission, and principlesLINK TO FINAL VERSION TO BE ADDED
15
MoreBrains research and analyses for the Canadian PID roadmap projectWorkshops covering ~40 participants representing a diverse range of stakeholders
16
Workshop 1.1 - https://miro.com/app/board/uXjVOK9XNbU=/?share_link_id=681219444233
17
Workshop 1.2 - https://miro.com/app/board/uXjVOJAYgQg=/?share_link_id=382274407041
18
Workshop 2.1 - https://miro.com/app/board/uXjVOKDJu8s=/?share_link_id=530450420590
19
Workshop 2.2 - https://miro.com/app/board/uXjVOJMkbc4=/?share_link_id=193118410498
20
Workshop 3.1 - https://miro.com/app/board/uXjVOGBDHPI=/?share_link_id=313599688596
21
Workshop 3.2 - https://miro.com/app/board/uXjVOGP87s8=/?share_link_id=144331457804
22
CPIDAC PID consultancy final report.pdf
23
PID workflow diagram - https://miro.com/app/board/uXjVMKNgl-M=/?share_link_id=124857254579
24
Workflow report
25
Research information workshops:
26
Workshop 1 - https://miro.com/app/board/uXjVMTlJSbA=/?share_link_id=737357617094
27
Workshop 2 - https://miro.com/app/board/uXjVMOW6VVY=/?share_link_id=180899594321
28
Note: not every criterion from all of these was included as orginally drafted. For example, overlaps were deduplicated, and some criteria were unhelpful for PID selection or were incoherent in this context.
29
Understanding and using the matrix
30
The matrix breaks down into nine main sections (highlighted below) with a column for each criterion, as specified below (with relevant links to detailed information where appropriate). Each PID evaluated using the matrix should be entered into a single row, with different PID offerings for the same entities in neighbouring rows to enable direct comparisons.
31
ColumnDescription
32
CoverageThe scope and characteristics of a PID service, the metadata associated with it, and the commmunities or systems it serves are all conditioned by the entities it identifies. Coverage may be be very broad, generally apllicable, or tightly specified.
33
Entity/entities identifiedWhat entity/entities are included in the PID service's scope? For example, ORCIDs identify people who conduct or contribute to research, Crossref identifiers cover a range of entity types.
34
Geographical coverageAre there geographical or location limits to the scope of the service? Is it limited to entities (e.g. staff, scientific instruments, projects) at a single institution? Is it national? International?
35
Disciplinary coverageDoes the PID service cover entities across multiple disciplines (e.g. peer review, datasets, or grants) or is its scope focused on a particular discipline or area of practice? If the latter, please specify the subject classification or code (e.g. Library of Congress) for the discipline or field of research covered.
36
OpennessPIDs enable interoperability between systems, and act as bridges for metadata or event notifications to be shared between institutions, systems, or nations, so the openness of the system is a critical consideration. To assess this, you may need to combine elements from this section. For example, is the PID proprietary, but the metadata open? Free is not the same as open, but minimizing paywall barriers is a component of openness, alongside licensing terms, so both may need to be considered here.
37
ProprietaryAspects of a PID system may be proprietary, locked to a specific companies systems or services, or otherwise have limitations on access or re-use. Alignment with the principles of the Canadian PID Strategy requires non- or minimally proprietary PIDs, metadata, or information.
38
Open metadata (specify license)The 'data' provided by a PID registry are most commonly descriptive information about the entities identified by the PIDs, i.e. metadata. A Creative Commons "no rights reserved" (CC-0), public domain, or similar license is the most permissive for metadata re-use, and is strongly recommended.
39
Open source code (specify license)The source code and tooling for technical PID provision should ideally be as open as possible, to maximize re-use, innovation, and accessibility. Please specify the license under which source code (if any) is made available.
40
Open documentationIs the documentation for APIs, services, capabilities, integrator guides, system specifications, etc. openly available online? Please provide the URL to relevant resources.
41
Free/public API access availableOpen Application Programming Interfaces (APIs) are crucial for PID integrations into computer programes, such as research information management or grant application systems. Minimizing paywall barriers, as noted above, is a crucial component of openness: is there a tier of access to the PID service's API that is free or otherwise publicly available?
42
PID functionality freely and openly accessibleAre the core functions of the PID service available to everyone (see the section 'interoperability' below for details of these)? For example, can a DOI resolve to an article landing page or metadata record without a fee or registration? Can anyone use the PID to access either the entity it identifies, or information about it?
43
InteroperabilityA key benefit of PIDs is that they enable systems to share and re-use information, and can be integrated into existing workflows in any computer program (such as registering a DOI for a dataset at the point of creation). The factors in this section are critical to enabling consistent and efficient interoperability.
44
APIs that use open web standardsApplication Programming Interfaces (APIs) enable one computer program to offer a service to another. They are distinct from human-facing user interfaces, and are designed to be incorporated into software — a process that is facilitated by the use of the W3C open web standards and protocols and best practices such as the REST architectural style. The exact standard(s) used may vary according to context, and new standards or versions may emerge over time. For this field, it is enough to confirm that the APIs provided do conform to current and applicable standards, whatever they may be.
45
ResolvableA PID is said to be resolvable when it can be formatted as a functioning URL, is machine-actionable and can serve as a long-lasting, redirectable link to a specific resource, landing page, or metadata record. For a detailed explanation of how resolution works for Handles and DOIs, see this page.
46
Formalized as a technical standardIs the PID itself specified as a standard, e.g. like ISNI and RAiD)? If so, is it an international or national standard?
47
Support for content negotiationContent negotiation is the process by which a specific representation of an entity can be requested by appending a request to the identifier URL. For example, a DOI defaults to redirecting resolution requests to the landing page for the item identified. However, it can also return the metadata record in various formats (such as a standard citation) or the content itself. For a detailed explanation of how this works for the DOI system, see this page. Content negotiation varies according to the PID system (e.g. an ORCID ID could never return the actual person it identifies), but the core mechanism underpinning it is consistent.
48
DescriptiveThe benefits of PIDs depend not just on their capacity to act as a stable, long-lasting reference to an entity, but also in the metadata asociated with the PID. The criteria listed in this section relate to the availability, reliability, and utility of these metadata. To assess these factors, it may be necessary to combine elements from this section. For example, if metadata records are updateable and have a clearly responsible custodian, corrections will be possible in the case of errors, from which inferrences may bedrawn about trustworthiness and data quality.
49
Publicly available metadata schemaIs there a structured metadata schema that supports consistent descriptions of the entities identified by the PID service? Are the schema structure, definitions, and documentation freely available online?
50
Updatable metadata recordsCan the metadata records associated with each PID be updated, extended, or amended after the initial record has been created?
51
Version and relationship managementDoes the schema support the recording of different versions of the entities it describes? Can the schema capture relationships between the versions and other related entities (e.g. 'replaces', 'is part of', 'is version of', and so on).
52
Transparent provenanceDoes the schema/interface allow for clear attribution of the provenance of metadata associated with the PID if more than one source is able to contribute to a record? If only one source is able to contribute metadata, are the source and their responsibilities openly and transparently stated?
53
GovernanceFor PID services to be trusted, the governance of the organizations must be robust and responsive. The criteria in this section are intended to enable evaluation of the strength and features of the governance stuctures and processes of the relevant PID organization(s).
54
Stakeholder governanceWhich stakeholders are included in governance structures, or are able to participate? What mechanisms exist to ensure balance, representation, etc? Please provide a URL for any documentation of these structures, processes, or rules.
55
Transparent governanceIs governance documentation (e.g. policies, board meeting records, etc.) publicly available? Please provide the URL of relevant webpages/archives, etc.
56
Elected boardIs the organization overseen by a democratically-elected board with an appropriate community electorate to represent the interests of those served?
57
Whistleblower policiesAre there robust whistleblower or similar policies in place? Please provide the URL for such policies if they are publicly available.
58
DEI policyDoes the organization have a Diversity, Equity, and Inclusion (DEI) policy and, if so, has it been made public? If so, please provide the URL.
59
Published values/ principlesIs there a publicly available articulation of the governing principles of the PID providing organization? See, for example, the ORCID principles. Please provide the URL for the principles if they have been published.
60
CommunityWhile a PID service may be of value across a wide range of stakeholders, many organizations have a specific community focus, either in the limits of their formal membership or in the applicability of their services. This section is designed to elicit information about their community focus, strengths, and mechanisms of participation.
61
Community/ communities servedWhich community or communities are the primary focus of service provision? For example, does the PID provider serve one or more of research-performing organizations, funding bodies, or publishers?
62
Membership optionsWho is eligible for formal membership of the PID community (if applicable)? Does membership operate at the individual or organizational level, or both?
63
User community structuresWhat structures have been implemented to enable the user commmunity to engage with and contribute to the development, promotion, or priorities of the PID service? These could be advisory groups, thematic user groups etc.
64
Equity or access programDoes the organization have a formal equity or access program? See, for example, DataCite's Global Access Program. Please provide a URL for any published details of the program.
65
Outreach programDoes the organization have a formal outreach program? See for example, Crossref's Ambassador Program. Please provide a URL for any published details of the program.
66
Training and support resourcesWhat training and support resources does the PID service provide to the community? Please provide the URL for published resources or descriptions.
67
SustainabilityAs noted above, persistence is a feature of organizations, not technology. To ensure the long-term resilience of a PID service, the organization providing it should either already be sustainable, or have a viable pathway to sustainability.
68
Revenue streamsWhat are the main sources of revenue for the organization? For example, is it dependent on grant funding, or does it generate income from membership or other services? Please add URL for the organization's most recent accounting year, if publicly available.
69
Not-for-profit or equivalent?A not-for-profit organization is one in which all income goes back into the running of the organization. In the US, it may also be a nonprofit (one that qualifies for tax-exempt status by the IRS). For more information, see Nonprofit vs. Not-for-Profit vs. For-Profit: What's the Difference?
70
Sustainabiility status (goal: long-term funding and adequate resourcing to generate a surplus/ financial reserves)Does the organization have enough long-term funding to generate a surplus and ensure financial reserves (typically at least 3-6 months running costs)? If not, do they have a concrete and realistic plan in place to achieve this?
71
Business continuity/ backup planDoes the organization have a published business continuity plan covering, for example, succession planning, risk mitigation, etc? Please provide URL for these plans if publicly available.
72
Long-term metadata stewardship policyEnsuring that the metadata associate with a PID continues to be managed in the long term is an important element of sustainability. Please provide the URL for any publicly available documentation on this, if applicable.
73
Cannot be soldNot-for -profit organizations cannot be sold (although their assets may be transferred to another not-for profit organization). Depending on their governance and legal status, other types of organizations may be sold, whether with or without the consent of their users and/or other stakeholders.
74
PreservationRelated to issues of persistence and sustainability, the long-term preservation of the metadata is a core requirement if a PID service is to be trusted.
75
Publicly available long-term preservation planDoes the organization have a publicly available preservation plan for the long term? If so, please provide the URL.
76
Publicly accessible living will/ mission fulfilment wind-downLiving wills and mission fulfillment wind-down statements are a powerful way to create trust. The former comprises a detailed plan for when the organization or service would be wound down. The latter relates to the need for organizations to regularly review community support and need for their activities. Please see the POSI principles for more information and provide URLs for any publicly available documentation,
77
SupportPID services are complex systems; in the absence of a robust support framework, access to these services is likely to be challenging . These criteria in this section are intended to identify the modes of support available and (combined with the sections on Openness, Interoperability, and Descriptiveness) to enable an assessment of the appropriateness of the support provided to the user and/or implementer communities.
78
Technical support provisionWhat forms of technical support are available (e.g. documentation, help line, email support, chat, etc)? Who can access them and what service levels are in place? Please provide URLs to more information if available.
79
French and English languages supported in user interfaces and documentationBilingual support is critical for the Canadian research community. Are French and English both well supported (for users and implementers) by the PID provider? Please add URL for more information if available.
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100