Terminology Development Protocols.xlsx

	A	B	C	D	E	F
1	Step	COKA Code System Development Protocol	SEVCO Development Protocol	GRADE Vocabulary Development Protocol	Simple Summary of GRADE Vocabulary Development Protocol	Framework Terminology Development Protocol
2	Link to Protocol	https://docs.google.com/document/d/1ze4fFCbIivkZo_WUif_JImmbTsmDjRd0ZaFxSmmJcDo	https://docs.google.com/document/d/1pzGLdyVCKcu3s2gfSfPpXDQLlQsFnLZR14ldw0nD1g0	https://docs.google.com/document/d/1P7bs8J1__JgJXSo5czSoDZp8_UKl9aKWqIIKZWSH6ls/edit#
3	1 Assemble expert working group	1. Assemble expert working group	1. Maintain one expert working group (combining the expert working groups for risk of bias terms, study design terms, and statistic terms).	1. Define the project team.
4	1a Expert working group membership eligibility criteria	1a. Expert working group membership will be open to any individual who self-identifies as a relevant expert for the code system. Relevant expertise for a code system may include but is not limited to experience evaluating or expressing the concepts to be included in the code system, either for human interpretation or for machine interpretation.	1a. Expert Working Group (EWG) membership will be open to any individual who self-identifies as a relevant expert for the code system. Relevant expertise for a code system may include but is not limited to experience evaluating or expressing the concepts to be included in the code system, either for human interpretation or for machine interpretation.	1a. Expert Working Group (EWG) membership will be open to any individual who self-identifies as a relevant expert for the GRADE Vocabulary. Relevant expertise for a code system may include but is not limited to experience evaluating or expressing the concepts to be included in the code system, either for human interpretation or for machine interpretation.
5	1b Target groups to recruit for EWG	1b. We will post open invitations as email messages to the distribution lists for the COKA Initiative, COVID-END, EBH listserv, GRADE Working Group, DECIDE project participants, AHRQ EPC listserv, HL7 CDS and BRR work groups, the Society for Clinical Trials, the Society for Participatory Medicine, International Society for Clinical Biostatistics, and PCORI.	1b. We will resend open invitations as email messages to the distribution lists or key representatives for the COKA Initiative, COVID-END, EBH listserv, GRADE Working Group, DECIDE project participants, AHRQ EPC listserv, HL7 CDS and BRR work groups, the Society for Clinical Trials, the Society for Participatory Medicine, International Society for Clinical Biostatistics, and PCORI.	1b. The GRADE Working Group will be notified about the ability to join the GRADE Vocabulary Project Group (which will be the EWG).
6	1c EWG recruitment process introduction	1c. With the invitation we will share an introduction to what is a code system, why we are doing this, a link to the protocol, and a link to a data entry form to sign up. Sign up at Code System Development Intake Form. [link no longer valid]	1c. With the invitation we will share an introduction to what is a code system, why we are doing this, a link to the protocol, and a link to the Project Page where one can join the Expert Working Group. (https://fevir.net/resources/Project/27845)
7	1d EWG intake application	1d. The data entry form will include optional demographic questions (age, gender, race/ethnicity) for the sole purpose of reporting demographic distribution of the expert working group in submitted publications of the code system.
8	1e Steering group definition	1e. Set up a code system steering group from the most actively engaged participants, specifically those who join open weekly work group meetings.	1d. Set up a code system steering group from the most actively engaged participants, specifically those who join open weekly work group meetings.	1d. Process decisions will be made by the EWG members who are present at an open meeting at the time of the decision.
9	2 Identify sources to set scope	2. For each code system, identify sources to signal the scope of (or common need for) a code system, namely tools or systems in common current use for reporting the concepts relevant to the code system. Expert working group members will be asked to identify such sources.		2. Scope of terms to cover certainty of evidence and evidence-to-decision judgments
10	3 Create list of draft terms/concepts	3. Create a list of non-redundant concepts that convey the concepts in commonly used tools and systems.		3. grab draft term list from https://fevir.net/resources/CodeSystem/27833 or https://fevir.net/resources/CodeSystem/27834
11	3a Categorical classifiers	3a. Categorical classifiers (names of code sets) may be added. (A concept may be a member of a code set.)		3a. set the categorical classifiers for the terms
12	3b Codeable term AND category	3b. A concept may be marked as “also serves as a categorical classifier” in which case the concept may be a “parent” in one or more IS-A relationships with other concepts. (A name of a code set may be a member of another code set.)
13	3c parent/child terms	3c. A concept may be marked as being a “child” in an IS-A relationship with another concept by listing the “parent” concept as a categorical classifier.
14	3d Review of scope	3d. This list will be reviewed in the open work group meetings.
15	4 Identify relevant vocabularies	4. Identify ontologies likely to include concepts on the lists created in step 3. Expert working group members will be asked to identify such ontologies. We will limit the effort to ontologies available for use without restrictions (or limited to Category 0 or 1 Restrictions per UMLS Restriction Levels described at https://uts.nlm.nih.gov/help/license/licensecategoryhelp.html).		4. Identify all the GRADE publications which may contain definitions of the terms.
16	5 Map terms to other vocabularies	5. For each concept, from each ontology, extract the display (or preferred term), synonym list (or alternative terms), and definition(s) that best match the concept, and note closely related variations.	2. For each term (to draft content for voting): 2a. For each relevant ontology, extract the display (or preferred term), synonym list (or alternative terms), and definition(s) that best match the concept, and note closely related variations.	5. For each term, from each publication, extract the relevant text to include preferred term, alternative terms, definitions, and guidance for application of the term.
17	6 Draft terms/concepts	6. For each concept:	2. For each term (to draft content for voting):	6. Draft for each term the preferred term, alternative terms (if any), definition, and guidance for application of the term.
18	6a Review mapped terms	6a. Review the displays, synonym lists, and definitions available from ontologies.	2b. Review the displays, synonym lists, and definitions available from ontologies.
19	6b Draft term	6b. Draft a preferred display, synonym list, and definition, and note matches to the ontologies to measure relative contributions.	2c. Draft a preferred display, synonym list, and definition.
20	6c Enter draft term into editing tool	6c. Enter the draft preferred display, synonym list, and definition into an ontology web editor (such as WebProtege). If approved, the dataset can be shared with National Cancer Institute (NCI) Enterprise Vocabulary Services (EVS) for entry in the NCI Thesaurus and exported for use with WebProtege.	2d. Enter the draft preferred display, synonym list, and definition into the Scientific Evidence Code System (SEVCO) -- DRAFT ONLY (Not published for current use) used for code system development.
21	6-7 Select term readiness for vote		3. For each term (for voting for approval): 3a. When the code system steering group determines the draft term is “ready for vote”, voting will be open to members of the expert working group and the EWG will be emailed a notice that the term is open for voting.	7. For each term (for voting for approval): 7a. When the code system steering group determines the draft term is “ready for vote”, voting will be open to members of the expert working group and the EWG will be emailed a notice that the term is open for voting.
22	7 Vote agreement or suggest change	7. Each member of the expert working group will, for each concept that will be a code system entry, note agreement (with the draft preferred display, synonym list and definition) or suggest changes.
23	7a Parent terms are also useful alone	7a. For concepts that are “parents” in IS-A relationships, agreement will also be sought that the concept is useful functionally without subordinate coding.
24	7b Child terms follow parent rules	7b. For concepts that are “children” in IS-A relationships, agreement will also be sought that if the child concept applies then the parent concept must apply AND the parent concept can apply while the child concept does not apply.
25	7c Vote online and asynchronous	7c. This process will be online and asynchronous.	3b. Voters will either select YES for approval of the term (including display, alternative terms, definition, comment for application, and hierarchical listing regarding parent term) or select NO with the addition of a comment suggesting the change that would make the term acceptable. The voting process will be online and asynchronous.	7b. Voters will either select YES for approval of the term (including display, alternative terms, definition, comment for application, and hierarchical listing regarding parent term) or select NO with the addition of a comment suggesting the change that would make the term acceptable. The voting process will be online and asynchronous.
26	7d Display of votes		3c. The content of individual votes (YES or NO selection, comments, and name of voter) will be visible to code system editors and may be shared during steering group meetings but will not be generally posted for open viewing outside steering group meetings and active editing of the code system.	7c. The content of individual votes (YES or NO selection, comments, and name of voter) will be visible to code system editors and may be shared during steering group meetings but will not be generally posted for open viewing outside steering group meetings and active editing of the code system.
27	7e threshold for agreement		3d. If a term has been open for vote for at least six days and has at least five votes and all votes are YES, the term will be considered approved and closed for voting.	7d. If a term has been open for vote for at least six days and has at least five votes and all votes are YES, the term will be considered approved and closed for voting.
28	8 Revise as needed for disagreements	8. For any concepts without universal agreement we will discuss the suggested changes in open meetings, revise as appropriate, then resend for voting as noted in step #7.	3e. If a term has a NO vote, the steering group will review the comments, determine if any changes to the term are warranted, then:	7e. If a term has a NO vote, the steering group will review the comments, determine if any changes to the term are warranted, then:
29	8a Restart voting process after change made		3ei. If changes are made, when the steering group considers the term “ready for vote” the voting start date will be reset and the term will be open for vote as noted above.	3ei. If changes are made, when the steering group considers the term “ready for vote” the voting start date will be reset and the term will be open for vote as noted above.
30	8b Extend voting if NO vote not warranting change		3eii. If the steering group determines no changes are warranted, the term will remain open for vote, the negative comment will be shared with the expert working group, and EWG voters may change their vote by voting again.	3eii. If the steering group determines no changes are warranted, the term will remain open for vote, the negative comment will be shared with the expert working group, and EWG voters may change their vote by voting again.
31	8c Agreement if all votes YES		3eii1. If on repeat steering group review all votes are YES, the term will be considered approved and closed for voting.	3eii1. If on repeat steering group review all votes are YES, the term will be considered approved and closed for voting.
32	8d Process new NO votes		3eii2. If on repeat steering group review there are new NO votes, the steering group will review the comments and determine if changes are warranted.	3eii2. If on repeat steering group review there are new NO votes, the steering group will review the comments and determine if changes are warranted.
33	8e Drop NO vote if no rationale provided		3eiii3. If on repeat steering group review there are no new NO votes and the original NO vote persists, the person recommending changes will be asked to provide a rationale for deliberation as described below and if not provided within 1 week the term will be considered approved and closed for voting.	3eiii3. If on repeat steering group review there are no new NO votes and the original NO vote persists, the person recommending changes will be asked to provide a rationale for deliberation as described below and if not provided within 1 week the term will be considered approved and closed for voting.
34	9 Manage persisting disagreements	9. If a concept does not achieve universal agreement (cycling through steps 7 and 8 with conflicting suggestions):	3f. If a term does not achieve universal agreement (cycling through steps a-e above with conflicting suggestions):	3f. If a term does not achieve universal agreement (cycling through steps a-e above with conflicting suggestions):
35	9a Provide rationale for dissent	9a. Each person recommending changes will write a rationale.	3fi. Each person recommending changes will write a rationale.	3fi. Each person recommending changes will write a rationale.
36	9b Share rationales before meeting	9b. The rationales will be shared with the expert working group prior to a group meeting.	3fii. The rationales will be shared with the expert working group prior to a group meeting.	3fii. The rationales will be shared with the expert working group prior to a group meeting.
37	9c Discuss preferred term	9c. The group meeting will discuss and prepare the preferred version. The preferred version and meeting discussion will be shared with the group.	3fiii. The group meeting will discuss and prepare the preferred version. The preferred version and meeting discussion will be shared with the group.	3fiii. The group meeting will discuss and prepare the preferred version. The preferred version and meeting discussion will be shared with the group.
38	9d Voting open for 48 hours	9d. Group members will have 48 hours to vote for the presented version.	3fiv. Group members will have 48 hours to vote for the presented version.	3fiv. Group members will have 48 hours to vote for the presented version.
39	9e Pass vote if at least 5 votes with 80% agreement	9e. The preferred version will become the included version if it achieves at least 80% agreement with at least 5 people voting.	3fv. The preferred version will become the included version if it achieves at least 80% agreement with at least 5 people voting.	3fv. The preferred version will become the included version if it achieves at least 80% agreement with at least 5 people voting.
40	9f Options if fail	9f. If unable to achieve at least 80% agreement with at least 5 people voting, options may include extending the voting period, dropping the item, or preparing for another group discussion.	3fvi. If unable to achieve at least 80% agreement with at least 5 people voting, options may include extending the voting period, dropping the item, or preparing for another group discussion.	3fvi. If unable to achieve at least 80% agreement with at least 5 people voting, options may include extending the voting period, dropping the item, or preparing for another group discussion.
41	10 Acknowledge contributing ontologies	10. For the first complete version of the code system with agreement reached for all entries, we will determine the percent contribution from the different ontologies. If an ontology provides > 50% contribution across the series of code systems or > 75% contribution to a single code system, we may consider deeper collaboration rather than continued maintenance of a new code system.
42	11 Publish terminology	11. We will publish the code system at terminology.hl7.org and seek publication of introductory articles to the code system in the biomedical literature.	4. When judged ready by the steering group, we will publish the code system at fevir.net and seek publication of introductory articles to the code system in the biomedical literature. When the code system is published, attribution of contributorship will be shared by:	4. When judged ready by the steering group, we will publish the code system at fevir.net and seek publication of introductory articles to the code system in the biomedical literature. When the code system is published, attribution of contributorship will be shared by:
43	11a author attributions		4a. Listing authors based on anyone who contributed as a “term editor” on any of the terms which occurs through the weekly webmeetings	4a. Listing authors based on anyone who contributed as a “term editor” on any of the terms which occurs through the weekly webmeetings
44	11b endorser attributions		4b. Listing endorsers based on anyone who contributed as a “voter” on any of the final-approval votes for any terms	4b. Listing endorsers based on anyone who contributed as a “voter” on any of the final-approval votes for any terms
45	11c reviewer attributions		4c. Listing reviewers based on anyone who contributed as a “commenter” or a “voter” on any of the disagreement votes on any of the terms	4c. Listing reviewers based on anyone who contributed as a “commenter” or a “voter” on any of the disagreement votes on any of the terms
46	11d attributions order		4d. All of these listings may be ordered by decreasing number of contribution instances across the code system.	4d. All of these listings may be ordered by decreasing number of contribution instances across the code system.
47	12 Evaluate implementation	12. For implementation and initial evaluation of the code system:	5. After the first published version of the code system is available, for implementation and initial evaluation of the code system, we will:	5. After the first published version of the code system is available, for implementation and initial evaluation of the code system, we will:
48	12a Identify groups to implement	12a. Identify tools and systems that could use the code system.	5a. identify tools and systems that could use the code system, including consideration of the previously “identified 23 commonly used tools and systems for which the first version of code systems will be developed”	5a. identify tools and systems that could use the code system, including consideration of the previously “identified 23 commonly used tools and systems for which the first version of code systems will be developed”
49	12b Offer support for implementation	12b. Offer support for implementation. Measure proportion of systems that get engaged.	5b. offer support for implementation and measure the proportion of systems that get engaged.	5b. offer support for implementation and measure the proportion of systems that get engaged.
50	12c Evaluate implementation	12c. Evaluate ease of use.	5c. evaluate ease of use.	5c. evaluate ease of use.
51	12d Identify changes from implementation	12d. Generate code system change requests as needed.	5d. generate code system change requests as needed.	5d. generate code system change requests as needed.
52	12e Repeat implementation evaluation	12e. Track systems that implement the code system and set a regular review interval to inquire about usefulness and change requests.	5e. track systems that implement the code system and set a regular review interval to inquire about usefulness and change requests.	5e. track systems that implement the code system and set a regular review interval to inquire about usefulness and change requests.
53	13 Ongoing maintenance	13. For ongoing maintenance and development of the code system:	6. For ongoing maintenance and development of the code system:	6. For ongoing maintenance and development of the code system:
54	13a Open invitation to EWG	13a. Maintain an open invitation for code system users to join the expert working group for continued feedback.	6b. We will maintain an open invitation for code system users to join the expert working group for recognized contribution.	6b. We will maintain an open invitation for code system users to join the expert working group for recognized contribution.
55	13b Identify more tools for scope	13b. Maintain a method for expert working group members to suggest additional tools or systems with common current use of concepts matching the code system.	6c. We will maintain an open invitation for anyone to suggest additional tools or systems with common current use of concepts matching the code system.	6c. We will maintain an open invitation for anyone to suggest additional tools or systems with common current use of concepts matching the code system.
56	13c Suggest changes	13c. Code system changes may be initiated by change requests from the community.	6a. We will maintain an open invitation for anyone to share comments regarding specific code system terms for continued feedback. 6d. Code system changes may be initiated by change requests from the community.	6a. We will maintain an open invitation for anyone to share comments regarding specific code system terms for continued feedback. 6d. Code system changes may be initiated by change requests from the community.
57	13d Validate change requests	13d. The code system steering group will validate that change requests are appropriate for group deliberation (eg, fits the purpose of the code system, has sufficient rationale, avoids duplication).	6e. The code system steering group will validate that change requests are appropriate for group deliberation (e.g., fits the purpose of the code system, has sufficient rationale, avoids duplication).	6e. The code system steering group will validate that change requests are appropriate for group deliberation (e.g., fits the purpose of the code system, has sufficient rationale, avoids duplication).
58	13e Draft changes	13e. Valid change requests will lead to drafting a preferred display, synonym list, and definition.	6f. Valid change requests will lead to drafting a preferred display, synonym list, and definition. Voting on changes will then be processed as previously described.	6f. Valid change requests will lead to drafting a preferred display, synonym list, and definition. Voting on changes will then be processed as previously described.
59	13f Vote on changes	13f. Each member of the expert working group will, for each valid change request, note agreement (with the draft preferred display, synonym list and definition) or suggest changes. This process will be online and asynchronous. (step #7)
60	13g Discuss disagreements	13g. For any concepts without universal agreement we will discuss the suggested changes in open meetings, then resend for voting as noted in steps #7 and #8. If not reaching universal agreement, manage as step #9.
61	13h Publish changes	13h. Changes to the code system will be published at terminology.hl7.org and released as needed.
62	14 Change management between approval and publish		7. If changes are requested after terms are approved while the code system is in development: The steering group may change Alternative terms and Comment for application through discussion in open meetings and changes will be reported to the Expert Working Group. Changes to the Preferred term or Definition for any term will result in setting the term back to draft and repeating the process for voting for approval. The prior “approval history” will be moved to a different term property but retained for documentation.	7. If changes are requested after terms are approved while the code system is in development: The steering group may change Alternative terms and Comment for application through discussion in open meetings and changes will be reported to the Expert Working Group. Changes to the Preferred term or Definition for any term will result in setting the term back to draft and repeating the process for voting for approval. The prior “approval history” will be moved to a different term property but retained for documentation.
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100