XI INTERNATIONAL CONFERENCE

“INFORMATION TECHNOLOGY AND IMPLEMENTATION” (IT&I-2024)

TaxoRankConstruct: A Novel Rank-Based Iterative Approach to Taxonomy Construction with Large Language Models

Oleksandr Marchenko and Danylo Dvoichenkov

Introduction - Context and Challenge

• Taxonomy: A structured classification of concepts based on shared characteristics, often organized hierarchically. Specific type of ontology.
• Taxonomies are critical for organizing knowledge across domains.
• Constructing large-scale taxonomies from scratch remains a significant challenge.
• Most existing methods depend on predefined hierarchies or data.

• Our goal: Develop a methodology for taxonomy construction from scratch using object attributes.

Source: carnivorousplantresource.com

Our approach:

• Focuses on attributes that define differences between concepts.
• Example: "Organisms can be divided into sub-groups based on criteria such as:
• Presence or absence of a cell nucleus.
• Ability to photosynthesize.
• Mode of reproduction.
• Presence of specific structural features, such as a notochord."
• Attributes define ranks, and ranks structure hierarchies.
• Enables flexibility and precision in capturing complex relationships.
• Supports iterative refinement and alternative perspectives.

Most of the reviewed approaches: Concepts are viewed as containers for subconcepts.

• Example: "Organism contains Eukaryotes, Eukaryotes contains Animal, Animal contains Mammal, and so on, eventually leading to Homo sapiens (human)."

Core Idea - Towards Attribute-Based Taxonomies

Source: M. Funk, S. Hosemann, J. C. Jung, and C. Lutz, "Towards ontology construction with language models," 2023

• Simplification: Although taxonomies can be graphs, we assume hierarchical tree structures for clarity.
• Multitaxonomies: A set of trees, each representing a different perspective.
• Properties as foundation:
• Concepts share a consistent set of properties.
• Parent concepts include all potential properties of their subconcepts.
• Taxonomy depth corresponds to the number of properties.

Conceptual Framework of TaxoRankConstruct

Main Task: Construct multitaxonomy for a root concept:

• Identify all subconcepts and their descendants across all trees and levels - generate subconcepts for each concept in the multitaxonomy using the root concept, its taxonomic ranks, and the specific rank.

Step 1: Identifying Key Properties:

• Determine the initial properties of the root concept.
• Derive the property sets as ordered, non-overlapping subsets of initial properties.
• Establish a bijection , where each subset corresponds to a tree.

Step 2: Iterative Concept Discovery:

• For each tree, explore concepts iteratively:
• Start with root concept marked as "unexplored."
• Identify subconcepts​ for each unexplored concept​.
• Add new subconcepts to "unexplored" and mark​ processed concept as "explored."
• Repeat until all subconcepts are discovered.

Methodology - Defining the Tasks

Process Workflow:

• Multiple generations of definitions and descriptions for root concept. Used prompts:
• Linguist: "You are an outstanding linguist." - definitions with clear, formal structures.
• Expert: "You are an outstanding ontology expert." - broader, contextualized descriptions.
• Extract properties from generated definitions and descriptions.
• Filter irrelevant properties to refine taxonomic criteria.
• Find Taxonomical Ranks: Generate ordered lists of key properties from criteria.
• Optimize Taxonomical Ranks: Reorder, modify, and finalize rank lists

​

Implementation Details - Initial Taxonomy Construction

Steps for Subconcept Generation:

• Contextual Prompts: Include target concept, root concept, ranks, and current level in the hierarchy. Used for concept definition generation.
• Generation of sub-concepts list Prompts. Use definition. Amount of sub-concepts can be set.
• Post-Processing: Refine subconcepts with examples to align with taxonomy.
• Validation: Ensure all subconcepts fit the taxonomy’s structure.
• Filter: Remove redundant and irrelevant subconcepts.

Quality Assurance:

• Retry up to 5 times if validation fails.

Implementation Details - Subconcepts Generation

Contextual Prompts Example:

• Context: "We are currently at the 'Grain pattern' level in the hierarchy (Grain pattern > Dimensional stability). The root concept of the taxonomy is Lumber wood."
• Instruction: "Give a 50-word definition for the Grain pattern of the ontological concept 'Cross-grain' for our taxonomy."
• Model’s Response: "Cross-grain refers to a grain pattern where the wood fibers run at an angle or perpendicular to the main length, resulting in challenges for working with and reducing dimensional stability. It often leads to uneven surfaces and difficulty in machining or finishing."

​

Post-Processing Example (Few-Shot Learning):

"root concept: 'Wound', taxonomical rank: 'Location', sub-concept candidates: 'Hands', 'Knees', 'Elbows'. Provide the true sub-concepts: Wounded Hand, Wounded Knee, Wounded Elbow."

• Purpose: Prevent "Domain Shift" and ensure consistency in subconcepts.

Contextual Prompts and Post-Processing Examples

Challenges of Quality Evaluation

Why Evaluation is Hard

• Subjectivity: Semantic relationships vary across contexts and domains.
• Automated metrics fall short on the chosen task:
• Can't assess semantic relationships.
• Limited resources like WordNet are inconsistent.
• Solution: Human Evaluation with domain experts.

​

Scenario 1: Basic Taxonomy Construction

• Objective: Generate simple taxonomies to establish a baseline.
• Procedure:
• Generate multiple taxonomies for a single root concept.
• Aggregate and analyze taxonomical ranks across iterations.
• Human evaluation of rank accuracy: "Does this rank highlight important features?"
• Outcome: Provides a reference for TaxoRankConstruct’s effectiveness.

Scenario 2: Comparative Evaluation with WordNet

• Objective: Compare taxonomies with WordNet hierarchies.
• Procedure:
• Generate taxonomies for root concepts and compare with WordNet hyponyms.
• Human evaluators validate relevance: "Is this an accepted sub-class?"
• Outcome: Highlights strengths and limitations relative to WordNet.

​

Experimental Setup

Experiments with Experts

• Evaluators from diverse fields (e.g., computer science, electronics).

​

Scenario 1: Basic Taxonomy Construction

• Agreement: Average expert agreement: 75.9%.
• Unique Ranks:
• 87% of ranks after the 10th iteration were marked as "Accurately" representing key features.
• Average ranks per iteration: 8.9.
• Accurately selected: 7.3; Inaccurately selected: 1.6.

​

Scenario 2: Comparative Evaluation with WordNet

• Agreement: Average expert agreement: 70.4%.
• Accepted Sub-concepts:
• TaxoRankConstruct: 79.2%.
• WordNet: 68.9%.

Human Evaluation and Examples

Examples of Results

Examples of Results

Examples of Results

Examples of Results

Conclusions

Key Contributions:

• Taxonomical Ranks: Developed a rank-based system for precise and clear hierarchies.
• Multi-Taxonomies: Introduced multitaxonomies to represent concepts through multiple hierarchical trees, accommodating diverse perspectives.
• Linguist/Expert Definitions: Combined linguistic and expert perspectives for rich, context-aware classifications.
• Few-Shot Post-Processing: Ensured coherence and prevented domain shift in subconcept generation.
• Experiments with Human validation

Future Work

Ongoing Developments

• Insights from Human Evaluation and experiments:
• Improved attribute-based generation.
• Enhanced validation mechanisms.
• New version shows better results.
• Results will be published soon.

​

​

Technical Details

GitHub and Implementation

• Open-source code available on GitHub: https://github.com/supersokol/TaxoRankConstruct.
• Easy to run with clear instructions.

​

Q&A

Thank you!

My contact information and GitHub link to the project repository:

• My Email: supersokol777@gmail.com
• My LinkedIn: https://www.linkedin.com/in/supersokol/
• Repo link: https://github.com/supersokol/TaxoRankConstruct

My ferrets: