Text2KGBench-LettrIA

​

A Refined Benchmark for

​

Text2Graph Systems

Julien Plu, Oscar Moreno Escobar, Edouard Trouillez, Axelle Gapin, Pasquale Lisena, Thibault Ehrhart, Raphaël Troncy

LettrIA, Paris, France

EURECOM, Sophia Antipolis, France

2

set of documents

ontology

LLM

populated KG

RQ: how much a LLM can populate a KG solely based on a set of text documents ?

Motivation

The Rise of Text-to-Knowledge-Graph (Text2KG)

• Context: Large Language Models (LLMs) are revolutionizing automated Knowledge Graph (KG) construction from unstructured text.�
• Importance: The KG-LLM synergy is crucial for applications like:
• Explainable AI (XAI)
• Neuro-symbolic fact-checking
• Advanced search and reasoning�
• The Challenge: Evaluating model performance reliably is essential for progress, but existing benchmarks have significant limitations.

3

Motivation

Text2KGBench

4

State-of-art

• Presented at ISWC 2023
• Benchmark composed of
• TekGen corpus�13k+ sentences, 10 ontologies
• WebNLG corpus�4860 sentences, 19 ontologies
• Repo: github.com/cenguix/Text2KGBench

Mihindukulasooriya, N., Tiwari, S., Enguix, C.F., Lata, K. (2023). Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text. In: The Semantic Web – ISWC 2023. Lecture Notes in Computer Science, vol 14266. Springer, Cham. https://doi.org/10.1007/978-3-031-47243-5_14

Our focus!

Critical Flaws in Text2KGBench

5

Ontological Flaws

⨯ Flat Structure�No formal class hierarchy (rdfs:subClassOf).

⨯ Semantic Ambiguity

Out-of-domain concepts and overly

generic properties (e.g., location).

⨯ Lack of Rigor

No formal distinction between object and datatype properties.

Annotation & Data Quality Flaws

⨯ Inconsistency�No standardized format for entities or literal values (dates, numbers).

⨯ Lack of Grounding

Annotations often relied on external knowledge.

Structural & Technical Flaws

⨯ Poor Usability�Missing metadata, undocumented ontologies, and complex URIs

Problem

Text2KGBench-LettrIA

6

A substantially revised and curated benchmark

Our Solution

Systematic�Ontology Refinement

Revised 19 domain ontologies to enforce hierarchical structure (subClassOf) and formal typing.

�Complete Re-Annotation

Re-annotated all 4,860 sentences, yielding over 14,000 high-fidelity, text-grounded triples.

�Enriched Data Format

Introduced enhanced metadata and explicit typing to ensure reproducibility and support multifaceted evaluation.

~1000 hours of human labor

Ontology Refinement

7

Methodology

Ontology Statistics

8

Computing Diff

21.80% fewer classes

​

37.81% fewer properties

Rigorous Annotation Guidelines

9

{

"sub":"University_Bath",

"rel":"staff",

"obj":"2900"

}

{

"sub":"University of Bath",

"subType":"University",

"rel":"academicStaff",

"obj": 2900 ,

"objType":"integer"

}

Methodology

Knowledge Graph Statistics

10

Computing Diff

Number of sentences and triples for each benchmark��Split into�test / train / train-ext for T2KGBench-LettrIA

​

New extended train dataset (synth. generated)

Experimental Setup

11

Evaluation

Anthropic�Claude 3, 3.5, 3.7�Haiku, Sonnet, Opus

Proprietary Models

Zero-Shot

Extended

Augmented training set with synthetic data�+500 sentences / ontology

Generalization

Leave-one-ontology-out setting (city ontology)

Open-Weights Models

Fine-Tuned

​

Google�Gemini 2.0, 2.5�Flash Lite, Flash, Pro

OpenAI�GPT-4.1, GPT-4o�Full, Mini, Nano

Gemma 3�4B, 12B, 27B

Mistral Small 3.2�24B

Phi-4�14B

Qwen 3�from 0.6B to 32B

Classic

Trained on the new, curated training set (SFT, LoRA)

FINE-TUNING CONFIGURATION

Key Result 1: Fine-Tuning is Paramount

12

Evaluation

Smaller, fine-tuned open-weights models massively outperform larger, proprietary models.

High-quality, schema-aligned training data is more critical than raw model size.

Key Result 2: Generalization, Reliability and Efficiency

13

Evaluation

• Fine-tuning teaches true generalization, not just memorization.
• Fine-tuned models maintain their performance advantage even on an unseen ontology.
• gemma-3-12b-it (F1 0.8376) significantly outperforms the best proprietary model, claude-sonnet-4 (F1 0.7829).
• Fine-tuned models achieve near-perfect (> 99%) structural validity and have extremely low hallucination rates.
• Proprietary models show variable reliability; some are strong (gemini-2.5-pro), while others fail catastrophically.
• Efficiency (in terms of latency) varies by orders of magnitude�[2s (gemini-flash) - 27s (claude-opus)] versus <0.02s (fine-tuned) models�which is directly correlated with cost

A Fundamental Trade-Off

14

• Specialization Wins�For complex, structured tasks like Text2KG, task-specific fine-tuning is the most effective strategy.�
• Fine-Tuning Teaches Generalization�The process instills a deep, transferable understanding of the task’s logic.�
• Accessibility vs. Efficiency
• Proprietary APIs�Invaluable for rapid prototyping. Higher latency and pay-per-call costs.
• Fine-Tuned Models�A strategic investment. Delivers superior performance, speed, and economics for high-volume applications.

Lesson Learned

15

Meet �Perseus

16

https://www.lettria.com/benchmarks/benchmark-empowering-graph-based-agents-with-lettrias-new-model

17

Conclusion and Future Work

• Introduced Text2KGBench-LettrIA, a revised benchmark for more precise evaluation of Text2KG systems.
• Key Finding: Smaller, fine-tuned open models can outperform larger, proprietary counterparts. This underscores the pivotal role of high-quality data and model specialization.
• Future Work
• Post-Hoc Alignment�Using RL (PPO/DPO) to reduce hallucinations.
• Explainability & Reasoning�Building a reasoning layer on top of extraction models.
• Context Window Extension�Applying fine-tuned models to larger documents
• Ontology�Populate a knowledge graph following more complex schema patterns (e.g. events based to go beyond binary relationships)

Thank you!

raphael.troncy@eurecom.fr

bit.ly/Text2KGBench-LettrIA