Structuring: Knowledge Structuring to Help Retrieval and Augmented Generation
Pengcheng Jiang, Siru Ouyang, Yizhu Jiao,
Ming Zhong, Runchu Tian, Jiawei Han
Department of Computer Science
University of Illinois at Urbana-Champaign
August 3, 2025
1
Outline
2
Outline
3
Outline
4
Text Classification
Sentiment Analysis
Location Prediction
News Topic Classification
Paper Topic Classification
Email Intent Identification
Hate Speech Detection
5
Different Text Classification Settings: �Single-Label vs. Multi-Label
6
Different Text Classification Settings: �Flat vs. Hierarchical
7
Weakly-Supervised Text Classification: Motivation
8
Weakly-Supervised Text Classification: Definition
?
9
General Ideas to Perform �Weakly-Supervised Text Classification
10
11
LOTClass: Label-Name-Only Text Classification
11
LOTClass: Meaning of Word Is Context-Dependent
12
LOTClass: Contextualized Word-Level Topic Prediction
13
LOTClass: Experiment Results
14
14
PIEClass: Prompt-based Fine-tuning for �Text Classification
Zhang, Y., Jiang, M., Meng, Y., Zhang, Y., & Han, J. “PIEClass: Weakly-Supervised Text Classification
with Prompting and Noise-Robust Iterative Ensemble Training”, EMNLP’23.
15
PIEClass: Integrating Head Token �& Prompt-based Fine-tuning
16
PIEClass: Experiment Results
17
TELEClass: Taxonomy Enrichment and LLM-Enhanced�Hierarchical Text Classification with Minimal Supervision
18
TELEClass: Performance Study and Cost for Text Classification
Yunyi Zhang, et al., “TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervision”, WWW’25
19
Outline
20
Entity Structure Mining
21
21
OntoType : Three Steps
22
OntoType: Step 1 - Candidate Type Generation
Four Hearst Patterns give the highest quality hypernyms with simple type mapping on the OntoNotes dataset
23
OntoType: Steps 2 & 3- High-Level Type Resolution & Progressive Type Refinement
24
OnEFET: Ontology Enrichment for FET
Usually organized as a structure — ontology
Siru Ouyang, et al., “Ontology Enrichment for Effective Fine-grained Entity Typing”, KDD’24
25
Overall Framework of OnEFET : Three Steps
26
OnEFET: Step 1 – Ontology Enrichment
[1] Zhang, Yu, et al. "Seed-guided topic discovery with out-of-vocabulary seeds." NAACL 2022.
[2] Jiao, Yizhu, et al. "Open-vocabulary argument role prediction for event extraction." EMNLP 2022 Findings.
[3] Zhang, Yu, et al. "Entity Set Co-Expansion in StackOverflow." Big Data 2022.
27
OnEFET: Steps 2 & 3 – Coarse-to-fine Typing
28
OnEFET: Performance Study
29
Open-Vocabulary Relation Type Discovery
Yizhu Jiao, Sha Li, Yiqing Xie, Ming Zhong, Heng Ji and Jiawei Han “Open-Vocabulary Argument Role Prediction for Event Extraction”, EMNLP’22
30
Framework for RolePred
Entity Type
Candidate Relation Type
Relation Types
Candidate Relation Types
31
Experiment: Relation Type Discovery
Relation Type Prediction
Relation Extraction w/o Golden Roles
Example of the generated relation types
Extracted Results by RolePred and baselines
32
Chemical Reaction Extraction with Weak Supervision
Ming Zhong, Siru Ouyang, Minhao Jiang, Vivian Hu, Yizhu Jiao, Xuan Wang, Jiawei Han, “ReactIE: Enhancing Chemical Reaction Extraction with Weak Supervision”, ACL’23 Findings
33
Chemical Reaction Extraction with Weak Supervision
34
Chemical Reaction Extraction with Weak Supervision
35
ZOES: Zero-shot Open-Schema Entity Structure Discovery
Xueqiang Xu, Jinfeng Xiao, James Barry, Mohab Elkaref, Jiaru Zou, Pengcheng Jiang, Yunyi Zhang, Max Giammona, Geeth de Mel, and Jiawei Han. "Zero-shot open-schema entity structure discovery", arxiv-2506.04458, 2025.
Mutual Dependency Principle: For a triplet t = ⟨e, a, v⟩, appropriate granularity is achieved when any one component can be reliably inferred fr. the other two within the context d
36
ZOES: Methodology Overview
Induce root attribute (e.g., discharge capacity): abstract over semantically similar attrs to further guide the triplet enrichment (i.e., embed all extracted attrs using a dense encoder and cluster them based on semantic similarity―This clustering step can group attributes that express the same underlying general attribute)
Value-Anchored Enrichment: Once root attributes are identified, use them to guide the discovery of additional value mentions. For each root attribute, prompt the LLM to revisit the document and list all corresponding values.
Some entities may lack explicitly stated attr.-value structures. But each semantically meaningful value (e.g., “80.6%”) should correspond to at least one valid triplet, and is treated as an anchor to elicit a missing triplet
37
ZOES: Performance Study
Example extracted results on a doc from the Economics domain for “Toyota,” using Granite-8B
Ablation results on the Finance domain using GPT-4o as the backbone
Dataset statistics across “Battery Science”, “Finance”, and “Politics” domains
Evaluation with user interested entity types across different backbone models & methods
38
Outline
39
TAGREAL: Pattern Mining for Prompt Generation
40
Textual Pattern Mining for Prompt Generation (I)
where wi,j is the weight of jth prompt for ith relation
All weights are learned from PLM
41
Textual Pattern Mining for Prompt Generation (II)
42
Example of Link Prediction with TAGREAL
Man denotes manual prompt. Optim denotes optimized prompt ensemble. Supp denotes support information. The ground truth tail entity is in red/yellow, helpful information in green and optimized prompts in blue
43
Performance of TagReal: Knowledge Graph Completion
Performance Variation of F1-score
Relation-wise Hits@10 on FB60K-NYT10
44
Evaluation Bottleneck of Generative KG Construction in the LLM Era
*GRE: Generative Relation Extraction (by LLMs)
Ideal metrics for GRE should be able to evaluate :
45
GenRES: A Comprehensive Evaluation Framework for Generative RE
46
GenRES: A Comprehensive Evaluation Framework for Generative RE
Four workers died in a massive oil rig fire that raged for hours off the coast of Mexico Wednesday. Mexican state oil company Pemex said 45 workers were injured in the blaze, which began early Wednesday morning. Two of them are in serious condition, the company said. Authorities evacuated about 300 people from the Abkatun Permanente platform after the fire started, Pemex said. At least 10 boats worked to battle the blaze for hours. The fire had been extinguished by Wednesday night, Pemex said in a Twitter post. The company denied rumors that the platform had collapsed and said there was no oil spill as a result of the fire. The state oil company hasn't said what caused the fire on the platform, which is located in the Gulf of Mexico's Campeche Sound. The fire began in the platform's dehydration and pumping area, Pemex said. CNN's Mayra Cuevas contributed to this report.
[Four workers | were died in | oil rig fire],
[45 workers | were injured in | the blaze],
[Two workers | are in | serious condition],
[300 people | were evacuated from | the platform],
[The fire | had been extinguished by | Wednesday night],
[The fire | did not result in | oil spill].
Generative Relation Extraction
Triples
Text
Topical Distribution
KL-Divergence
Topical
Similarity Score
Latent Topics
Latent Topics
Topical Similarity Score (TS)
“How much content of the source text are covered by the relationships extracted (by comparing triples* to the source text)”
47
GenRES: A Comprehensive Evaluation Framework for Generative RE
Evaluate the factualness of an extracted relationship (triplet) based on the given source text. Indicate whether the relationship accurately reflects the information in the source text by responding with "true" or "false".
You should only output "true" or "false" with no additional information.
Example 1:
Source Text: The Great Barrier Reef, located off the coast of Australia, is the world's largest coral reef system. It has been severely affected by climate change, leading to coral bleaching.
Relationship: ["Great Barrier Reef", "affected by", "climate change"]
Factualness: true
Example 2:
Source Text: The Eiffel Tower was constructed in 1889 and is located in Paris, France. It is one of the most recognizable structures in the world.
Relationship: ["Eiffel Tower", "located in", "London"]
Factualness: false
Example 3:
Source Text: The novel "Moby-Dick" by Herman Melville features a ship named Pequod. The narrative follows the ship and its crew in their pursuit of a giant white sperm whale.
Relationship: ["Moby-Dick", "is about", "a whale named Pequod"]
Factualness: false
Source Text: $TEXT$
Relationship: $TRIPLE$
Factualness:
“How factual the extracted triples are, referring to the source text (by factualness verification treating source text as the “knowledge base”)”
48
GenRES: A Comprehensive Evaluation Framework for Generative RE
Granularity-checking prompt:
“How atomic the extracted triples are (by asking LLM to split each triple)”
49
GenRES: Robustness and Human Preference Alignment Results
50
PriORE: Open Relation Extraction with a Priori Seed Generation
51
Methods Outlined with An Example
52
PrioRE: Experiment Results
53
Case study: Inheritance_type_of
54
Embedding-Based KG Construction – KG-FIT
Up-to-Date Global Knowledge
Fast-Iterating LLMs
Global Knowledge
Local Knowledge
Small-scale PLMs
Structure-based Methods
Pros over PLMs/LLMs:
Pros over Structure-based Methods:
Pros over PLMs:
Can we combine them?
55
Embedding-Based KG Construction – KG-FIT
56
KG-FIT: Experimental Results
Metrics:
Mean Rank (MR):
Mean Reciprocal Rank (MRR):
Hits@N:
Findings:
Datasets:
57
KG-FIT: Embedding Visualization
HAKE
KG-FIT preserves both local and global semantics!
“CAA”
Hits@1022
“Exertional dyspnea”
Hits@981
“CAA”
Hits@45
“Exertional dyspnea”
Hits@26
“CAA”
Hits@12
“Exertional dyspnea”
Hits@5
58
References
59
59
References
60
60