Hands-On-Tutorial: Neurosymbolic AI and the Logic-Knowledge Graph Spectrum
#AIISC, http://aiisc.ai
Yuxin Zi, Kaushik Roy, Krishnaprasad Thirunarayanan, Amit Sheth
May 5th, 2025
Preliminaries: Logic-based Knowledge Representation and Reasoning
Consider the Text Below:
Knowledge Representation in First-Order Logic (FOL)
Roger Federer (/ˈfɛdərər/ FED-ər-ər, Swiss Standard German: [ˈrɔdʒər ˈfeːdərər]; born 8 August 1981) is a Swiss former professional tennis player. He was ranked as the world No. 1 in men's singles by the Association of Tennis Professionals (ATP) for 310 weeks (second-most of all time), including a record 237 consecutive weeks, and finished as the year-end No. 1 five times. Federer won 103 singles titles on the ATP Tour, the second most since the start of the Open Era in 1968, including 20 major men's singles titles (among which a record eight men's singles Wimbledon titles, and an Open Era joint-record five men's singles US Open titles) and six year-end championships.
“Roger Federer … born 8 Aug 1981”
“… is a Swiss former professional tennis player”
“ranked world No. 1 … for 310 weeks … incl. 237 consecutive”
Born(Roger_Federer, 1981-08-08)
Nationality(Roger_Federer, Swiss)
Profession(Roger_Federer, TennisPlayer)
FormerProfessional(Roger_Federer)
WorldRank(Roger_Federer, ATP, MensSingles, 1, 310)
English fragment
FOL fact (predicate-style)
Preliminaries: Logic-based Knowledge Representation and Reasoning
Knowledge Representation in First-Order Logic (FOL)
Queries for the Reasoning Engine in First-Order Logic (FOL)
“Roger Federer … born 8 Aug 1981”
“… is a Swiss former professional tennis player”
“ranked world No. 1 … for 310 weeks … incl. 237 consecutive”
Born(Roger_Federer, 1981-08-08)
Nationality(Roger_Federer, Swiss)
Profession(Roger_Federer, TennisPlayer)
FormerProfessional(Roger_Federer)
WorldRank(Roger_Federer, ATP, MensSingles, 1, 310)
English fragment
FOL fact (predicate-style)
Preliminaries: Logic-based Knowledge Representation and Reasoning
Queries for the Reasoning Engine in First-Order Logic (FOL)
Indexed Reasoning
𝛳-Subsumption Reasoning
What about Resource Consumption During Index Creation?
Index Evolution: Resource Description Frameworks (RDFs)
Index Evolution: Unified Medical Language (UMLS)
Index Evolution: Labeled Property Graphs (LPGs)
vs.
Index Evolution: Conceptnet
{
"@id": "/a/[/r/UsedFor/,/c/en/example/,/c/en/explain/]",
"dataset": "/d/conceptnet/4/en",
"end": {
"@id": "/c/en/explain",
"label": "explain something",
"language": "en",
"term": "/c/en/explain"
},
"license": "cc:by/4.0",
"rel": {
"@id": "/r/UsedFor",
"label": "UsedFor"
},
"sources": [
{
"activity": "/s/activity/omcs/omcs1_possibly_free_text",
"contributor": "/s/contributor/omcs/pavlos"
}
],
"start": {
"@id": "/c/en/example",
"label": "an example",
"language": "en",
"term": "/c/en/example"
},
"surfaceText": "You can use [[an example]] to [[explain something]]",
"weight": 1.0,
}
Why Cannot we Index FOL the Same Way?
Neurosymbolic Methods
Knowledge Graph (Labeled Nodes and Edges)
NeuroSymbolic Reasoning
System 2
Neural Network and Deep Learning
Decisions/Actions
System 1
Low-level Data
Sensors, Text, Image, and Collection
Symbolic Explicit Knowledge Representation
Neural Implicit/Parametric Knowledge Representations
Expert Human
Neurosymbolic Methods Lanscape
The two primary types of neurosymbolic techniques—lowering and lifting—can be further divided into four sub-categories. Across the low (L), medium (M), and high (H) scales, these methods can be used to provide a variety of functions at both algorithmic and application levels.
What is Semantics?
What is Semantics?
What is Semantics?
Example Semantic Interpretations constructed either from manual effort (A, B, C), automatically (D, E), or semi-automatically (F).
(A) is empathi ontology designed to identify concepts in disaster scenarios (Gaur et al. 2019).
(B) Chem2Bio2RDF (Chen et al. 2010).
(C) ATOMIC (Sap et al. 2019).
(D) Education Knowledge Graph by Embibe (Faldu et al. 2020).
(E) Event Cascade Graph in WildFire (Jiang et al. 2019).
(F) Opioid Drug Knowledge Graph (Kamdar et al. 2019)
Symbolic AI
Semantic Interpreter
Map raw data to useful features
Reasoner
Use rules of inference to infer targets from features
Downstream
Perform downstream task
—-----
—-----
—------
Semantic Interpreter
Map raw data to useful features
Reasoner
Use rules of inference to infer targets from features
Target: Return search results
Inference rule: If keyword match exists return page
—-----
—-----
—------
Keywords Obtained Using TF-IDF Vectorization, for example.
—-----
Search Results
Neural AI
Semantic Interpreter
Map raw data to useful features
Reasoner
Use rules of inference to infer targets from features
—-----
—-----
—------
Raw Big data + Minimal Target Demonstrations
Semantic Interpreter
Map raw data to useful features
Reasoner
Use rules of inference to infer targets from features
Target Prediction
with probability score
Visual Information processing
- object classification, object segmentation
Natural language processing
- part-of-speech tagging, constituency parsing
Long List of Successes!!
Downstream
Perform downstream task
Search Results
Neurosymbolic AI
Semantic Interpreter
Map raw data to useful features
Reasoner
Use rules of inference to infer targets from features
—-----
—-----
—------
Semantic Interpreter
Map raw data to useful features
Heterogeneous
Entities (nodes) and Relationships (directed edges)
Reasoner
Use rules of inference to infer targets from features
Target: Return search results
Inference rule: If path exists between nodes through certain intermediate nodes
Search Graph
Downstream
Perform downstream task
Search Results
Process-Triggered Neurosymbolic AI
Semantic Interpreter
Map raw data to useful features
Reasoner
Use rules of inference to infer targets from features
—-----
—-----
—------
Semantic Interpreter
Map raw data to useful features
Heterogeneous
Entities (nodes) and Relationships (directed edges)
Reasoner
Use rules of inference to infer targets from features
Target: Return search results
Inference rule: If path exists between nodes through certain intermediate nodes
Downstream
Perform downstream task
Search Results
Search Graph
Process Trigger
If search non-toxic
Background Knowledge
Apple
is_a
Fruit
Grape
has
Antioxidants
Watermelon
Apple
Grape
Watermelon
Apple
Grape
Watermelon
| | | |
| 1 | 1 | 0 |
| 1 | 1 | 0 |
| 0 | 0 | 1 |
1. Antioxidants has-1 Apple is_a Fruit
2. Antioxidants has-1 Grape is_a Fruit
3. Watermelon is_a Fruit
Inductive Bias-level Structured Knowledge Compression
Apple
Grape
Watermelon
Representation-level Structured Knowledge Compression
Structured Knowledge Compression Methods
The figure illustrates two methods for compressing knowledge graphs to integrate them with neural processing pipelines. One approach involves embedding knowledge graph paths into vector spaces, enabling integration with the neural network’s hidden representations. The other method involves encoding knowledge graphs as masks to modify the neural network’s inductive biases. An example of an inductive bias is the correlation information stored in the self-attention matrices of a transformer neural network
Knowledge Graph Compression - Vectors
Data
Through Embeddings
Flexible
Open-Ended
Advantages
Less Manual Effort,
Scalable,
Shows Modest Improvement
Lossy Compression of Graph Information (World Model)
DisAdvantages
Explainable, Consistent, Safe, Interpretable
Knowledge graph embedding: A survey from the perspective of representation spaces.
ACM Computing Surveys 56.6 (2024)
Knowledge Graph Compression - Matrices
Kang, M., Baek, J., & Hwang, S. J. (2022, July). KALA: Knowledge-Augmented Language Model Adaptation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 5144-5167)., link
Flexible
Open-Ended
Less Manual Effort,
Scalable,
Shows Modest Improvement
Knowledge Integration via Tool Calling - Wolfram
An empirical study on challenging math problem solving with gpt-4
arXiv preprint arXiv:2306.01337 (2023)
Flexible
Symbolic Parts Robust and Interpretable
Open-Ended
Neural Parts Robust Under Right Interpretation
Knowledge Integration with Trainable Parts
External Knowledge
Data
Flexible
World Model
Open-Ended
Advantages and Disadvantages
Explainable, Consistent, Safe, Interpretable
Elicit
Graph
Compare, and Fix
Contextualized Sub-graph
Knowledge Integration with Trainable Parts
Inputs (Question and Video Frames) and Outputs (Answer(s))
Example Explanation Graph
Neural Network
Plug in a guidance module that allows us to guide what the neural network focuses on, i.e., the “relevant” spatio temporal events in the scene, during output generation
CEG Guidance Module
Causal Event Graph-Guided Language-based Spatiotemporal Question Answering�AAAI Spring Symposium on Empowering Machine Learning and Large Language Models
with Domain and Commonsense Knowledge 2024.
Scene Ontology
Knowledge Integration End-to-End Trainability
Process Knowledge-infused Learning for Clinician-friendly Explanations
AAAI Symposium on Human Partnership with Medical AI 1 (1), 154-160, 2023
Flexible
Process Model
Open-Ended
Explainable,
Consistent,
Safe
Interpretable
Takeaway: What is the Sweet Spot?
Logic → to → Ontology → Labelled Property Graphs → + Neural Networks