1 of 18

Knowledge Graphs to facilitate Domain Adaptation?

A biomedicine study case

February 23, 2024

Edouard Albert-Roulhac &

Abdelhakim Sehad

2 of 18

What is a Knowledge Graph (KG)?

Represents a network of real-world entities—i.e. objects, events, situations, or concepts—and illustrates the relationship between them. (Source : IBM)

3 of 18

How do Knowledge Graphs work?

A knowledge graph works by representing knowledge in a structured format using nodes, edges, and properties.

Knowledge graphs often integrate data from multiple sources.

Once the knowledge graph is constructed, it can be queried and analyzed using graph-based query languages or APIs.

By organizing knowledge in a structured and interconnected way, knowledge graphs enable more efficient data access, retrieval, and analysis across various domains.

4 of 18

Applications of KG

Source : https://www.researchgate.net/publication/340534222_A_Survey_on_Application_of_Knowledge_Graph.

5 of 18

Why are KG’s valuable in biomedicine and healthcare?

6 of 18

Why Knowledge Graphs in healthcare?

Biomedical data is vast and intricate, encompassing genes, proteins, diseases, drugs, and countless other entities.

They can integrate patient-specific data, such as genetic information and medical history.

7 of 18

Research question

Can Multimodal Knowledge Graphs enable domain adaptation?

8 of 18

Relevant work

Knowledge Graph Embeddings in the Biomedical Domain: Are They Useful? A Look at Link Prediction, Rule Learning, and Downstream Polypharmacy Tasks — August, 2023.

(https://arxiv.org/pdf/2305.19979.pdf)

9 of 18

What is BioKG?

Biomedical Knowledge Graph that aims to aggregate a wide range of entity and link types within the biomedical domain.

2,067,998 entries across 17 relations.

Does not explicitly track the directionality of relationships.

10 of 18

Their approach

The performance of the knowledge graph embedding (KGE) models was evaluated in the biomedical domain by applying state-of-the-art KGE models to BioKG.

The evaluation included assessing their performance and potential downstream uses.

Best-performing model was evaluated on tasks representing real-life polypharmacy situations

11 of 18

Some of the models

TransE

TransR

ComplEx

RotatE

12 of 18

Lack of Multimodal Knowledge Graphs

Multimodal Learning on Graphs for Disease Relation Extraction

(https://arxiv.org/pdf/2203.08893.pdf)

Introduces REMAP to build a high quality Knowledge Graph of relations between diseases

When Radiology Report Generation Meets Knowledge Graph

(https://arxiv.org/pdf/2002.08277.pdf)

Builds a Knowledge Graph specialized in radiology

These methods build KGs from image & text datasets but the resulting graph is not multimodal

13 of 18

Dataset

Chest X-Ray reports

Generate reports from X-Ray images

Variants include other tasks for chest X-Ray:

Image multi-label classification
KG construction

14 of 18

Experiments & Results

RaDialog results on the Chest X-Ray Report benchmark (SOTA)

CE: clinical efficacy

BS: BertScore

B-i: Bleu

MTR: METEOR

R-L: Rouge

Source: https://arxiv.org/pdf/2311.18681.pdf

15 of 18

Radiology Report Generation

RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance

Feed results from a multi-label feature extractor to the LLM

This results in little text information

Source: https://arxiv.org/pdf/2311.18681.pdf

16 of 18

Approach

Our approach is to incorporate knowledge from a graph to the LVM

Use a general purpose Vision-Language model, BLIP2

map image from a radiology vision encoder
feed KG enhanced information

or use the KGE to feed more related words to the model
either map learnt specialized KG embeddings

Fine-tune the model using LoRA

KG embedding model will be used on top of the feature extractor

The intuition

leverage specialized knowledge (using KG)
give the LVM more information to work with in the report
general purpose large LVMs should not encode specialized information

17 of 18

Timeline

By March 10th

Reproduce the RaDialog method
Reproduce the Knowledge Graph Embedding method

By March 30th

Train the adapter to feed KG embeddings to the BLIP2 model
LoRA fine-tuning

By April 10th

Debugging
Evaluate on Radiology Report benchmarks