Shades of Knowledge-infused Learning

Outline

• Motivation for Knowledge Infusion
• Task and Challenges
• Strategy
• Scenarios

• Introduction to Knowledge Graphs

• Types of Knowledge-Infused Learning and their Real-world Applications

• Resources

• Future Research Directions

3

Role of Knowledge-enabled AI

Information Retrieval

Natural Language Understanding

Conversational Assistance

Summarization

​

Learning to Rank/ Re-Ranking

Recommender System

4

Recommendation

Summarization

Matching

Conversational Assistance

Efforts towards Knowledge-enabled Artificial Intelligence (AI)

5

[Stanford Knowledge Graph Seminar 2020, Christopher Re]

Ananthram et al. (Traffic data annotation) In TIST, 2015

AI in Recommender Systems

6

• Task
• Large dataset containing counts of likes and dislikes between a User and Item.
• Item-Item dataset for characterizing similarity between items
• State of the art models are embedding-based and focus on optimal placing of entities in numeric vector space.

​

​

• Challenges:
• ​
• Can a recommender system capture human belief implicit in dataset?
• Patients switching doctors is an example of change in human belief

http://bit.ly/netflix_recsys

AI in Recommender Systems

7

​

• Challenges:
• Ignores contextual features
• Ignores guidelines, that act as ground truth for matching
• Harder to match users with opposite characteristics behavior
• Support Seeker and Support Provider
• Explainability in recommender system using implicit knowledge has not be explored

AI in Learning to Rank / Re-Ranking

8

• Task
• Given a query, rank the documents by relevance through latent topics.
• State of the art models: BERT, RoBERTa, ELECTRA

​

​

• Challenges:
• Heavily relies on the co-occurrences of pair of words.
• Difficult to rank document/content when query is about an emerging topic (e.g. long tail entities).
• Hard to explain the ranking of the retrieved documents/content as it relies on latent topics.

​

AI in Summarization

9

• Task
• Requires large corpus and reference summaries to train the summarizer

​

• State of the art (SOTA) focuses on maximizing coverage/completeness and minimizing redundancy.

• Challenges:
• Hard to model knowledge constraints in summarizer.
• Hard to make summarizer focus on terms that are relevant to the end-user.
• Difficult to exploit relationships between selected sentences using SOTA.
• Therefore fails to generalize over tasks.

AI in Conversational Assistance

10

• Task
• Existing state of the art transformer-based language models have shown significant improvements:
• General Language Understanding Evaluation (GLUE) tasks
• Question Generation Tasks

• Challenge
• Knowledge Intensive Learning Tasks
• Tasks concerning with:
• Information Seeking
• Information Curiosity
• Conversational Assistance

AI in Conversational Assistance

11

• Challenge -- Information Curiosity
• Example:
• Question asked: More than half of the week, I feel trouble in relaxing
• Definition of “Trouble in Relaxing”
• Concept Flow of Questions from Agent
• Q1: What medications have you tried?
• Q2: How many medications have you tried?
• Q3: Are you feeling nauseous?
• Q4: Do you get 8 hours of sleep?

Identifying the Aspect and Theme of the question and creating an order of information seeking question --- requires knowledge infusion

With Background Knowledge Infusion

• Fine tune language models by using machine readable knowledge structures.

• Leverage language models with knowledge context for contextualizing the input

• Abstraction help AI models work efficiently with limited labeled data

Such a process develop explainable systems that generalize over tasks in Information retrieval and natural language understanding

• Identifying long tail entities and use them to guide abstraction of the input
• Attention on long-tail entities is often ignored in transformer-based language models
• Abstraction preserve textual semantics

12

Gaur, Manas, Keyur Faldu, and Amit Sheth. "Semantics of the Black-Box: Can knowledge graphs help make deep learning systems more interpretable and explainable?." IEEE Internet Computing 25, 2021

Brown, Tom B., et al. "Language models are few-shot learners." arXiv preprint arXiv:2005.14165 (2020).

​

Knowledge-infused Learning

Can incorporation of knowledge enhance performance and explainability of data-intensive learning models?

​

13

Sheth, Amit, et al. "Shades of knowledge-infused learning for enhancing deep learning." IEEE Internet Computing 2019

Gaur, Manas, Keyur Faldu, and Amit Sheth. "Semantics of the Black-Box: Can knowledge graphs help make deep learning systems more interpretable and explainable?." IEEE Internet Computing 25, no. 1 (2021): 51-59.

14

Multi Hop

Domain Specific

Low Resource

Symbolic Knowledge

What can cause a forest fire?

Static electricity can cause sparks

Sparks can start a forest fire

Need some task-specific knowledge (implicit/explicit) to apply generic AI models in downstream applications

Used rules as constraints (presence/absence, similarity thresholding, correlations, etc.)

Limited labeled data, Difficult to create gold standard

15

Role of Knowledge-enabled AI

Information Retrieval

Natural Language Understanding

Conversational Assistance

Summarization

​

Learning to Rank/ Re-Ranking

Recommender System

Multi Hop

Domain Specific

Symbolic Knowledge

Low Resource

Domain Specific

Symbolic Knowledge

Domain Specific

Low Resource

Multi Hop

Low Resource

Symbolic Knowledge

Scenario 1a: Trivial Case of Question Answering

16

Given a Context: I sometimes wonder how many alcoholics are relapsing under the lockdowns

Question asked: Does the person have addiction?

Response from a Seq2Seq Model: Yes

co-occurrence

Any Seq2Seq Model trained on these dataset

Multi Hop

Symbolic Knowledge

Scenario 1b: Non-Trivial Case of Question Answering -- Negation

17

Given a Context: Then others that insisted that what I have is depression even though

manic episodes aren’t characteristic to depression. I dread having to retread all this again because the clinic where I get my mental health addressed is closing down due to loss in business caused by the pandemic

Question asked: Does the person have Depression?

Response from a Seq2Seq Model: Yes

Bipolar Disorder

Manic episodes

has_a

co-occurrence

Multi Hop

Symbolic Knowledge

Scenario 2 : Recommending Support Providers to Support Seekers

18

Corona vaccine suggest that we do not meet anyone besides people from the house. As an extrovert human being, I don’t buy this.

I feel fucking alone, lonely, and my depression has started trigger warning. I’m afraid, but I might become suicidal. First COVID19 and now its vaccine. Please any advice

Support Seeking

(SS)

User

Domain Specific

Low Resource

Symbolic Knowledge

19

Corona vaccine suggest that we do not meet anyone besides people from the house. As an extrovert human being, I don’t buy this.

I feel fucking alone, lonely, and my depression has started trigger warning. I’m afraid, but I might become suicidal. First COVID19 and now its vaccine. Please any advice

SS User

Do not let your mind slip to some extreme conditions in mental health. Listen to podcasts, read books, facetime your friends.

Lockdown is temporary and administration of vaccine in process. I am

here to support you all.

Supportive

SP (Support Provider)

Matched Support Provider to Support Seeker

Domain Specific

Low Resource

Symbolic Knowledge

20

Corona vaccine suggest that we do not meet anyone besides people from the house. As an extrovert human being, I don’t buy this.

I feel fucking alone, lonely, and my depression has started trigger warning. I’m afraid, but I might become suicidal. First COVID19 and now its vaccine. Please any advice

Do not let your mind slip to some extreme conditions in mental health. Listen to podcasts, read books, facetime your friends.

Lockdown is temporary and administration of vaccine in process. I am

here to support you all.

Supportive

SP (Support Provider)

SS User

Concept: Suicide in Lexicon

Human Annotation of Supportive Behavior Online

Depression

Concept: COVID19 in lexicon

Word senses

Domain Specific

Low Resource

Symbolic Knowledge

Research Questions

21

• How to generate contextualized representation using knowledge infusion?

• How to derive semantic preserving representations through abstractions using knowledge infusion?

• How to resolve the discrepancy between knowledge-infused representation and statistically learned patterns?

• How to help model function during uncertainty and explain model behavior ?

22

Corona vaccine suggest that we do not meet anyone besides people from the house. As an extrovert human being, I don’t buy this.

I feel fucking alone, lonely, and my depression has started trigger warning. I’m afraid, but I might become suicidal. First COVID19 and now its vaccine. Please any advice

Do not let your mind slip to some extreme conditions in mental health. Listen to podcasts, read books, facetime your friends.

Lockdown is temporary and administration of vaccine in process. I am

here to support you all.

No advice, just that I am on same boat. extroverted introvert, slipping to suicidality. But, i suggest, Hang in there, I will be doing the same

Supportive

SP

Similar SP

SS User

Stay away from provocative news on COVID19 and vaccine.

Stay free from risk and hold on to faith.

Informative SP

Knowledge Graphs

A Primer

23

1

Knowledge Graph (KG) is a machine readable structured representation of knowledge consisting of entities (entity and entity type) and relationships in various forms (e.g., labeled property graphs and RDFs).

​

24

Examples of

Knowledge Graphs

25

Empathi: Gaur et al. 2018

ATOMIC: Sap et al. 2019

Crisis Event Graph:

Jiang et al. 2019

Chem2Bio2RDF:

Chen et al. 2010

Embibe Education KG:

Faldu et al. 2020

Knowledge-infused Learning

26

2

Theoretical Perspective : Probably Approximately Correct Learning (PAC)

27

28

How do you know that a training set has a good domain coverage?

Robust Classifier → Low Generalizability Error

Consistent Classifier → Low Training Error

Confidence: More Certainty (lower δ) means more number of samples.

Complexity: More complicated hypothesis (|H|) means more number of samples

29

Challenge:

Existing ML Models:

Infusion:

True Data Distribution

Hypothesis Data Distribution

Confidence: More Certainty (lower δ) means less number of samples.

Complexity: Less complicated hypothesis (|H|) means less number of samples

K-iL Types

30

Shallow Infusion

Semi Deep Infusion

Deep Infusion

K-iL Types

31

Shallow Infusion

Multi Hop

Symbolic Knowledge

Low Resource

32

Neural Self Attention

DBPedia

SNOMED-CT

DBPedia: Knowledge Graph constructed from WikiPedia

SNOMED-CT: Systematized Nomenclature in Medicine Clinical Terms

Search through Wikipedia was performed in Jan-Feb 2020

33

Large Scale Knowledge Graphs (e.g. BabelNet)

• BabelNet KG (Python: https://babelnet.io/v5/getVersion?key={key}):
• Crisis Tweet: Waterborne diseases on rise in Texas as hurricane harvey water recedes
• With BabelNet:

​

​

​

​

​

​

​

​

• BabelNet Hypernyms : Hurricane is a geological phenomenon, Earthquake is a geological phenomenon, as a result, at an abstract level there is some commonality among tweets.
• BabelNet Descriptions:
• For providing the context for the Tweets
• Create a parallel corpus of text for training a problem-specific embedding model

​

Named entities

Concept

http://bit.ly/babelnet_AIISC

34

Machine Learning Model

​

​

How well the model learned the task?

Shapley plots on Feature Importances or Dependencies

Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "" Why should I trust you?" Explaining the predictions of any classifier."

Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016.

PT_t = t^th topic/phrase extracted from free form input text

​

KS_c = c^th concept in a knowledge source ( graph, base, ontology, and/or lexicon

Mapping

Definition:

Shallow Infusion

35

Shallow Infusion of knowledge source to improve the semantic and conceptual processing of data before passing into machine learning model

Benefits of Shallow Infusion:

• Since features are concepts from a knowledge source (KS_C ), their importance score explains the model behavior
• Importance scores of KS_C support better human evaluation
• Coverage can be measure on a sub-task or transfer learning task
• The transformed data can reveal model’s sensitivity to a specific phenomenon [Kursuncu et al., In CSCW 2019 and Gaur et al. In WWW 2019]

K-iL Types

36

Shallow Infusion

Semi- Deep Infusion

Multi Hop

Domain Specific

Low Resource

Symbolic Knowledge

KiL: Semi-deep Infusion

37

Deeper and congruent incorporation or integration of the knowledge sources in the learning techniques.

Benefits of Semi-Deep Infusion:

• In comparison with shallow infusion, Semi-Deep Infusion is more focused on model interpretability.
• Explainability can be achieved by combining Shallow and Semi-Deep Infusion
• With Semi-Deep infusion of knowledge , model’s complexity is less [Gaur et al. In JMIR 2021]
• Confidence on the model is high as attention-modulated representations can be inspected [Gaur et al., In CIKM 2018]

38

Machine Learning Model

​

​

39

• Semi-Deep KiL : Initialize a prior attention matrix for the tokens, i.e., how much they should attend to in comparison with other tokens in the sentence
• For example: COVID19 → “unable to pay rent”, 0.92 (attention weight)

• The attention matrix can be learned from data to lean more towards the data likelihood estimate
• The advantage over shallow infusion is that even where a prior isn’t specified, the model is still interpretable by analysis of attention weights

• The prior model can be obtained by random walk probabilities on a knowledge source

40

Prior and posterior distributions can be modeled using

• Variational Autoencoders (VAEs) [Du et al. 2021, Context-aware VAE]
• Generative Adversarial Neural Networks (GANs) [Chang et al. 2019, KG GANs]
• Relational Models [Tresp 2018]
• Or a combination

1. Start with a structured prior P(𝛳)
2. Post seeing some data, X
3. Apply Bayes rule
4. Obtain a posterior P(𝛳|X)
5. Sample 𝛳 from posterior (MCMC)

Or get by optimization (Variational-Inference)

• Use in modeling downstream function f(X,𝛳)

Shallow Infusion

Applications

• Suicide Risk Assessment with Perceived Risk Loss - WWW 2019 + PLoS One 2021
• Knowledge-infused Contrastive Clustering of Support Seekers and Support Providers -- ICHI 2021
• Trust-based Recommender System for Patient-Clinician Matchmaking -- DSAA 2018 + ICHI 2018

41

3

Suicide Risk Assessment with Perceived Risk Loss -

WWW 2019 + PLoS One 2021

42

3a

Domain Specific

Low Resource

43

Problem Domain: Suicide Risk Assessment

​

Given a sequence of posts made by a user with suicidal tendencies, how can we better predict the suicide risk with explainability?

In summary, prior works:

​

Use of classification labels for Suicide Risk Assessment: No Risk, Low Risk, Medium Risk, High Risk

• No appropriate guidelines (e.g. Range)
• No scope of improvement by domain knowledge -- such as Definitions
• Labels do not reflect clinical settings
• Most importantly, clinicians’ disagreement cannot be judged, hence model cannot learn to predict precisely under uncertainty

​

​

44

C-SSRS

Clinical Guideline to identify the Severity level of Suicide Risk

Adapting C-SSRS to Social Media

45

Challenge: We have but the data isn’t generated using

SW

SSH

SLH

BPD

DPR

ADD

SCZ

BPR

ANX

Querying these subreddits

SW

SSH

SLH

BPD

DPR

ADD

SCZ

BPR

ANX

1.0

0.66

0.40

0.40

0.44

0.39

0.30

0.34

Index of tokens relevant to Suicide in SuicideWatch

46

Identifying MH disorder with suicide risk

Contextualization

Outcome of Mapping the Reddit

47

A glimpse of the Dataset

+2800 Downloads

Link: https://zenodo.org/record/2667859#.YKXODx0pATs

Reliability Assessment with 4 Clinicians = 78%

​

→ 22% disagreement

48

prediction

actual

Annotator disagreement

Group of Annotators

Perceived Risk Loss

Calculated as the ratio of agreement of prediction with any of the annotators over the total number of annotators.

​

In cases when the r′ disagrees with all the annotators in G, the risk reducing factor is set to 1

Risk Reducing Factor

49

5-Labels: Supportive, Suicide Indication, Suicide Ideation, Suicide Behavior, Suicide Attempt

(3+1)-Labels: {Supportive, Suicide Indication}, Suicide Ideation, Suicide Behavior, Suicide Attempt

Safe Users/Negative Users

• For CNN under 5-label, there is 14% chance that model will provide an outcome that disagrees with every annotator, whereas SVM-linear has 60% chance of going wrong on PM Measure

• Negative Users category :
• Domain experts showed disagreements in Supportive and Suicide Indication user
• SVM-Linear showed reduced PR score due to Negative category, CNN predicts them ideation

Perceived Risk Measure

50

Time-Variant LSTM+CNN and Time-Invariant CNN

• Not all suicide risk severity levels are time-variant
• Suicide Behavior (SB) : involves action
• Suicide Attempt (SA) : involves action
• Convolutions have the property to capture features that are persistent across different samples
• A user’s post labeled as SB and SA shows repeated mentions of actions (after transitioning from suicide ideation/suicide indication)

Highway Network

51

Tvar. LSTM+CNN

Tinv. CNN

Explainable Evaluation: Behavior

Explainable Evaluation: Behavior

Knowledge-infused Contrastive Clustering of Support Seekers and Support Providers

IEEE ICHI 2021

54

3b

Domain Specific

Low Resource

55

Married with a supportive husband but my serious health issues including depression and PTSD has made me feel as if I am losing everything.

COVID-19 has increased the severity.

Actively seeing a psychologist but still see unstability in life.

I don’t feel like I will survive this. Any piece of advice would save me and my family.

Tough position and i can kind of relate. I don’t think your marriage is dying.

My advice would be to meditate, prioritize, and act patiently.

You have to keep positive, explore news things with family and capture positive influence on mood.

My MDD is affecting my married life.

I am an outdoor enthusiast and so is my husband.

But my health concerns keep pulling him down.

Making both of us miserable. I wanted him to let me go.

Supportive

SP

Similar SP

SS User

TASK

56

Big concept

57

Event-Specific Filtering

Business Closure

School Closure

Lockdown

Shelter-in-place

Hospitalization

​

Domain-Specific

Filtering

​

Anxiety

Depression

​

​

First Person Pronouns

Subordinate Conjunction

Max Height

Max Verb Phrase Length

Syntactic Features

Sentiment and Emotions

Focus Future

Cognitive Features

Biological Features

Psycholinguistic

Gaur 2018, Gkotsis 2017

Linguistic Inquiry and Word Count http://liwc.wpengine.com/

Knowledge on COVID-19 Events, Mental Health and its language, Psycholinguistics

Gaur et al. ICHI 2021

PLoS 2021

WWW 2019

58

Simulation Study: Proof of Concept (better than random assignment)

Probabilistic Greedy: Choosing the SP with highest rating across conversations among Support Seekers (SS) and Support Providers (SP), where each SS rates the SP response during the conversation.

Knowledge-infusion (Features): Features about SP-SS matches in the ideal scenario. For example: Support Provider (SP) having a similar mental health issue to the Support Seeker (SS)

Knowledge-infusion (User Perspectives): Choice of SS user among SPs based on mental health preferences. For example, a Support Seeker (SS), having an issue with a particular type of addiction.

59

• Knowledge and Preferences are centric for effective matching

​

• Questions:
• How to capture knowledge in Reddit posts?

​

• How to capture preferences implicitly?

60

Interest

​

We are interested in Matching Support Seekers with Support Providers

Approach: Convolutional Siamese Network Model with contrastive loss

​

​

​

​

​

​

​

​

​

​

61

Content

F1-Score

Content+

​

Content+ Psycholinguistic+

​

Content+ Psycholinguistic+

COVID-19-Event+

Outcomes influenced by Contextual Features

62

Matching SS having Anxiety with 3 Support providers having experience in helping people with Anxiety

63

8.16/10 agreement with Subject Matter Expert

​

Avg. 6 out of 8 recommendation picked

Trust-based Recommender System for Patient-Clinician Matchmaking

IEEE DSAA 2018 + IEEE ICHI 2018

64

3c

Domain Specific

Low Resource

Multi Hop

Symbolic Knowledge

72 Million Patients Data

​

ICD-10 Knowledge

65

Task: If the dataset is made with labels, , for the purpose of explainability

• There is knowledge bias in labeling
• The data annotated with has an information gap

​

• How to better guide machine learning model with necessary information?
• How to assess model’s behavior in uncertainty when evaluated by experts?

[Gaur et al. In WWW 2019 and In PLoS One 2021]

66

Task: Can we better match Patients with Primary Care Physicians using Trust and knowledge on Disease Severity

​

67

👍

👍

68

👍

X

X

X

X

X

X

X

X

Diagnosis

Severity

Causes

Hyperbolic Poincare Embedding

69

Benefit:

• Model Severity of Disease
• Model trust in Clinicians

​

We were able to recommend over

80% of patients

with relevant primary care doctors

​

compared to just

37% using the heuristic baseline

or

69% using CF without the trust

With Trust and ICD-10

Han et al., In IEEE DSAA 2018 and IEEE ICHI 2018

Semi-Deep Infusion

Applications

• Semantic Encoding And Decoding for DSM-5-based Post-level disorder prediction -- CIKM 2018
• Abstractive Summarization using Integer Linear Programming and Knowledge-based Constraint -- JMIR 2021

70

4

“Let Me Tell You About Your Mental Health” Semantic Encoding And Decoding using DSM-5 for disorder prediction

ACM CIKM 2018

71

4a

Domain Specific

Low Resource

Multi Hop

Symbolic Knowledge

13 Million Reddit Posts and Comments

​

Subreddit names as Labels (Weak)

Not Domain-specific Labels

DSM-5 as Knowledge

72

Really struggling with my bisexuality which is causing chaos in my relationship with a girl. Being a fan of LGBTQ community, I am equal to worthless for her. I’m now starting to get drunk because I can’t cope with the obsessive, intrusive thoughts, and need to get it out of my head.

​

Is mental health related ? Yes: 0.71 , No: 0.29

Which Mental Health condition?

Predicted: Depression (False)

True: Obsessive Compulsive Disorder

Reasoning over Model:

​

Why model predicted Depression?

Unknown

73

Really struggling with my bisexuality which is causing chaos in my relationship with a girl. Being a fan of LGBTQ community, I am equal to worthless for her. I’m now starting to get drunk because I can’t cope with the obsessive intrusive thoughts, and need to get it out of my head.

​

Is mental health related ? Yes: 0.82 , No: 0.18

Which Mental Health condition?

Predicted: Obsessive Compulsive Disorder(True)

True: Obsessive Compulsive Disorder

DSM-5 Knowledge Graph

DSM-5 and Post Correlation Matrix

Reasoning over Model:

​

Why model predicted Obsessive Compulsive Disorder ? known

Interpretable learning

D εR^N

P εR^N

W

f(W)

74

12808 Words

300 dimension embedding

300 dimension embedding

R

Reddit Word Embedding Model

DSM-5 -DAO Lexicon

W

Solvable Sylvester Equation

75

Really struggling with my bisexuality which is causing chaos in my relationship with a girl. I am equal to worthless for her. I’m now starting to get drunk because I can’t cope with the obsessive, intrusive thoughts, and need to get out of my head.

288291000119102: High risk bisexual behavior

365949003: Health-related behavior finding

365949003: Health-related behavior finding

307077003: Feeling hopeless

365107007: level of mood

225445003: Intrusive thoughts

55956009: Disturbance in content of thought

26628009: Disturbance in thinking

1376001: Obsessive compulsive personality disorder

Multi-hop traversal on Medical knowledge graphs

<is symptom>

Achieving Explainability through Medical Entity Normalization :

Replacing Entities in the post with Concepts in the Medical Knowledge Graph through Semantic Annotation

76

Really struggling with my [health-related behavior] which is causing [health-related behavior] with a girl. Being a fan of [XXX] community, I am equal to [level of mood] for her. I’m now starting to [drinking] because I can’t cope with the [obsessive compulsive personality disorder] [disturbance in thinking], and [disturbance in thinking].

​

Is mental health related ? Yes: 0.96 , No: 0.04

Which Mental Health condition?

Predicted: Obsessive Compulsive Disorder(True)

True: Obsessive Compulsive Disorder

DSM-5 Knowledge Graph

DSM-5 and Post Correlation Matrix

Reasoning over Model:

​

Why model predicted Obsessive Compulsive Disorder ? known

Interpretable and Explainable Learning

D εR^N

P εR^N

W

f(W)

77

Domain-specific Knowledge lowers False Alarm Rates.

Abstractive Summarization using Integer Linear Programming and Knowledge-based Constraints

JMIR Mental Health 2021

78

4b

Domain Specific

Low Resource

Symbolic Knowledge

185 Patients Interviews

57 Sentences per Interview

PHQ-9 Lexicon

​

No labeled Reference Summaries

79

Question:

How come DL Summarizer records that Patient has been diagnosed with PTSD, when scripts says otherwise?

Q1: Do you feel restless in sleep?

A1: Two-three times a week

Q2: Do you wake early? How often does it happen?

A2: Too early, happens two times a week.

Q3: Do you persistently feel sad?

A3: Yes. A sense of loneliness

Q4: How often you stay alone?

A4: I am divorced.

Q5: Have you been diagnosed with PTSD?

A5: No

Q6: Have you seen an MHP for your anxiety disorder?

A6: hmm recently.

​

Participant was asked do they feel restless in sleep, then participant said hmm,two three times week

​

Participant was asked, do they persistently feel sad, participant said divorce loneliness

​

Participant was asked, have they been diagnosed with PTSD, participant said hmm recently.

​

Diagnostic Interview Summaries

This happens because

PTSD is a special case of Anxiety with overlaps in symptoms.

TASK

80

81

KS: Knowledge Source (PHQ-9 Lexicon)

V : Vocabulary of tokens

C_i : concept in KS

E: V X V

W: ConceptNet NumberBatch Embedding

(Speer 2017)

82

Results

FRE: Flesch Reading Scale

JSD: Jensen Shannon Divergence

83

Outcomes

Unsupervised Detection of Subevent in Large Scale Disasters

AAAI 2020

84

4c

Domain Specific

Low Resource

Symbolic Knowledge

Multi Hop

10 Million Unlabeled Tweets

~4000 Labeled Tweets

Empathi Ontology

85

Problem Statement

• Sub-event cluster:
• Grouping sub-events for high-level understanding of crises
• {public health, medical experts, drug shortage, epilepsy needs} → Human Health (sub-event cluster)

Challenge :

86

Methodology

87

Empathi Ontology

https://shekarpour.github.io/empathi.io/

88

Empath+Noun/Verb (NV) +Phrases

DEPSUB

Empath+NV+Phrases + Re-Ranked

Hurricane Harvey

89

Empath+NV+Phrases

DEPSUB

Empath+NV+Phrases + Re-ranked

Nepal Earthquake

90

Hurricane Harvey

Nepal Earthquake

Output Examples from Hurricane Harvey and Nepal Earthquake

DEPSUB: Dependency Parsing-based Sub-event Extraction

Ranking Function: Szymkiewicz-Simpson

Our Approach: Phrase Extraction

Ranking Function: Least Common Subsumer + Cosine Similarity

Empathi+NV+Phrase Filtered

Empathi+NV+Phrase Filtered

91

Mechanical Turk Evaluation Labels from Empathi

92

Mechanical Turk Evaluation: Methods and Results

​

Table showing results from human annotators

​

Resources

93

5

Dataset 1: Predicting the Severity Class of Suicide on Reddit

94

Groupwise Annotator Agreement: 0.76

​

Pairwise Annotator Agreement : 0.88

95

TwADR and AskaPatient

Lexicon

https://zenodo.org/record/55013#.XsYEH8YpBQI

Ref: Limsopatham, Nut, and Nigel Collier. "Normalising medical concepts in social media texts by learning semantic representation." Association for Computational Linguistics, 2016.

​

Suicide-Risk Severity Lexicon

https://bit.ly/SRS_lexicon

Ref: Gaur, Manas, Amanuel Alambo, Joy Prakash Sain, Ugur Kurşuncu, Krishnaprasad Thirunarayan, Ramakanth Kavuluru, Amit Sheth, Randy Welton, and Jyotishman Pathak. "Knowledge-aware assessment of severity of suicide risk for early intervention." In The World Wide Web Conference, 2019.

​

​

DSM-5 and Drug Abuse Ontology

Lexicon

https://bit.ly/DSM5_DAO

Ref: Gaur, Manas, Ugur Kurşuncu, Amanuel Alambo, Amit Sheth, Raminta Daniulaityte, Krishnaprasad Thirunarayan, and Jyotishman Pathak. "" Let Me Tell You About Your Mental Health!" Contextualized Classification of Reddit Posts to DSM-5 for Web-based Intervention." In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, 2018.

​

​

​

Suicide Risk Severity Dataset (Reddit)

https://zenodo.org/record/2667859#.XsYH7MYpBQI

Ref: Gaur, Manas, Amanuel Alambo, Joy Prakash Sain, Ugur Kurşuncu, Krishnaprasad Thirunarayan, Ramakanth Kavuluru, Amit Sheth, Randy Welton, and Jyotishman Pathak. "Knowledge-aware assessment of severity of suicide risk for early intervention." In The World Wide Web Conference, 2019.

​

​

96

96

Task: On 50,000 tweets, develop a multi-label predictor for mental health conditions : Depression, Anxiety, and Addiction.

The labeled dataset is created partially through annotation and self-declaration from twitter users.

​

Link to the dataset: http://bit.ly/covid_text_emoji

​

8 Million COVID Tweet Dataset (from which random selection was made) = http://bit.ly/8M_covid

​

Resources:

• ConceptNet : http://conceptnet.io/c/en/download
• EmojiNet : https://www.kaggle.com/rtatman/emojinet
• DBpedia : https://www.dbpedia-spotlight.org/api​
• Wikidata : https://dumps.wikimedia.org/wikidatawiki/entities/​
• Lexicons:
• http://bit.ly/anxiety_lex
• https://bit.ly/dsm5_dao_lex
• http://bit.ly/phq_lex

Dataset 2b: Rank Tweets with Emoji and Mental Health Condition

97

98

AskaPatient

Lexicon

I2b2 Suicide Notes

• Relevant Suicide Notes : 817
• Emotion Labels : 6 {abuse, anger, fear, hopelessness, guilt, and sorrow.}
• Example: My heart is hurt hard and everyday I am grieving.

• SNOMED-CT Concepts : 3051
• Twitter Phrases Per Concept : 56
• Example: Anxiety → {anzity, anxious, anxiety issues, anxy, etc.}

Suicide Risk Severity Lexicon

• Suicide Indicator : 1535 Concepts
• Suicide Ideation : 472 Concepts
• Suicide Behavior : 146 Concepts
• Suicide Attempt : 124 Concepts

Resources

99

Fast Querying of SNOMED-CT, MedDRA, ICD-10

API: https://drive.google.com/file/d/1PPashsy7B32tCDqimKPTsq0UoGo0JnLU/view?usp=sharing

Instruction Manual: https://drive.google.com/file/d/1LAVos4hjO5p_Y8JRfwO2c2x3BtJcglKj/view?usp=sharing

Other Ways to use Knowledge Sources in AI System

100

• Data Enrichment by augmenting concept definitions
• Subtasks: Entity Linking (EL), Entity Extraction (EE), Entity Disambiguation (ED)
• Resources:
• EL → {FreeBase, NELL}
• EE → {Dependency Parsing, Constituency Parsing, Neural IE}
• ED → After EL/EE, use BabelNet for Senses.

• Data Transformation using Embedding Models
• Data Enrichment is local whereas Embedding-based transformation is global.
• Good Way to identify concept and neighboring concepts
• Also, generate relational embedding using functions in Knowledge graph embedding (KGE) models:
• Example KGE models:
• Multiplicative Models
• HoIE, RESCAL
• Additive Models
• TRANSE, PTRANSE
• TRANSH

Other Ways to use Knowledge Sources in AI Systems

101

• Vocabulary Improvement and Fine-tuning/Pre-training
• Adding semantic lexicons to vocab.txt in BERT and re-training
• K-BERT (Liu et al. 2020)
• Query and Inject
• KEPLER (Wang et al. 2019)
• KI-BERT (Faldu et al. 2021)
• ​

• Learning Beyond Dataset (Annervaz et al. 2018)
• Making model work efficiently with limited training data on natural language inference
• Language Models as Knowledge Bases
• Language Model Analysis (LAMA) (Petroni et al. 2019, 2020)

Future Research Directions

102

6

KiL-Explainability as the ability of the framework to compute the difference between learned concepts and actual concepts (conceptual information loss)

​

103

KiL-interpretability as the ability of the framework to proportionately propagate the conceptual information loss among the hidden layers by modulating the hidden representation with/without backpropagation.

​

104

Knowledge Graph-based Interpretable Learning

+

Knowledge Graph-based Explainable Learning

=

Knowledge-infused Learning

105

Challenges Addressed

106

• Defined validated methods of knowledge infusion
• Shallow Infusion
• Semi-Deep Infusion
• Contextualized representation generation through semantic encoding and decoding
• Modeling knowledge constraints in complex clinical interviews
• Modeling Trust based on patient’s visit and ICD-10 knowledge
• We outperform some state of the art systems in
• Suicide Risk Prediction (Reduction in False Alarm)
• Park et. al. -- 91.6%
• Saravia et al. -- 90.74%
• Gkotsis et al. -- 80.76%
• Patient Clinician Match-making (Precision)
• Collaborative filtering approach -- 59.56%
• Collaborative filtering (CF) + Trust -- 21.7%
• Hybrid (CF + Content) -- 13.04%

​

Challenges Addressed

107

• Also, we outperform some SOTA systems in
• Summarization of Clinical Interviews :
• Baselines:
• Extractive Summarization:
• Vanderwende et al. 2007
• Abstractive Summarization:
• Banerjee et al. 2015, and Lewis et al. 2019
• Average Improvement measured using:
• Thematic Overlap - 23.3%
• Flesch Reading Scale - 4.4%
• Contextual Similarity - 2.5%
• Jensen Shannon Divergence - 2.2%
• Finding Emerging Events in Crisis Scenarios
• Baseline:
• Relevance Model (RM3) -- 19.2%
• Dependency Parsing and Statistical Ranking -- 51.7%

​

References

• Valiant, L. G. . Knowledge infusion. In AAAI 2006
• Michael, L., & Valiant, L. G. A First Experimental Demonstration of Massive Knowledge Infusion. In KR, 2008
• Michael, L. . Partial observability and learnability. Artificial Intelligence, 2010.
• Gaur, M., Faldu, K., & Sheth, A. Semantics of the Black-Box: Can knowledge graphs help make deep learning systems more interpretable and explainable?. IEEE Internet Computing, 2021
• Kursuncu, U., Gaur, M., & Sheth, A. Knowledge infused learning (K-IL): Towards deep incorporation of knowledge in deep learning. In AAAI Fall Symposium, 2019
• Gaur, M. et al. " Let Me Tell You About Your Mental Health!" Contextualized Classification of Reddit Posts to DSM-5 for Web-based Intervention. In ACM CIKM 2018
• Gaur, M. et al. Unsupervised detection of sub-events in large scale disasters. In AAAI 2020.

​

108

References

• Gaur, M. et al. empathi: An ontology for emergency managing and planning about hazard crisis. In IEEE Conference on Semantic Computing 2019
• Gaur M. et al. Knowledge-Infused Abstractive Summarization of Clinical Diagnostic Interviews: Framework Development Study. JMIR mental health, 2021
• Gaur, M. et al. Characterization of Time-variant and Time-invariant Assessment of Suicidality on Reddit using C-SSRS. In Plos one 2021
• Roy, K. et al. Knowledge Infused Policy Gradients for Adaptive Pandemic Control. In AAAI Spring Symposium, 2021
• Alambo, A. et al. Question answering for suicide risk assessment using reddit. In IEEE Conference on Semantic Computing, 2019
• Sheth, A., Gaur, M., Kursuncu, U., & Wickramarachchi, R. Shades of knowledge-infused learning for enhancing deep learning. IEEE Internet Computing, 2019.
• Kursuncu, U., Gaur, M., Lokala, U., Thirunarayan, K., Sheth, A., & Arpinar, I. B., Predictive analysis on Twitter: Techniques and applications. In Springer Nature, 2019

109

References

• Anantharam, Pramod, Payam Barnaghi, Krishnaprasad Thirunarayan, and Amit Sheth. "Extracting city traffic events from social streams." ACM Transactions on Intelligent Systems and Technology (TIST) 6, no. 4 (2015): 1-27.
• Mitra, Rajarshee, Manish Gupta, and Sandipan Dandapat. "Transformer Models for Recommending Related Questions in Web Search." In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 2153-2156. 2020.
• Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer." Journal of Machine Learning Research 21 (2020): 1-67.
• Lewis, Mike, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension." In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880. 2020.

110

References

• Bosselut, Antoine, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. "COMET: Commonsense Transformers for Automatic Knowledge Graph Construction." In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4762-4779. 2019.
• Speer, Robyn, Joshua Chin, and Catherine Havasi. "Conceptnet 5.5: An open multilingual graph of general knowledge." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1. 2017.
• Guu, Kelvin, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. "Realm: Retrieval-augmented language model pre-training." arXiv preprint arXiv:2002.08909 (2020).
• Faldu, Keyur, Amit Sheth, Prashant Kikani, and Hemang Akabari. "KI-BERT: Infusing Knowledge Context for Better Language and Domain Understanding." arXiv preprint arXiv:2104.08145 (2021).
• Rudra, Koustav, Pawan Goyal, Niloy Ganguly, Prasenjit Mitra, and Muhammad Imran. "Identifying sub-events and summarizing disaster-related information from microblogs." In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 265-274. 2018.

111

References

• Vanderwende, Lucy, Hisami Suzuki, Chris Brockett, and Ani Nenkova. "Beyond SumBasic: Task-focused summarization with sentence simplification and lexical expansion." Information Processing & Management 43, no. 6 (2007): 1606-1618.
• Gkotsis, George, Anika Oellrich, Sumithra Velupillai, Maria Liakata, Tim JP Hubbard, Richard JB Dobson, and Rina Dutta. "Characterisation of mental health conditions in social media using Informed Deep Learning." Scientific reports 7, no. 1 (2017): 1-11.
• Gkotsis, George, Anika Oellrich, Tim Hubbard, Richard Dobson, Maria Liakata, Sumithra Velupillai, and Rina Dutta. "The language of mental health problems in social media." In Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology, pp. 63-73. 2016.
• Lavrenko, Victor, and W. Bruce Croft. "Relevance-based language models." In ACM SIGIR Forum, vol. 51, no. 2, pp. 260-267. New York, NY, USA: ACM, 2017.
• Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. "Roberta: A robustly optimized bert pretraining approach." arXiv preprint arXiv:1907.11692 (2019).
• Clark, Kevin, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. "Electra: Pre-training text encoders as discriminators rather than generators." arXiv preprint arXiv:2003.10555 (2020).

112

THANKS!!

​

Open to Questions?

​

​

113

Things that kept me going!!

114