Understanding Digital Social Trace Data Via Information Extraction

Shubhanshu Mishra^1*, NLP Researcher

¹ Twitter, Inc.

*Work presented here was done during my PhD at UIUC with multiple collaborators

Content and views expressed in this tutorial are solely the responsibility of the presenter.

PyData MTL - Feb 25, 2021

https://shubhanshu.com/phd_thesis/

https://github.com/socialmediaie

Agenda

• Information extraction for Social Media Text: Multi-task, Human-in-the-loop, Multi-lingual
• Digital Social Trace Data (DSTD)

2

10/2/2020

3

10/2/2020

Information extraction tasks https://shubhanshu.com/phd_thesis

Information extraction from Social Media Text

4

10/2/2020

Why social media data is challenging?

• Most publicly available models are trained on formally written corpora like one from Reuters, New York Times,

e.g. CoNLL 03 NER dataset is derived from Reuters.

• Most annotated NLP datasets are derived from the above formal corpora.
• Fewer number of corpora for social media e.g. Facebook, Twitter, Reddit, etc.

5

Why social media data is challenging?

Social Media text often has a inherent structure, which provides context, e.g.

​

• user mentions
• hashtags
• comment threads
• less formally written language
• lot of unseen words
• typos, etc.

6

NER performance difference

7

Source: Derczynski, L., Maynard, D., Rizzo, G., van Erp, M., Gorrell, G., Troncy, R., Petrak, J., & Bontcheva, K. (2015). Analysis of named entity recognition and linking for tweets. Information Processing & Management, 51(2), 32–49. https://doi.org/10.1016/j.ipm.2014.10.006

We need social media specific models to perform well

• ​

8

Text classification https://github.com/socialmediaie/SocialMediaIE

9

10/2/2020

Sequence tagging https://github.com/socialmediaie/SocialMediaIE

10

10/2/2020

Applications of information extraction

Index documents by entities

11

10/2/2020

Where is the data?

• MetaCorpus: A list of curated annotated datasets for various social media tasks and social media platforms. https://github.com/socialmediaie/MetaCorpus
• MetaCorpus - benchmark: A selected set of datasets which can be used for benchmarking multi-task learning or NLP for social media data

12

Tagging data

13

10/2/2020

Super sense tagging

Part of speech tagging

Named entity recognition

Chunking

Classification data

14

10/2/2020

Sentiment classification

Abusive content identification

Uncertainty indicator classification

Methods for Extracting Information from Social Media Data

Rules + Models

Multi-task learning

Multilingual learning

Active learning

15

10/2/2020

Rule based Twitter NER Mishra & Diesner (2016). https://github.com/napsternxg/TwitterNER

16

10/2/2020

Mishra, Shubhanshu, & Diesner, Jana (2016). Semi-supervised Named Entity Recognition in noisy-text. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT) (pp. 203–212). Osaka, Japan: The COLING 2016 Organizing Committee. Retrieved from https://aclweb.org/anthology/papers/W/W16/W16-3927/  

Evaluating Twitter NER (F1-score) Mishra & Diesner (2016).

17

10/2/2020

Source: https://blog.maxar.com/earth-intelligence/2017/named-entity-recognition-for-twitter

Code: https://github.com/humangeo/twitter-ner-eval

Multi-task-multi-dataset learning Mishra 2019, HT’ 19

Slide # 18

10/2/2020

MTL – Multi task Stacked (Layered)

MD – Multi-dataset

MTS – Multi task Shared

S - Single

Shubhanshu Mishra. 2019. Multi-dataset-multi-task Neural Sequence Tagging for Information Extraction from Tweets. In Proceedings of the 30th ACM Conference on Hypertext and Social Media (HT '19). ACM, New York, NY, USA, 283-284. DOI: https://doi.org/10.1145/3342220.3344929

Evaluating MTL models Mishra 2019, HT’ 19

19

10/2/2020

Super sense tagging (micro f1)

Part of speech tagging (overall accuracy)

Named entity recognition (micro f1)

Chunking (micro f1)

Shubhanshu Mishra. 2019. Multi-dataset-multi-task Neural Sequence Tagging for Information Extraction from Tweets. In Proceedings of the 30th ACM Conference on Hypertext and Social Media (HT '19). ACM, New York, NY, USA, 283-284. DOI: https://doi.org/10.1145/3342220.3344929

Training Mishra 2019, HT’ 19

• Sample mini-batches from a task/data
• Compute loss for the mini-batch
• Individual loss is the log loss for conditional random field
• Update the model except the Elmo module
• During an epoch go through all tasks and datasets
• Train for a max number of epochs
• Use early stopping to stop training

• Models trained on single datasets have prefix S
• Models trained on all datasets of same task have prefix MD
• Models trained on all datasets have prefix MTS for multitask models with shared module, and MTL for stacked modules
• Models with LR=1e-3 and no L2 regularization have suffix "*"
• Models trained without NEEL2016 have suffix "#"

Slide # 20

10/2/2020

Label embeddings (POS)

Slide # 21

10/2/2020

• MDMT model learns similarity between labels without this knowledge being encoded in the model
• This leads to consistent relationship between similar labels across datasets

Label embeddings (NER)

Slide # 22

10/2/2020

• MDMT model learns similarity between labels without this knowledge being encoded in the model
• This leads to consistent relationship between similar labels across datasets

Label embeddings (chunking)

Slide # 23

10/2/2020

• MDMT model learns similarity between labels without this knowledge being encoded in the model
• This leads to consistent relationship between similar labels across datasets

Label embeddings (super-sense tagging)

Slide # 24

10/2/2020

• MDMT model learns similarity between labels without this knowledge being encoded in the model
• This leads to consistent relationship between similar labels across datasets

Label embeddings (super-sense tagging)

Slide # 25

10/2/2020

• MDMT model learns similarity between labels without this knowledge being encoded in the model
• This leads to consistent relationship between similar labels across datasets

Web based UI https://github.com/socialmediaie/SocialMediaIE

Slide # 26

10/2/2020

Sentiment classification results https://github.com/socialmediaie/SocialMediaIE

Slide # 27

10/2/2020

Slide # 28

10/2/2020

Uncertainty indicators

Abusive content identification

https://github.com/socialmediaie/SocialMediaIE

Label embeddings

Slide # 29

10/2/2020

https://github.com/socialmediaie/SocialMediaIE

• MDMT model learns similarity between labels without this knowledge being encoded in the model
• This leads to consistent relationship between similar labels across datasets

Web based UI https://github.com/socialmediaie/SocialMediaIE

Slide # 30

10/2/2020

31

10/2/2020

Multilingual transformer models for hate and abusive speech

https://github.com/socialmediaie/TRAC2020 - Fine tuned models at https://huggingface.co/socialmediaie

2nd in 1/6 sub-tasks: ENG A

3rd in 3/6 sub-tasks: HIN A, B, and IBEN B

4th in 1/6 sub-tasks: ENG B

Computationally faster and cheaper inference cost.

Social Media IE: https://github.com/socialmediaie/SocialMediaIE

• A library for training Multi-task Multi-dataset models
• Pre-trained multi-task models for 20 social media classification and tagging tasks
• Currently uses ELMo as base embedding layer, transformers supported version planned soon (feel free to contribute)
• Returns multi-task outputs as JSON or pandas DataFrame

32

33

Mishra, S., Prasad, S. & Mishra, S. Exploring Multi-Task Multi-Lingual Learning of Transformer Models for Hate Speech and Offensive Speech Identification in Social Media. SN COMPUT. SCI. 2, 72 (2021). https://doi.org/10.1007/s42979-021-00455-5

​

​

Code: https://github.com/socialmediaie/MTML_HateSpeech

Incremental learning of text classifiers with human-in-the-loop

• Given a large unlabeled corpus, can we label it efficiently using fewer human annotations?
• Can existing models be updated efficiently to work with new data?
• Proposal:
• Use active learning for data labeling
• Use incremental learning algorithms for model updates
• Highly application to social media data:
• Streaming data
• Model should adapt to new data

Slide # 34

10/2/2020

Mishra, Shubhanshu, Jana Diesner, Jason Byrne, and Elizabeth Surbeck. 2015. “Sentiment Analysis with Incremental Human-in-the-Loop Learning and Lexical Resource Customization.” In Proceedings of the 26th ACM Conference on Hypertext & Social Media - HT ’15, 323–25. New York, New York, USA: ACM Press. https://doi.org/10.1145/2700171.2791022.

Active Learning

1. Given a model and unlabeled data
2. Select samples from the unlabeled data to be annotated, based on selection criterion
3. Update model with collected labeled examples
4. Repeat steps 2 to 3 till desired accuracy is reached or data exhausted

Slide # 35

10/2/2020

Mishra et al. (2015)

Slide # 36

10/2/2020

Mishra et al. (2015)

Slide # 37

10/2/2020

• Each round query 100 samples
• Classifier is logistic regression with unigram and lexicon features
• Max rounds is 100 (except Clarin)

Data ordered alphabetically and X and Y axes are not shared.

https://github.com/socialmediaie/SocialMediaIE

Slide # 38

10/2/2020

• Evaluate only on the data not used for training
• Top strategy queries efficiently and can help in labeling full data more quickly.

Data ordered alphabetically and X and Y axes are not shared.

https://github.com/socialmediaie/SocialMediaIE

List of social media IE tools

• SocialMediaIE - https://github.com/socialmediaie/SocialMediaIE
• TwitterNER - https://github.com/socialmediaie/TwitterNER (more lightweight NER focused on English tweets)
• Social Communication Temporal Graph - https://github.com/napsternxg/social-comm-temporal-graph/ (visualizing temporal networks)
• ConText - https://github.com/uiuc-ischool-scanr/ConText (generate networks from text data)
• SAIL - https://github.com/uiuc-ischool-scanr/SAIL (active learning for text classification, python version coming soon at https://github.com/socialmediaie/)

39

10/2/2020

Other models for multi-task learning

• Hierarchical labels or multi-label settings
• Mishra, S., Prasad, S., & Mishra, S. (2020). Multilingual Joint Fine-tuning of Transformer models for identifying Trolling, Aggression and Cyberbullying at TRAC 2020. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying (pp. 120–125). Marseille, France: European Language Resources Association (ELRA). Retrieved from https://www.aclweb.org/anthology/2020.trac-1.19. Code: https://github.com/socialmediaie/TRAC2020
• Mishra, S., & Mishra, S. (2019). 3Idiots at HASOC 2019: Fine-tuning Transformer Neural Networks for Hate Speech Identification in Indo-European Languages. In FIRE (Working Notes) (pp. 208-213). Retrieved from http://ceur-ws.org/Vol-2517/T3-4.pdf. Code: https://github.com/socialmediaie/HASOC2019 �

40

10/2/2020

Digital Social Trace Data (DSTD)

41

42

10/2/2020

Information extraction https://shubhanshu.com/phd_thesis/

“Information Extraction refers to the automatic extraction of structured information such as entities, relationships between entities, and attributes describing entities from unstructured sources.”

– (Sarawagi, 2008)

Information extraction from semi-structured data

Slide # 43

10/2/2020

However, not all data is unstructured. Many datasets of interest have some inherent structure imposed because of the data generating process.

44

10/2/2020

Digital Social Trace Data https://shubhanshu.com/phd_thesis/

Digital Social Trace Data (DSTD) are digital activity traces generated by individuals as part of a social interactions, such as interactions on social media websites like Twitter, Facebook; or in scientific publications.

​

​

Inspired from Digital Trace Data (Howison et. al, 2011)

​

45

10/2/2020

DSTD properties and examples

Slide # 46

10/2/2020

47

10/2/2020

Information extraction tasks https://shubhanshu.com/phd_thesis

Improving sentiment classification using user and tweet metadata

48

Twitter Sentiment Corpora

• Are our corpora biased to certain meta-data attributes?
• Can those biases propagate into systems trained on these corpora?
• How correlated are these meta-data features with the annotated sentiment?
• Do these correlations hold outside of the annotated data for the same users?
• Can sentiment classifiers exploit this bias to do well on these datasets?

Sentiment is usually identified as positive, negative, and neutral.

Mishra, S., & Diesner, J. (2018, July 3). Detecting the Correlation between Sentiment and User-level as well as Text-Level Meta-data from Benchmark Corpora. Proceedings of the 29th on Hypertext and Social Media. HT ’18: 29th ACM Conference on Hypertext and Social Media. https://doi.org/10.1145/3209542.3209562

49

Types of metadata and what they quantify

50

User metadata v/s Sentiment

51

Using metadata features can improve sentiment classification

Boost in F1 is mostly due to better recall. Precision is lower.

​

MESC might be helping with tweets with high OOV rates, where text classifiers don’t do well.

Visualizing DSTDs using Social�Communications Temporal Graph

• Social communication on social media, community forums, and Wikipedia, is intrinsically networked as well as temporal.
• Common ways of visualizing social communication are line or scatter plot, where the reader can see the temporally changing aspect of either posts or comments.
• Both, line and scatter plots, hide the connected nature of social communication.

Slide # 52

10/2/2020

Approach

Social Communication Temporal Graphs (SCTG) overcome this issue:

• It use two vertically stacked scatter plots.
• The points in the top plot are connected to points in the bottom plot.
• Both plots share the same time axis.
• Each point in either plot can be styled using additional attributes. This adds at-least 3 more dimensions of data to each point by styling their vertical height, point color, and point shape.
• Using SCTG, the reader can observe the connected as well as temporal nature of the social communication.

​

Slide # 53

10/2/2020

Components

• Core communication components: This can be a user in a feed or a specific post
• Child components: This can be associated posts by a user or comments to a post
• Component links: Core communication is liked to its children
• Activity timeline: This quantifies the temporal activity measurement
• Tool tips: They provide additional data about each component
• Component heights, scaling, and color: Visualize additional metadata

Slide # 54

10/2/2020

Visualizing FB groups

Slide # 55

10/2/2020

https://shubhanshu.com/social-comm-temporal-graph/

Slide # 56

10/2/2020

Tweet sentiment over time https://shubhanshu.com/social-comm-temporal-graph/

Visualize temporal network of social media data in your browser

• Social Communication Temporal Graph: https://shubhanshu.com/social-comm-temporal-graph/
• Recent tweet comparison – Compare user-tweet network on tweets about 2 search queries
• Recent Tweet Sentiments – Compare user and tweet level sentiment on tweets about a single search query
• Wikipedia Revisions – Compare Wikipedia edit activity across 2 pages and identify common users�

57

10/2/2020

Inspiration from scholarly data using DSTD framework

58

Paper = Set of concepts

Slide # 59

10/2/2020

Title: Geographic assessment of breast cancer screening by towns, zip codes, and census tracts. [PMID: 18019960]

No. of Authors: 7

Year: 2000

https://go.illinois.edu/legolas

Temporal profile of a concept

Slide # 60

10/2/2020

Mishra, Shubhanshu, and Vetle I. Torvik. 2016. “Quantifying Conceptual Novelty in the Biomedical Literature.” D-Lib Magazine : The Magazine of the Digital Library Forum 22 (9–10). https://doi.org/10.1045/september2016-mishra.

Conceptual novelty

Motivation

• Identify concept level novelty of articles and authors
• Concepts identifies using MeSH and pair of MeSH terms for MEDLINE

Application

• Temporal profile of concepts in MEDLINE
• Temporal profile of author novelty
• Visualizing novelty of paper, author over time: http://abel.ischool.illinois.edu/gimli/

Slide # 61

10/2/2020

Conceptual expertise

• Conceptual coverage of a paper
• Collaborations:
• Which authors contribute?
• Does author position give us a clue?
• Expertise over time and careers

​

Slide # 62

10/2/2020

Mishra, Shubhanshu, Brent D. Fegley, Jana Diesner, and Vetle I. Torvik. 2018. “Expertise as an Aspect of Author Contributions.” In WORKSHOP ON INFORMETRIC AND SCIENTOMETRIC RESEARCH (SIG/MET). Vancouver.

Thank you

• Questions and tweets at @TheShubhanshu
• Some related material presented here can be found at: https://shubhanshu.com/phd_thesis/
• If you have questions or feature requests about any of the tools open an issue on github e.g. for SocialMediaIE at: https://github.com/socialmediaie/SocialMediaIE/issues

​

63

10/2/2020

References

• Mishra, Shubhanshu (2019): Trained models for multi-task multi-dataset learning for text classification as well as sequence tagging in tweets. University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-1094364_V1 
• Mishra, Shubhanshu (2019): Trained models for multi-task multi-dataset learning for text classification in tweets. University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-1917934_V1
• Mishra, Shubhanshu (2019): Trained models for multi-task multi-dataset learning for sequence prediction in tweets. University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-0934773_V1 
• Shubhanshu Mishra. 2019. Multi-dataset-multi-task Neural Sequence Tagging for Information Extraction from Tweets. In Proceedings of the 30th ACM Conference on Hypertext and Social Media (HT '19). ACM, New York, NY, USA, 283-284. DOI: https://doi.org/10.1145/3342220.3344929

​

​

64

10/2/2020

References

• Mishra, Shubhanshu, & Diesner, Jana (2016). Semi-supervised Named Entity Recognition in noisy-text. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT) (pp. 203–212). Osaka, Japan: The COLING 2016 Organizing Committee. Retrieved from https://aclweb.org/anthology/papers/W/W16/W16-3927/  
• Mishra, Shubhanshu, Diesner, Jana, Byrne, Jason, & Surbeck, Elizabeth (2015). Sentiment Analysis with Incremental Human-in-the-Loop Learning and Lexical Resource Customization. In Proceedings of the 26th ACM Conference on Hypertext & Social Media - HT ’15 (pp. 323–325). New York, New York, USA: ACM Press. https://doi.org/10.1145/2700171.2791022 

65

10/2/2020