Validating Labeling Functions

in Domain Shift

20223137 Yewon Kim

20224560 Seungjoo Lee

Presentation template was adopted from SlidesCarnival.

To obtain more labeled training data,

weak supervision leverages cheaper & noisy labels

2

Get cheaper labels

from non-experts

e.g., crowdsourcing

Get higher-level supervision from experts

e.g., labeling functions

Get pseudo-labels from pre-trained models

e.g., knowledge distillation

Labeling function (LF) is a lightweight and cost effective way to generate labels in unlabeled data.

3

Source: https://snorkel.ai/weak-supervision-modeling/

However, what if data distribution shifts?

4

Developer side

^(?=.*\bkid\b)(?=.*\blike\b).*$

Domain: toy

​

review/”My kid likes it”

⇒ positive

Example scenario:

sentiment analysis from review data

However, what if data distribution shifts?

5

Developer side

^(?=.*\bkid\b)(?=.*\blike\b).*$

Domain: toy

​

review/”My kid likes it”

⇒ positive

Domain: book

​

review/”A true love story”

⇒ ???

LFs are no longer valid;

need to update!

Example scenario:

sentiment analysis from review data

However, what if data distribution shifts?

6

Developer side

^(?=.*\bkid\b)(?=.*\blike\b).*$

Domain: toy

​

review/”My kid likes it”

⇒ positive

Domain: book

​

review/”A true love story”

⇒ ???

Accurately and timely detecting the data shift and

prompting engineers to update LFs is crucial

in order to ensure the reliable performance of an end model!

Example scenario:

sentiment analysis from review data

Our idea: Use the outputs from LFs to detect domain shift!

7

• Previous works: observe an input itself to determine if it is out-of-distribution (OOD)
• Specifically, define a score function and classify it as OOD if �where is a predefined threshold.
• Score functions: e.g., language models
• Instead, we observe outputs of LFs to determine OOD
• Outputs of LFs contain richer information as LFs are specifically designed to �identify certain aspects of the data.
• More efficient and scalable, as it does not necessarily require models to capture �important features from raw data.

Method: (1) Changing discrete LFs to continuous LFs

8

Example of discrete LF on NLP sentiment analysis:�Keyword-based heuristic function

​

@labeling_function

def positive_keyword_lf(text, keyword):

if keyword in text.lower():

return POSITIVE

return ABSTAIN

• Outputs limited values (POSITIVE, NEGATIVE, ABSTAIN)
• Information from discrete LFs are not enough to detect OOD

T-SNE result, 8 discrete LFs

Method: (1) Changing discrete LFs to continuous LFs

9

T-SNE result, 8 discrete LFs

Method: (1) Changing discrete LFs to continuous LFs

10

Example of continuous LF:�Using cosine similarity of GloVe word embedding

​

@labeling_function

def positive_keyword_lf(text, keyword):

text_emb = glove(text.lower().split())

keyword_emb = glove([keyword])

​

return get_cosine_similarity(text_emb, keyword_emb)

• Cosine similarity between passage and keyword
• Outputs continuous values → dense information

T-SNE result, 8 continuous LFs

Method: (2) Kernel density estimation

11

Overall pipeline : Training phase

12

Unlabeled training data

Discrete LFs

Continuous LFs

Kernel density estimation

Overall pipeline : Testing phase

13

Unlabeled test data

Discrete LFs

Labeled data

Continuous LFs

Kernel density estimation

OOD detection

Evaluation setup: task and dataset

14

Sentiment analysis task (binary classification); we used IMDB [1], Yelp [2], and Amazon reviews [3].

• Train : ID (20000) / Test : ID (5000) + OOD(5000)

[1] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. (2011). Learning Word Vectors for Sentiment Analysis. The 49th Annual Meeting of the Association for Computational Linguistics (ACL 2011).

[2] Xiang Zhang, Junbo Zhao, Yann LeCun. Character-level Convolutional Networks for Text Classification. Advances in Neural Information Processing Systems 28 (NIPS 2015).

[3] R. He, J. McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. WWW, 2016

Evaluation setup: LF development

15

• Keyword-based interactive LF generation using Argilla[1]

[1] https://docs.argilla.io/en/latest/

Evaluation setup: LF development

16

• Keyword-based interactive LF generation using Argilla[1]
• 12 positive keywords, 20 negative keywords

[1] https://docs.argilla.io/en/latest/

Results: OOD Detection

17

KDE h = 0.05, batch 16 fixed

Batch-AUROC Tradeoff

18

Discussion & Future work

19

• Providing explainable prompts to engineers
• Train separate OOD detector for each LF
• When the OOD detected, run LF OOD detectors to find out wrong LFs
• Other ways to convert discrete LFs to continuous LFs
• Using coverage as OOD predictor
• Coverage drops significantly with OOD data
• Experiments on different shift scenarios & domains
• Only IMDB is used as source distribution
• Applying to other NLP tasks
• Applying to other domains such as vision