1 of 20

Text Encoders Lack Knowledge: Leveraging Generative LLMs for

Domain-Specific Semantic Textual Similarity

Joseph Gatto, Omar Sharif, Parker Seegmiller, Philip Bohlman, Sarah M. Preum

Dartmouth College

joseph.m.gatto.gr@dartmouth.edu

2 of 20

Semantic Textual Similarity (STS)

2

a girl is styling her hair

a girl is brushing her hair

Text 1:

Text 2:

STS Model

Similarity Score [0-5]

STS Objective: Predict a continuous value between 0-5 that represents the semantic similarity between two texts

Sample from STS-B Test Set

3 of 20

Semantic Textual Similarity (STS)

3

a girl is styling her hair

a girl is brushing her hair

Text 1:

Text 2:

RoBERTa

πŸ‘ Existing STS models work well on simple texts

Sample from STS-B Test Set

2.36

Prediction

True Score = 2.5

Fine-Tuned on STS

4 of 20

Semantic Textual Similarity (STS)

4

ugh, these doctors are gaslighting me!

My dr is making me feel like im crazy!

Text 1:

Text 2:

RoBERTa

❌ However, they struggle to model the relationship between complex domain-specific texts

Sample from our STS-Health Dataset

1.73

Prediction

True Score = 4.65

Fine-Tuned on STS

5 of 20

Semantic Textual Similarity (STS)

5

ugh, these doctors are gaslighting me!

My dr is making me feel like im crazy!

Text 1:

Text 2:

ChatGPT

We show that LLMs significantly outperform text encoders when predicting STS between complex text pairs which require significant world knowledge to understand!

Sample from our STS-Health Dataset

4.0

Prediction

True Score = 4.65

0 Shot Prompt

6 of 20

Why Use Text Generation for a Regression Task?

​

​

6

Recent works have claimed that LLMs have β€œno use case” on regression tasks such as STS-B*

* Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023. Harnessing the power of llms in practice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304.13712.

7 of 20

Why Use Text Generation for a Regression Task?

​

​

7

Intuition: 3 Reasons Why LLMs are Well-Suited for STS

8 of 20

Why Use Text Generation for a Regression Task?

​

​

8

🍐 LLMs are great at pairwise textual comparisons Making them well-suited for STS

9 of 20

Why Use Text Generation for a Regression Task?

​

​

9

🍐 LLMs are great at pairwise textual comparisons Making them well-suited for STS

🧠 LLMs contain significant world knowledge Remove need for domain-specific STS annotation

10 of 20

Why Use Text Generation for a Regression Task?

​

​

10

🍐 LLMs are great at pairwise textual comparisons Making them well-suited for STS

🧠 LLMs contain significant world knowledge Remove need for domain-specific STS annotation

πŸ”’ LLMs are familiar with percentages STS is a percentage prediction task!

11 of 20

Research Questions

​

​

11

  1. What is the optimal LLM prompting strategy for STS?

​

  1. How do LLMs compare to text encoders on popular STS benchmarks?

Eval Sets: STS12-16, STS-B, SICK-R

​

  1. How do LLMs compare to text encoders on domain-specific STS challenge sets?

Domains: Health πŸš‘, News πŸ“°, Sports πŸ€

12 of 20

Methods: Optimal STS Prompting Strategies

​

​

12

Prompt: Output a number between 0 and 5 describing the semantic similarity between the following two sentences:

​

Sentence 1: <text>

Sentence 2: <text>

Prompt: Output a number between 0 and 1 describing the semantic similarity between the following two sentences:

​

Sentence 1: <text>

Sentence 2: <text>

πŸ’‘Key Idea: Reformulate STS as a percentage prediction task

Prior works only explore LLMs for STS in the 0-5 scale

We map the labels between 0-1 so that LLMs can leverage learned knowledge of percentages

13 of 20

RQ1: What is the optimal prompting strategy for STS?

​

​

​

​

13

Result: πŸ’‘Prompting for percentages improves performance on 6/7 datasets!

14 of 20

RQ2: How do LLMs compare to Text Encoders on STS Benchmarks?

​

​

14

Result: πŸ’‘On 6/7 benchmark datasets, our LLM approach performs on-par or better than SOTA text encoders!

15 of 20

RQ3: How do LLMs compare to text encoders on domain-specific STS challenge sets?

​

​

​

15

​

πŸ“°STS-News Example

Text 1: Governors Kathy Hochul & Phil Murphy join 8 other governors in opposing school textbook censorship

Text 2: The New Jersey and New York governors oppose censorship of school textbooks.

Label: 0.86/1.0

​

Three STS Challenge Sets (n=100) collected after May 2023

​

​

  1. STS-Sports πŸ€

Data Source: r/NBA, r/NBATalk, r/NFL

​

  1. STS-Health πŸš‘

Data Source: r/covidlonghaulers, r/LongCovid

​

  1. STS-News πŸ“°

Data Source: r/politics

16 of 20

RQ3: How do LLMs compare to text encoders on domain-specific STS challenge sets?

​

​

​

16

17 of 20

RQ3: How do LLMs compare to text encoders on domain-specific STS challenge sets?

​

​

​

17

18 of 20

18

Limitation: The ChatGPT API is not free!πŸ’°πŸ’°πŸ’°

​

This method may only be well-suited to small-scale tasks requiring STS

19 of 20

Key Takeaways

​

​

​

19

  1. Reformulating STS in the context of percentage prediction improves LLMs’ performance on STS

​

  1. LLMs can perform on-par with SOTA unsupervised text encoders on popular STS benchmarks

  1. LLMs significantly outperform existing STS models on complex, domain-specific STS samples

20 of 20

Thank You!

20

Full Paper

Data