1 of 18

Hyperboloid embeddings and Hypersphere similarity metrics for Sentence-BERT

Garvit Goyal, Gunjan Gupta, Tusheet Goli

2 of 18

Sentence-BERT

(SBERT)

  • SBERT is a modified BERT network

​

  • Better for deriving semantically meaningful sentence embeddings

​

  • State-of-art sentence embedding model that is computationally efficient to get sentence embeddings

​

  • Uses siamese and triplet network structures

​

  • Uses distance as similarity metrics and contrastive loss

​

3 of 18

Sentence-BERT

(SBERT)

Architecture

4 of 18

Our Goal

  • Analyze and understand SBERT
  • Aim to improve SBERT
    • Changing the pooling layer to go from BERT word embeddings to sentence embeddings
    • Exploring other similarity metrics that might be more suited for the semantic similarity task

5 of 18

Datasets

6 of 18

STSb

(Semantic Textual Similarity Benchmark)

  • A selection of English datasets that have been primarily used as a benchmark dataset for SemEval between 2012 and 2017
  • Includes text from image captions, news headlines, and user forum
  • Consists of 8628 sentence pairs from multiple genres
    • Divided into a train, dev, and test splits

7 of 18

SNLI

(Stanford Natural Language Inference)

  • Contains 570,000 sentence pairs labeled as entailment, neutral, or contradiction
  • Binary labeled dataset of sentence pairs
  • Used for testing our uniformity and alignment as the similarity metrics in our loss function

8 of 18

Model

9 of 18

Proposed Architecture

10 of 18

Hyperspherical loss functions

Source: Wang and Isola (2020)

11 of 18

Hyperboloid Embeddings

12 of 18

Implementation Details

13 of 18

Implementation Details

  • Baseline: bert-base-uncased followed by mean pooling
  • Optimizer: SGD with lr = 0.001 and momentum = 0.9
  • The NLI versions were trained on NLI for one epoch, and the STS versions were trained on the STS dataset for 10 epochs.
  • Both types of models were validated every 10% of iterations on dev split of STS benchmark, and tested on test split of STS benchmark

14 of 18

Implementation Details

15 of 18

Results

Model

Distance Function

Loss Function

STS Benchmark Test Performance (Spearman coefficient)

BERT-STSB-Baseline

Cosine

Contrastive

0.839

BERT-STSB-CNN + Mean Pooling

Cosine

Contrastive

0.840

BERT-STSB-CNN + Max Pooling

Cosine

Contrastive

0.824

BERT-STSB-hyperboloid

Lorentz

Contrastive

0.591

BERT-NLI-hypersphere

Euclidean

Uniformity - Alignment

0.547

BERT-NLI-hypersphere + hyperboloid

Lorentz

Uniformity - Alignment

0.536

16 of 18

Future Work

  • Hyperparameter optimization: more so than usual machine learning models, SBERT’s performance is greatly influenced by hyperparameters of the loss function formulation, such as the relative weights of the uniformity and alignment losses and their parameters. However, over the course of experiments, we observed that hyperparameter search is computationally expensive and could be explored further in future works.

​

  • One specific analysis could be to observe the model’s performance when optimizing combinations of alignment loss, uniformity loss, and contrastive loss functions. Our work focused on comparing a few combinations of the first two functions against the sole optimization of the third.

  • Further, we observed numerical instability when using the logarithm of the expected value of gaussian potential, as proposed by Wang & Isola (2020). Instead, we optimized the expected value of gaussian potential in this work. Future works could try to provide concrete mathematical analyses of this behavior in BERT embeddings since the original work was only tested on CNN and RNN encodings.

17 of 18

References

  • Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.
  • Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere.
  • Qian Chen, Zhen-Hua Ling, and Xiaodan Zhu. 2018. Enhancing sentence embedding with generalized pooling.
  • Maximilian Nickel and Douwe Kiela. 2017a. Poincaré embeddings for learning hierarchical representations.

18 of 18

Thank You!