CS 263:

Advanced NLP

Saadia Gabriel

Lecture 1

NLP has a growing and diverse research community

Association for Computational Linguistics

(Founded in the 1960s)

PTE Requests

I will start addressing these after the first class and will give an update on Wednesday.

What is Natural Language Processing?

Why are we all in this room?

Most people use NLP in their daily lives…

Launched 2006

Most people use NLP in their daily lives…

​

credit: ifunny.com

Siri: Launched 2011

Alexa: Launched 2014

Most people use NLP in their daily lives…

Claude,

Copilot: Launched 2023

​

Gemini: Launched 2024

​

NLP

NLP ≠ LLM Science

LLMs

10 or 11 years ago…

Zipf’s Law:

The frequency of a word is inversely proportional to its rank, for example the most frequent word appears twice as often as the second most frequent.

​

You can visualize Zipf’s Law as a power-law distribution dominated by common words.

10 or 11 years ago…

You might have been parsing corpora with regular expressions (Regex)

Find sentences beginning with a pronoun:

NLP bridges

Computer Science

Computational

Linguistics

Modern NLP focuses heavily on AI

Levels of Language

Courtesy of Nanyun (Violet) Peng

Levels of Language

In English, why is “hangry” a word and “digging a foundation” is three words?

Levels of Language

Papa learns computer programing with an AI assistant.

Papa is an Agent, an entity that carries out an action (e.g. learning).

Let’s think more formally about meaning and relatedness

Terminology: lemma and wordform

• A lemma or citation form
• Representation of all forms with the same stem, part of speech, rough semantics

• A wordform
• The inflected word as it appears in text

Lemmas have senses

• One lemma “bank” can have many meanings:

Sense 2:

“…as agriculture burgeons on the east bank the river will shrink even more”

• Sense (or word sense)
• A discrete representation 

                  of an aspect of a word’s meaning.

Hyponymy and Hypernymy

One sense is a hyponym of another if the first sense is more specific, denoting a subclass of the other

• car is a hyponym of vehicle

mango is a hyponym of fruit

Conversely hypernym/superordinate (“hyper is super”)

vehicle is a hypernym  of car

fruit is a hypernym of mango

Meronymy

• The part-whole relation

A leg is part of a chair; a wheel is part of a car.

• Wheel is a meronym of car, and car is a holonym of wheel. 

Homonymy:

multi-sense as an artifact

Homonyms: words that share a form (spelling or pronunciation) but have unrelated, distinct meanings:

• bank₁: financial institution,    bank₂:  sloping land

bat₁: club for hitting a ball,    bat₂:  nocturnal flying mammal

Homonymy:

multi-sense as an artifact

A related multilingual concept is “false friends,” which have identical or similar forms in 2 languages but have different meanings across languages.

Think “pain” in French vs. “pain” in English.

Homonymy causes problems for NLP applications

• Information retrieval

“bat care”

• Machine Translation

bat:  murciélago  (animal) or  bate (for baseball)

• Text-to-Speech

bass (stringed instrument) vs. bass (fish)

Polysemy: related multi-sense

• 1. The bank was constructed in 1875 out of local red brick.

​

• 2. I withdrew the money from the bank 

​

• Are those the same sense?

Sense 2: “A financial institution”

Sense 1: “The building belonging to a financial institution”

• A polysemous word has related meanings. Most non-rare words have multiple meanings.

Synonyms

• Words (different forms) that have the same meaning in some or all contexts.

​

• couch / sofa
• big / large
• automobile / car
• vomit / throw up
• Water / H₂0

Antonyms

• Senses that are opposites with respect to one feature of meaning

• Otherwise, they are very similar!

dark/light   short/long fast/slow rise/fall

hot/cold     up/down       in/out

Levels of Language

You should get a hammer and hit it?

Do you have the ability to crack the window?

Can you open the window right now?

Can you crack the window a little bit?

When this is important in the real world

5 years ago…

What could go wrong with this?

Offensive Content Warning

Breaking Content Filtering Systems

OpenAI content filter

HateBERT

Incorrectly given 73% probability of toxicity

Incorrectly given 8% probability of toxicity

34

Toxicity detectors use loaded words and group-mentions as signals

35

Toxicity detectors use loaded words and group-mentions as signals

36

Risk of Racial Bias in Hate Speech Detection

Systemic risk of silencing already marginalized populations

- Sap, Card and Gabriel et al. (2019)

37

Data collection

(benign and toxic examples)

Prompt GPT-3 to generate novel examples

sample

Prompt

Engineering

Examples with group mentions

• Passover is an annual Jewish holiday\n
• I think he said his parents were Jewish\n
• Nissenbaum is historically a Jewish name\n
• Even as a Jew I think we can all have fun\n
• I’ve really come to embrace my Judaism\n

​

• Jewish weddings can be a lot of fun\n

GPT-3 responds

Pass to GPT-3

Prompt

Toxigen (Hartvigsen and Gabriel et al., 2022)

38

39

Detecting Toxicity is Highly Subjective

40

41

42

43

Modeling Communication

Language exists within certain contexts…

Linguistic, e.g. interpreting the meaning of “Do you want to go there?” based on the previous conversational history “Hey, the park reopened.”

Extra-linguistic (social, spatial or temporal factors),

e.g. is it offensive to say something?

Implicature is what lies beneath explicit statements

Ann

Bill

Hirschberg (1985)

Do you sell paste?

I sell rubber cement

Implication:

Bill doesn’t sell paste,

or he would have said so

Ann

Do you sell paste?

We can describe this as an instance of a “speech act”

A speech act is an utterance that not only conveys information but also performs an action (Austin, 1975).

Rational Speech Act (RSA)

and state of the world (s) based on S₁’s utterance

The speaker chooses their utterance (u) to maximize probability of a literal speaker (Lit) interpreting object w

Pragmatic Speaker (S)

The pragmatic listener interprets S’s utterance and uses Bayes’ rule identify w

Pragmatic Listener (L)

Rational Speech Act (RSA)

Pragmatic Listener (L₁)

and state of the world (s) based on S₁’s utterance

Pragmatic Speaker (S₁)

Rational Speech Act (RSA)

Pragmatic Listener (L₁)

and state of the world (s) based on S₁’s utterance

Pragmatic Speaker (S₁)

Rational Speech Act (RSA)

Pragmatic Listener (L₁)

and state of the world (s) based on S₁’s utterance

Pragmatic Speaker (S₁)

How do we represent words and concepts?

How do we represent a word?

• How do we “understand” a word?

​

• How can we know the relation/distance/similarity between words computationally?

​

Representing words as discrete symbols

• Naïve way: represent words as atomic symbols: student, talk, university (BoW)

​

• Represent word as a “one-hot” vector �[ 0      0     0        1         0   …   0 ]

egg   student   talk    university     happy      buy     

​

​

• How large is (what’s the dimension of) this vector?
• Vector dimension = number of words in vocabulary 
• PTB data: ~50k, 
• Google 1T data: 13M

`

Issues?

• Dimensionality is large; vector is sparse

​

• No similarity

   V_happy  =  [0  0  0  1  0 ... 0 ]

   V_sad      = [0  0  1  0  0 ... 0 ]

   V_milk     = [1  0  0  0  0 ... 0 ]

   _{Vhappy • Vsad  = Vhappy • Vmilk = 0}

​

• Cannot represent new words

`

Word meanings that can help decide:

​

• Word Similarity 
• Distributional (Vector) Models of Meaning

​

• Word Relations

​

• Word Sense Disambiguation

​

• Semantic Roles

Distributional Hypothesis (Harris, 1954):

​

A word is characterized by the company it keeps. In other terms, words that occur in the same contexts tend to have similar meanings.

Theoretical foundation of distributional semantics

Intuitions:  Zellig Harris (1954):

• “oculist and eye-doctor … occur in almost the same environments”

​

• “If A and B have almost identical environments we say that they are synonyms.”

Word-context matrix for word similarity

• Two words are similar in meaning if their context vectors are similar

Note: Very sparse! (~ 50,000 x 50,000)

Word-word matrix

Problem with raw counts

Two classes of vector representation

• Sparse vector representations

• Mutual information weighted co-occurence matrices

Pointwise Mutual Information

Pointwise Mutual Information

Now how do we compute similarity?

Courtesy of pyimagesearch

Measuring similarity: cosine

Cosine for computing similarity

Two classes of vector representation

• Sparse vector representations

• Dense vector representations

• In the past, skip-grams or CBOW

• More likely these days pretrained transformers like BERT

• Mutual information weighted co-occurence matrices

What’s ahead

Next Time…

There are now many types of embeddings:

https://huggingface.co/blog/mteb

Overview of the Class

Representation Learning, Semantics, Pragmatics

Goals of the Course

• Deeply delve into modern NLP problems & solutions: 
• Models, algorithms, and tools that are out there to solve language-related problems you want to tackle
• How to design and evaluate your own task/model/algorithm/tool
• Using data and statistics
• Using your own creativity
• Using the latest advances

​

• At the end you should:
• Agree that language is subtle & interesting 
• Feel some ownership over the models/algorithms/tools
• Understand research papers in the field

Spoiler:

Not AGI

Main Course Website

https://saadiagabriel.com/cs263

Weeks 1 and 2:

In-class Paper

Presentation Guidelines

Presenting in groups of up to 3

(sign up using the spreadsheet linked on the website and Bruin Learn)

Each presentation is ~7 minutes with 3 minutes for Q&A

Select a peer-reviewed paper

(you can take a look at ACL Anthology or other conference proceedings)

Weeks 3 and 4:

Weeks 5-8:

Weeks 9 and 10

4 days of in-person final project presentations

Students will individually provide feedback for other project groups

If you are registered for the course and you are not added by tomorrow night, email me.

Piazza

For general course and homework discussion.

Bruin Learn

The website will be up by next Monday.

This is primarily for submitting homework assignments and viewing grades.

Grading

Cheating

• You MAY
• talk with other students, friends, or others about your homework assignments IF you acknowledge such discussion in your submission
• ask questions about the homework and subject material in the forums

​

• You MAY NOT
• copy code or answers from any source including friends, homework/test services, NLP or other software libraries. This includes making slight changes to previously written code
• Find solutions to these problems online
• Share solutions to these problems online
• hack the scoring servers
• allow your code to be copied, even if unintentionally
• attempt to communicate with or read from any other person or device while taking exams

Cheating

• Unfortunately, about 5% of students will be caught cheating, based on previous experience.

​

• Suspected cheats (including those who were plagiarized from) will be reported to the University. Punishment includes but is not limited to:
• Zero on assignment, exam, or class
• Loss of career services privileges
• Loss of CPT rights

Prerequisites

• This is an advanced graduate-level course, so it is expected you have prior knowledge of machine learning. While we review basic NLP concepts, this is faster-paced than an undergraduate NLP course.

​

• I expect you to program (in Python) at the level of a CS first-year PhD or better.

​

• I expect you to know or be comfortable learning PyTorch, which is a deep learning framework based on Python.

​

There will be basic probability, statistics, linear algebra, and machine learning (CM146, CS260 or equivalent), which will be reviewed as needed.

Questions?