Natural Language Processing

By

​

S.V.V.D.Jagadeesh

Sr. Assistant Professor

Dept of Artificial Intelligence & Data Science

​

​

LAKIREDDY BALI REDDY COLLEGE OF ENGINEERING

• Previously Discussed Topics
• Session Outcomes
• Selection Restrictions
• Representing Selection Restrictions
• Selection Preferences
• Selection Associations
• Selection Preference via conditional Probability
• Evaluating Selection Preferences

​

​

​

​

​

​

​

​

​

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Previously Discussed Topics

LBRCE

NLP

At the end of this session, Student will be able to:

• Understand WSD ,Vector representations and Bayes Method(Understand-L2)

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Session Outcomes

LBRCE

NLP

• In natural language, many words are ambiguous, meaning they have multiple possible meanings (senses).
• Humans easily resolve this ambiguity using context, but computers need explicit methods.
• Word Sense Disambiguation (WSD) is the computational task of:
• Selecting the correct sense of a word given its surrounding context.

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Word Sense Disambiguation

LBRCE

NLP

• WSD plays a crucial role in:
• Machine Translation → correct translation depends on sense
• Information Retrieval → improves search relevance
• Question Answering → ensures correct interpretation
• Text Understanding → avoids semantic confusion

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Why WSD is Important?

LBRCE

NLP

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Mathematical Representation of WSD

LBRCE

NLP

• If we have data which has been hand-labeled with correct word senses, we can use a supervised learning approach to the problem of sense disambiguation.
• Extracting features from the text that are helpful in predicting particular senses, and then training a classifier to assign the correct sense given these features.
• The output of training is thus a classifier system capable of assigning sense labels to unlabeled words in context.
• For lexical sample tasks, there are various labeled corpora for individual words, consisting of context sentences labeled with the correct sense for the target word.

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Supervised WSD

LBRCE

NLP

• To extract useful features from a text window, a minimal amount of processing is first performed on the sentence containing the window.
• This processing varies from approach to approach but typically includes part-of-speech tagging, lemmatization or stemming, and in some cases syntactic parsing to reveal information such as head words and dependency relations.
• Context features relevant to the target word can then be extracted from this enriched input.
• A feature vector consisting of numeric or nominal values is used to encode this linguistic information as an input to most machine learning algorithms.

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Extracting Feature Vectors for Supervised WSD

LBRCE

NLP

• Two classes of features are generally extracted from these neighboring contexts: collocational features and bag-of-words features.
• A collocation is a word or phrase in a position-specific relationship to a target word (i.e., exactly one word to the right, or exactly 4 words to the left, and so on).
• Thus collocational features encode information about specific positions located to the left or right of the target word.
• Typical features extracted for these context words include the word itself, the root form of the word, and the word’s part-of-speech.
• Such features are effective at encoding local lexical and grammatical information that can often accurately isolate a given sense.

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Extracting Feature Vectors for Supervised WSD

LBRCE

NLP

• Example:
• An electric guitar and bass player stand off to one side, not really part of the scene, just as a sort of nod to gringo expectations perhaps.
• A collocational feature-vector, extracted from a window of two words to the right and left of the target word, made up of the words themselves and their respective parts-of speech, i.e.,
• [wi−2,POSi−2,wi−1,POSi−1,wi+1, POSi+1,wi+2,POSi+2]
• would yield the following vector:
• [guitar, NN, and, CC, player, NN, stand, VB]

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Extracting Feature Vectors for Supervised WSD

LBRCE

NLP

• For example, selectional restriction violations (like inedible arguments of eat) often occur in well-formed sentences, for example because they are negated or because selectional restrictions are overstated.
• Examples:
• But it fell apart in 1931, perhaps because people realized you can’t eat gold for lunch if you’re hungry.
• In his two championship trials, Mr. Kulkarni ate glass on an empty stomach, accompanied only by water and tea.

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Extracting Feature Vectors for Supervised WSD

LBRCE

NLP

• The second type of feature consists of bag-of-words information about neighboring words.
• A bag-of-words means an unordered set of words, ignoring their exact position.
• The simplest bag-of-words approach represents the context of a target word by a vector of features, each binary feature indicating whether a vocabulary word w does or doesn’t occur in the context.
• This vocabulary is typically preselected as some useful subset of words in a training corpus.

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Extracting Feature Vectors for Supervised WSD

LBRCE

NLP

• In most WSD applications, the context region surrounding the target word is generally a small symmetric fixed size window with the target word at the center.
• Bag-of-word features are effective at capturing the general topic of the discourse in which the target word has occurred.
• This, in turn, tends to identify senses of a word that are specific to certain domains.

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Extracting Feature Vectors for Supervised WSD

LBRCE

NLP

• Example:
• For example a bag-of-words vector consisting of the 12 most frequent content words from a collection of bass sentences drawn from the WSJ corpus would have the following ordered word feature set:
• [fishing, big, sound, player, fly, rod, pound, double, runs, playing, guitar, band]
• Using these word features with a window size of 10, would be represented by the following binary vector:
• [0,0,0,1,0,0,0,0,0,0,1,0]

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Extracting Feature Vectors for Supervised WSD

LBRCE

NLP

• The naive Bayes classifier approach toWSD is based on the premise that choosing the best sense ˆ s out of the set of possible senses S for a feature vector ~f amounts to choosing the most probable sense given that vector.
• In other words:

​

• As is almost always the case, it would be difficult to collect reasonable statistics for this equation directly.
• To get around this problem we first reformulate our problem in the usual Bayesian manner as follows:

​

​

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Naïve Bayes WSD

LBRCE

NLP

• We naively assume that the features are independent of one another.
• Making this assumption that the features are conditionally independent given the word sense yields the following approximation

​

• Thus we can reformulate the naïve bayes WSD as

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Naïve Bayes WSD

LBRCE

NLP

• We get the maximum likelihood estimate of this probability from the senset aged training corpus by counting the number of times the sense si occurs and dividing by the total count of the target word wj (i.e. the sum of the instances of each sense of the word).
• That is:
• We also need to know each of the individual feature probabilities P( f j |s).
• The maximum likelihood estimate for these would be:

​

​

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Naïve Bayes WSD

LBRCE

NLP

• Decision list classifiers are equivalent to simple case statements in most programming languages.
• In a decision list classifier, a sequence of tests is applied to each target word feature vector.
• Each test is indicative of a particular sense.
• If a test succeeds, then the sense associated with that test is returned.
• If the test fails, then the next test in the sequence is applied.
• This continues until the end of the list, where a default test simply returns the majority sense.

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Decision List Classifier

LBRCE

NLP

• Learning a decision list classifier consists of generating and ordering individual tests based on the characteristics of the training data.
• There are a wide number of methods that can be used to create such lists.
• We can measure how much a feature indicates a particular sense by computing the log-likelihood of the sense given the feature.
• The ratio between the log-likelihoods of the two senses tells us how discriminative a feature is between senses:

​

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Decision List Classifier

LBRCE

NLP

• The decision list is then created from these tests by simply ordering the tests in the list according to the log-likelihood ratio.
• Each test is checked in order and returns the appropriate sense.
• This training method differs quite a bit from standard decision list learning algorithms.

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Decision List Classifier

LBRCE

NLP

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Decision List Classifier

LBRCE

NLP

• Previously Discussed Topics
• Session Outcomes
• Word Sense Disambiguation
• Why WSD is Important?
• Mathematical Representation of WSD
• Supervised WSD
• Extracting feature vectors for Supervised WSD
• Naïve Bayes WSD
• Decision List Classifier

​

​

​

​

​

​

​

​

​

S.V.V.D.Jagadeesh

Wednesday, March 25, 2026

Summary

LBRCE

NLP