Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Part-of-Speech (POS) tagging
Week 3: Lecture 4
2 / 18
Part-of-Speech (POS) tagging
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Task
Given a text of English, identify the parts of speech of each word
Week 3: Lecture 4
2 / 18
Parts of Speech: How many?
Open class words (content words)
nouns, verbs, adjectives, adverbs
mostly content-bearing: they refer to objects, actions, and features in the world
open class, since new words are added all the time
E.g., new words like ‘googling’, ‘photoshop’, etc. get added to English
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Week 3: Lecture 4
3 / 18
Parts of Speech: How many?
Open class words (content words)
nouns, verbs, adjectives, adverbs
mostly content-bearing: they refer to objects, actions, and features in the world
open class, since new words are added all the time
Closed class words
pronouns, determiners, prepositions, connectives, ...
there is a limited number of these
mostly functional: to tie the concepts of a sentence together
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Week 3: Lecture 4
3 / 18
Introduction to POS Tagging
4 / 18
Pawan Goyal (IIT Kharagpur)
POS examples
Week 3: Lecture 4
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
POS tagging: Choosing a tagset
To do POS tagging, a standard set needs to be chosen
Week 3: Lecture 4
5 / 18
POS tagging: Choosing a tagset
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
To do POS tagging, a standard set needs to be chosen Could pick very coarse tagsets
N, V, Adj, Adv
Week 3: Lecture 4
5 / 18
POS tagging: Choosing a tagset
To do POS tagging, a standard set needs to be chosen Could pick very coarse tagsets
N, V, Adj, Adv
More commonly used set is finer grained, “UPenn TreeBank tagset”, 45 tags
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Week 3: Lecture 4
5 / 18
Introduction to POS Tagging
6 / 18
Pawan Goyal (IIT Kharagpur)
UPenn TreeBank POS tag set
Variations of adjectives
Week 3: Lecture 4
Variations of nouns
Variations of verbs
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Using the UPenn tagset
Example Sentence
The grand jury commented on a number of other topics.
Week 3: Lecture 4
7 / 18
Using the UPenn tagset
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Example Sentence
The grand jury commented on a number of other topics.
POS tagged sentence
The/DT grand/JJ jury/NN commmented/VBD on/IN a/DT number/NN of/IN other/JJ topics/NNS ./.
Week 3: Lecture 4
7 / 18
Why is POS tagging hard?
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Main reason: the same word can have different POS tags depending on the context in which it is used
Week 3: Lecture 4
8 / 18
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Why is POS tagging hard?
Words often have more than one POS: back
The back door:
Week 3: Lecture 4
8 / 18
Why is POS tagging hard?
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Words often have more than one POS: back
The back door: back/JJ
On my back:
Week 3: Lecture 4
8 / 18
Why is POS tagging hard?
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Words often have more than one POS: back
The back door: back/JJ On my back: back/NN Win the voters back:
Week 3: Lecture 4
8 / 18
Why is POS tagging hard?
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Words often have more than one POS: back
The back door: back/JJ
On my back: back/NN
Win the voters back: back/RB
Promised to back the bill:
Adverb
Week 3: Lecture 4
8 / 18
Why is POS tagging hard?
Words often have more than one POS: back
The back door: back/JJ
On my back: back/NN
Win the voters back: back/RB
Promised to back the bill: back/VB
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Week 3: Lecture 4
8 / 18
Why is POS tagging hard?
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Words often have more than one POS: back
The back door: back/JJ
On my back: back/NN
Win the voters back: back/RB
Promised to back the bill: back/VB
POS tagging problem
To determine the POS tag for a particular instance of a word
How common is the problem of a word having ambiguous POS tags?
Week 3: Lecture 4
8 / 18
Ambiguous word types in the Brown Corpus
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Ambiguity in the Brown corpus
40% of word tokens are ambiguous 12% of word types are ambiguous
12% distinct words
Week 3: Lecture 4
9 / 18
Ambiguous word types in the Brown Corpus
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Ambiguity in the Brown corpus
40% of word tokens are ambiguous 12% of word types are ambiguous Breakdown of ambiguous word types:
Number of distinct words in the Brown corpus that have k tags
Week 3: Lecture 4
9 / 18
How bad is the ambiguity problem?
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
One tag is usually more likely than the others.
In the Brown corpus, race is a noun 98% of the time, and a verb 2% of the time
A tagger for English that simply chooses the most likely tag for each word can achieve good performance
Any new approach should be compared against the unigram baseline (assigning each token to its most likely tag)
Week 3: Lecture 4
10 / 18
Relevant knowledge for POS tagging
Introduction to POS Tagging
13 / 18
Pawan Goyal (IIT Kharagpur)
Need two types of knowledge to understand POS tag of a word —
The word itself
Some words may only be nouns, e.g. arrow
Some words are ambiguous, e.g. like, flies
Probabilities may help, if one tag is more likely than another
Local context
Two determiners rarely follow each other
Two base form verbs rarely follow each other
Determiner is almost always followed by adjective or noun
Week 3: Lecture 4
POS tagging: Two approaches
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Rule-based Approach
Assign each word in the input a list of potential POS tags
Then winnow down this list to a single tag using hand-written rules
Knowledge-driven approach, e.g., using knowledge of English grammar.
Week 3: Lecture 4
14 / 18
POS tagging: Two approaches
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Rule-based Approach
Assign each word in the input a list of potential POS tags
Then winnow down this list to a single tag using hand-written rules
Statistical tagging
Get a training corpus of tagged text, learn the transformation rules from the most frequent tags (TBL tagger)
Probabilistic: Find the most likely sequence of tags T for a sequence of words W
TBL - Transformation Based Learning; learn transformation rules over POS tags
Week 3: Lecture 4
14 / 18
TBL Tagger
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Assume we have a training set where the words have been tagged with their correct (most likely) tags.
Label the training set with most frequent tags
The can was rusted.
The/DT can/MD was/VBD rusted/VBD.
This is our first guess (most frequent tag for each word).
The guess is correct for ‘The’ and ‘was’, but incorrect for ‘can’ and ‘rusted’. From the training corpus, we know that the correct tag for ‘can’ is NN and the correct tag for ‘rusted’ is VBN.
Add some transformation rules to correct these mistakes.
Week 3: Lecture 4
15 / 18
TBL Tagger
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Label the training set with most frequent tags
The can was rusted.
The/DT can/MD was/VBD rusted/VBD.
Add transformation rules to reduce training mistakes
MD !NN: DT_ MD should be changed to NN, if preceded by DT
VBD!VBN: VBD_
We will frame such rules whenever our predicted tag does not match the gold standard tag (in the training data).
Week 3: Lecture 4
15 / 18
Probabilistic Tagging: Two different families of models
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Problem at hand
We have some data {(d, c)} of paired observations d and hidden classes c.
Week 3: Lecture 4
16 / 18
Probabilistic Tagging: Two different families of models
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Problem at hand
We have some data {(d, c)} of paired observations d and hidden classes c.
Different instances of d and c
Part-of-Speech Tagging: words are observed and tags are hidden.
Text Classification: sentences/documents are observed and the category is hidden.
Categories can be positive/negative for sentiments .. sports/politics/business for documents ...
What gives rise to the two families?
Whether they generate the observed data from hidden stuff or the hidden structure given the data?
Week 3: Lecture 4
16 / 18
Generative vs. Discriminative Models
Introduction to POS Tagging
Pawan Goyal (IIT Kharagpur)
Example problem: Classify documents into a set of classes, e.g., politics, religion, sports, entertainment, …
Discriminative approach: given a document, which is the most probable class?
Generative approach: given a document, from which class is this document most likely to have been generated?
Week 3: Lecture 4
18 / 18