1 of 54

CSCI-SHU 376: Natural Language Processing

Hua Shen

Course Agenda: 2026 Fall-NLP-[CSCI-SHU-376]-Class Schedule

2026-09-14

Fall 2026

Lecture 5: Sequence Modeling and Neural Networks

​

2 of 54

Plans

  • Part-of-speech Tagging
  • Named Entity Recognition
  • Hidden Markov Model (HMM)

​

​

​

​

3 of 54

Standard NLP Pipelines

4 of 54

Sequence Modeling

5 of 54

What are part of speech Tags?

  • Word classes or syntactic categories
  • Reveal useful information about a word and its neighbors

6 of 54

Part of Speech

  • Different words have different functions
  • Closed Class: fixed membership, function words
    • E.g., prepositions (in, on, of), determiners (the, a)
  • Open Class: new words get added frequently
    • E.g., nouns, verbs, adjectives etc

7 of 54

Part of Speech Tagging

  • Tag each word in a sentence with its part of speech
  • Disambiguation task: each word might have different functions in different contexts

8 of 54

Part of Speech Tagging: A simple Baseline

  • Many words are easy to tag
  • Most frequent class: assign each word to the class it occurred most in the training set (e.g., man/NN)
  • Accuracy tags 92.34% of word tokens on wall street journal (WSJ)!
  • State of the art: 97%
  • Better to make decisions on entire sentences over individual words
  • POS Tagging still not solved yet!

9 of 54

Sequence Modeling

10 of 54

Named Entities

  • Named entity means anything that can be referred to with a proper name
  • NER: 1) finding spans of text that constitute proper names; 2) tagging the type of the entity
  • Most common tags:
    • PER (Person)
    • LOC (Location)
    • ORG (Organization)
    • MISC (Miscellaneous)

11 of 54

Named Entity Recognition

12 of 54

Named Entity Recognition: BIO Tagging

  • B: token that begins a span
  • I: tokens that inside a span
  • O: tokens outside of a span

13 of 54

Plans

  • Part-of-speech Tagging
  • Named Entity Recognition
  • Hidden Markov Model (HMM)

​

​

​

​

14 of 54

Markov Chains

 

15 of 54

Markov Chains

16 of 54

Hidden Markov Model

  • We do not see sequences of POS tags in text
  • However, we do observe the words
  • HMM allows us to jointly reason over both hidden and observed events
    • Assume each position has a tag that generates a word

​

​

​

17 of 54

Hidden Markov Model

 

18 of 54

Assumptions

​

​

​

 

19 of 54

Sequence likelihood

​

​

​

 

Transition probability

Emission probability

20 of 54

Sequence likelihood (Example)

​

​

​

 

21 of 54

Sequence likelihood (Example)

​

​

​

 

 

22 of 54

Learning HMM

​

​

​

 

23 of 54

Learning Examples

 

 

24 of 54

Decoding with HMMs

  • Task: Find the most probable states given observations
  • E.g., POS Tagging

​

​

25 of 54

Greedy Decoding

  • Decode one state at a time

​

​

26 of 54

Greedy Decoding

  • It does not guarantee to produce the overall optimal sequence
  • Decode one state at a time

​

​

27 of 54

Beam Search

 

28 of 54

Beam Search

 

29 of 54

Beam Search

 

30 of 54

Beam Search

 

31 of 54

Beam Search

  • Beam search is a very common decoding method for any language generation tasks (e.g., machine translation)

32 of 54

Today’s Plan

  • Neural Networks
  • Recurrent neural networks (RNNs)

​

​

​

​

​

33 of 54

Neural Network

  • A network of small computing units
  • Deep learning: Modern neural network (have many layers)
  • Possible to learn any function

​

​

​

​

34 of 54

Neural Network: Units

 

35 of 54

Neural Network: Units

  • Tanh function
  • Sigmoid function
  • ReLU function

36 of 54

Neural Network: Units

37 of 54

Feedforward Neural Networks

  • Sometimes called multi-layer perceptrons (MLPs)
  • Input units, hidden units, output units
  • Fully-connected: each unit in each layer takes input from all units in the previous layer

​

​

​

​

38 of 54

Feedforward Neural Networks

  • Feedforward NN with a single hidden layer
  • Input vector x, output probability distribution y

​

​

​

​

39 of 54

XOR Problem

  • Cannot be solved by a linear separator (e.g., perceptron)

​

​

​

​

40 of 54

XOR Problem

  • Input: [0, 0]
  • Linear transformation: [0, -1] -> ReLU [0, 0]
  • Y [0]
  • [0, 1] OR [1, 0] -> 1; [0, 0] OR [1, 1] -> 0

​

​

41 of 54

Feedforward NN for text classification

  • Input: n input words / tokens
  • Output: positive, negative and neutral

​

​

​

​

42 of 54

Training Neural Network

  • Multi-class Classification:

​

​

  • Goal: Learn the parameter w and b from training data
  • Loss function: Cross Entropy
  • Binary Classification:

​

​

43 of 54

Training Neural Network: Gradient

 

​

​

44 of 54

Training Neural Network: Gradient

  • Computation Graph: mathematical expressions with a directed graph
  • Use Graph to compute gradients

​

​

​

​

45 of 54

Training Neural Network: Gradient

46 of 54

Recap: n-gram language models

  • N-gram models:

​

  • Issues: Unable to model bigger context..

​

47 of 54

Feedforward neural language models

  • Approximate the probability based on the previous m (e.g., 5) words

​

48 of 54

Feedforward neural language models

  • Limitations: The models learn separate patterns for different positions

​

49 of 54

Viterbi Decoding

 

50 of 54

Viterbi Decoding

51 of 54

Viterbi Decoding

52 of 54

Viterbi Decoding

 

53 of 54

Beam Search

  • If K (number of hidden states) is too large, Viterbi is too expensive!!

54 of 54

Beam Search

  • If K (number of hidden states) is too large, Viterbi is too expensive!!