Introduction to Natural Language Processing (NLP)

Fardina Alam

CMSC - 320 Introduction to Data Science

2026

Today We will Cover

1. What is NLP
2. Language Structure: Syntax vs Semantics
3. Brief Summary NLP Applications
4. Preprocessing and Feature Extraction steps of NLP
5. NLP Applications
6. Cosine Similarity- important measure use in NLP specially for similarity measure

Natural Language Processing (NLP)

NLP is a branch of Artificial Intelligence that enables computers to understand, interpret, and generate human language.

Key Goals:

• Extract meaning from text and speech
• Generate human-like responses
• Automate language tasks (translation, summarization)

Why is NLP Important?Powers tools like Google Translate, Siri, Alexa, chatbots, and spam filters

Evolution: Rule-based → Statistical → Neural models (BERT, GPT).

Language Structure: Syntax vs. Semantics

​

How Language is Structured:

• English has ~170,000 words, but sentence construction is highly constrained
1. Why? Constraints come from grammar, syntax, and meaning (semantics)
• Each word choice limits the next possible words
• Valid sentences follow rules → e.g., “The the” is invalid

Syntax and Sematics:

There are rules that govern the order in which words can appear.

Example of Syntax vs. Semantics in NLP:

• "Time flies like an arrow." (Syntactically correct, semantically clear).
• "Fruit flies like a banana." (Syntactically correct, but ambiguous meaning).

UNDERSTANDING LANGUAGE IS HARD

5

Challenges in NLP

• Ambiguity – Words can have multiple meanings
• e.g., "He sat by the bank." → financial institution or riverbank.
• Context Understanding – Sentences change meaning based on situation.
• e.g., cold ice cream vs. cold soup.
• Slang & Informal Text – Handling emojis, abbreviations (e.g., "BRB," "LOL").
• Multilingual Support – Different languages have different grammar rules.
• e.g., "Das war nicht schlecht." (German) → "That was not bad."

NLP’s practical applications

​

Sentiment analysis: Determines if the tone of a text is positive or negative, handling ambiguity in sentiment and tone.

Topic modeling: Identifies the main topics of an article, overcoming challenges in word meaning and context variability.

What type of article is this?

• Sports
• Political
• Dark comedy
• Reality TV

Question Answering: Develops systems that can answer questions in any natural language, addressing ambiguity in phrasing and grammar.

Named Entity Resolution: Identifies people, places, or things in text, resolving ambiguity about what a phrase refers to.

Text Summarization: What it sounds like: Condensing a text to its essential points.

Pipeline of NLP

Text Acquisition (Data Collection)

• Gather raw text data from sources like:
• Websites (Web Scraping, APIs)
• Databases (SQL, NoSQL)
• Documents (PDFs, Word files)
• Social Media (Twitter, Reddit)
• Example: Scraping customer reviews from an e-commerce site.

Text Preprocessing

Advanced Preprocessing:

​

​

​

​

​

Text Preprocessing in NLP

Transform raw text into a clean, consistent, and structured format to prepare it for feature extraction and modeling.

1. Text Cleaning: Removing noise, unwanted symbols, and irrelevant data.
2. Tokenization – Split text into words/tokens (making it easier for machines to process).
1. Example: "I love NLP!" → ["I", "love", "NLP", "!"]
3. Lowercasing – Convert all text to lowercase. Ex. treat "Apple" and "apple" as the same word.
4. Stopword Removal – Remove common words (the, is, and).
5. Stemming/Lemmatization – Reduce words to root form.
• Stemming: "running" → "run" (fast but crude).
• Lemmatization: "better" → "good" (more accurate).
6. Handling Punctuation & Numbers – Remove or normalize.
7. Spell Correction – Fix typos ("NLP is awsome" → "awesome").
8. Special Character Removal: Remove punctuation (e.g., "Hello!" → "Hello"), numbers, emojis, HTML tags (e.g., "<b>good</b>" → "good"), and other non-alphabetic symbols. Extra spaces, tabs, and formatting characters are also cleaned to ensure uniform text.

English-Specific Challenges:

• Contractions: e.g., "I’m" → should be split into ["I", "am"].
• Possessives: e.g., "France’s" → should be split into ["France", "’s"].
• Hyphenated Words:� "Hewlett-Packard", "US-Student" — should they be one token or split? Depends on context and use case (e.g., NER vs. sentiment).
• Named Entities / Phrases: "San Francisco" → ideally kept as one token for NER tasks.
• Special Characters:� Remove symbols if meaningless (e.g., “$”, “%”),� but preserve within meaningful tokens (e.g., "C++", "e-mail").
• Case Sensitivity & Capitonyms:� "May" ≠ "may", "Pole" ≠ "pole" — case changes meaning.� Lowercasing may cause semantic loss.�

Challenges in Other Languages:

• French:� "L'ensemble" → split into ["L'", "ensemble"] or keep as one depending on task.�
• German:� Long compounds (e.g., "Lebensversicherungsgesellschaftsangestellter") must be split into meaningful subwords.�
• Chinese/Japanese:� No spaces between words — requires segmentation algorithms like Jieba (Chinese) or MeCab (Japanese).�

Tokenization is not straightforward!

Stemming or Lemmatization

​

Goal: Reduce inflected words to a common form so that they're counted as one.

​

Advanced: POS Tagging (Optional for many ML tasks)

Part-of-Speech (POS) tagging → annotate words with lexical categories

Goal: assign a lexical category such as noun, verb, adjective, etc. to each word, needed for lemmatization, helps machines understand grammatical structure.

Next: Feature Extraction: Turning texts into vectors

​

Convert text data into numerical representations (e.g. numerical vectors that represent text features like word frequencies or semantic meanings) that machine learning models can understand and use for tasks like sentiment analysis, classification, or translation.

Example:

"Hi. This is an example sentence in an Example Document." → [hi, example, sentence, document] → [1, 2, 1, 1]

Machine learning algorithms require numerical inputs for processing.

Feature Engineering / Feature Extraction

Also called as “Text Representation”

Raw Text Documents → Data Cleaning by Tokenization → Vectorization

(BoW/N-Grams/TF-IDF etc.)

Feature Extraction Techniques

1. Bag of Words (BoW)

Converts text into word frequency vectors (ignores word order).

• Example:
• Doc 1: "I love NLP and machine learning."
• Doc 2: "I hate spam emails but love NLP."
• Vocabulary: [I, love, NLP, and, machine, learning, hate, spam, emails, but]
• BoW Vectors:
• Doc 1: [1, 1, 1, 1, 1, 1, 0, 0, 0, 0]
• Doc 2: [1, 1, 1, 0, 0, 0, 1, 1, 1, 1]

Limitation: Loses word order & context (e.g., "not good" vs. "good").

counts the frequency of words in a document disregarding grammar and word order.

BoW represent each document as a vector of word frequencies: Order of words does not matter, just #occurrences

Corpus: Consider, our textual data -

1. Document 1: "I like school and I like homework"
2. Document 2: "I have homework to do"
3. Document 3: "School has homework"
4. Document 4: "Homework has school"

​

Next Step: Design the Vocabulary: make a list of all of the words in our model vocabulary.

The unique words here (ignoring case and punctuation)

​

​

Example: BoW or Unigram Approach

Final Step: Bow: Create Document Vectors: score the words in each document.

a fixed-length document representation of 9, with in each vector, the numbers represent the frequency of the corresponding word in the given sentence.

*Empty are 0

Pros and Cons: Bag of Words (BoW) or Unigram Approach:

​

​

​

​

​

Limitation of BoW :

"I like school and I like homework"

"I have homework to do"

"School has homework"

"Homework has school"

"School has homework" and "Homework has school" have the same unigram representation, despite having different meanings (Limited Context Understanding).

(2) N-Grams

A variation of Bag of Words that captures sequences of N adjacent words.

How it works: Instead of single words (unigrams), it considers pairs (bigrams) or triplets (trigrams) of words, capturing local word patterns.

• Each word or token is called a “gram”.
• Capture local context: "not good" (Bigram) shows different sentiment than "not" and "good" individually.
• Challenges: Higher N = more features → sparse data; Misses long-distance dependencies

A more sophisticated approach is to create a vocabulary of grouped words.

• N-Grams are contiguous sequences of N items from a given text, typically words or characters.
• They capture the context and sequential information present in the text.

(2) Example: Bi-Grams

Creating a vocabulary of two-word pairs is, in turn, called a bigram model.

Examples:�"School has homework"

"Homework has school"

​

​

​

​

These are called 'bigrams'

Note: only the bigrams that appear in the corpus are modeled, not all possible bigrams.

​

​

Next: TF-IDF

While N-grams help capture word sequences and local context, they still rely on raw frequency.

• This means common words like 'the' or 'very' may dominate the feature space, even though they carry little unique meaning.

To understand, let's Try: Topic Modeling which is Just Supervised Learning!

Topic Modeling!

Assign a general topic to a set of documents (corpus) that best describes and fits the contents of those documents (based on the words present).

Key Challenge: While Bag of Words and N-Grams provide raw counts or frequencies of words, but lack of understanding the importance of each word (or N-gram) in the context of classification.

Example:

​

Thought Experiment: Classifying Documents with Just Two Words: Spaceship vs. Dragon

A Thought Experiment: Seeing if two books are on the same topic

Imagine books/documents made up of only two words: Spaceship or dragon.

The more occurences of "spaceship" it has, the more purely science fiction it is. The more occurences of "dragon" it is, the more fantasy.

​

What about this issue?

Let's include another common, non-stop word

​

• Common words can overshadow more meaningful/ important words like "spaceship" or "dragon".
• Need a way to weigh words by importance. → TF-IDF (a way of measuring how relevant a word is to a document in a collection of documents.)

(Term Frequency-Inverse Document Frequency)

Weights words by their importance in a document vs. entire corpus. TF-IDF tries to decrease the weight of words that occur across many documents → lower the weight of common words.

Measures how often a specific word (term) appears (frequency) in a single document.

Document Frequency (DF) = Measures in how many documents a word appears in the entire corpus.

DF(t)=Number of documents containing term

Example 1: TF-IDF

​

• N = Total documents,
• DF(t) = Number of documents containing term t

Example: In a Corpus, there are 2 (N=2) documents

• Document 1: "NLP is amazing."
• Document 2: "I love NLP and AI."

TF-IDF for term "NLP":

• TF in Doc1: TF(“NLP”,Doc1) = 1/3, DF("NLP") = 2, IDF = log(2/2) = 0 → TF-IDF = 0 (not unique/common across all documents)).
• TF in Doc2: TF(“NLP”,Doc2) = = 1/5, DF("NLP") = 2, IDF = log(2/2) ≈ 0 → TF-IDF ≈ 0(not unique/common across all documents)).
• Try: TF-IDF for term "AI":

Use Case: Search engines (ranks documents based on keyword relevance).

Example 2: TF-IDF

Table: DF & IDF values

New Example:

Example: Document A

Total words in A: 10(Dragon)+0(Spaceship)+20(They)=30. So:

• TF(Dragon, A) = 10/30=0.33
• TF(Spaceship, A) = 0
• TF(They, A) = 20/30=0.67

Step 1: Calculate TF

​

Calculate TF for all other documents

Step 2: Calculate TF and IDF

log(6/4) =

log(6/4) =

Identify N

Documents: A, B, C, D, E, F → N=6

Example 2: TF-IDF

Table: TF values

Table: DF & IDF values

Table: TF-IDF (TF * Log(N/Df) )

Example 3: TF - IDF Calculation

Document 1: read SVM algorithm article dataaspirant blog

Document 2: read randomforest algorithm article dataaspirant blog

​

These documents differ mainly in the algorithms they discuss: SVM for the first and random forest for the second. This distinction is clear in TF-IDF scores, where all other words score 0, while 'SVM' and 'randomforest' have non-zero values

Measure similarity between documents: Cosine Similarity

Cosine Similarity measures the angle between two vectors — the closer the angle to 0°, the more similar the texts are.

Example: Use Case: Finding Similar Documents or Plagiarism Detection

• Preprocessing: Tokenization, stopword removal, stemming/lemmatization.
• Vectorization: TF-IDF (or word embeddings).
• Similarity Calculation: Compute cosine similarity between the query and each document in the corpus.
• Ranking: Rank documents based on cosine similarity scores.

Emphasizes the direction, not the magnitude (total occurrences), of the vectors.

Output: A value between -1 (completely dissimilar) to 1 (completely similar):

• Value of 1 : the vectors have the same directions/orientations.
• Value of -1: indicates opposite orientations.

A Thought Experiment: Seeing if two books are on the same topic

Once we compute TF-IDF scores, each document becomes a vector—a list of numbers representing how important each word or phrase is to that document. For example, a book might be represented as:

[0.8 (spaceship), 0.2 (dragon)]

A natural idea is to use distance to compare how similar or different two documents are.

A Thought Experiment: Seeing if two books are on the same topic

Looking at this graph, should we use Euclidean distance?

Example: Two books with similar occurrences of "spaceship" and "dragon" may appear close in Euclidean distance.

• But differences in magnitudes (total occurrences) can lead to dissimilarities.

A Thought Experiment: Seeing if two books are on the same topic

Looking at this graph, should we use Euclidean distance?

​

For instance, using "dragon" 10 times versus 100 times is actually similar, but Euclidean distance might not reflect this accurately.

• This is because Euclidean distance focuses on the spatial distance between points rather than the angle between them

We need a new measure!!

Cosine Similarity

Books on similar topics will have smaller angles.

• smaller angles indicates greater similarity → similar word distributions, even if the magnitudes (total word counts) differ.

Back to Feature Extraction: TF-IDF vs. BoW vs. N-grams

Another type: Word Embeddings (Word2Vec, GloVe) –

Dense vector representations of words that capture semantic meaning and relationships.

• Capture context beyond simple word counts.
• Learned from large corpora using models like Word2Vec, GloVe, or FastText.
• Example (Word2Vec):
• "King" – "Man" + "Woman" ≈ "Queen"
• Similar words have similar vectors (close in space).

Advantage: Captures relationships (synonyms, analogies).

So far, we have seen some vectorization process with BoW, N-Grams, TF-IDF. We can choose appropriate vectorization and will apply to some NLP task (next topic).

Model Building

​

Train a machine learning model to perform tasks like:

• Classification (e.g., spam vs. non-spam) : Supervised learning where the model learns from labeled data to predict categories.
• Clustering (e.g., grouping news articles): Unsupervised learning to group similar items based on features without labels.
• Named Entity Recognition (NER) (e.g., identifying names, places, organizations, etc.): Extracting specific types of information from text using NLP models.
• Topic Modeling (e.g., discovering themes like politics, sports, or tech in a set of news articles): Unsupervised learning technique used to automatically find hidden topics in large text corpora.

Example: Text Classification & Sentiment Analysis

Text Classification: Assigning categories (e.g., spam detection).

Sentiment Analysis: Detecting emotions (positive/negative/neutral).

​

Algorithms Used:

• Naive Bayes
• Logistic Regression
• Deep Learning (RNNs, Transformers)

Example: Sentiment Analysis Using TF-IDF and Logistic Regression

From Counting Words to Understanding Language

To go beyond static vectors, we turn to deep learning models that read text like humans do—sequentially and contextually:

The Evolution of Context-Aware Models

• 🔁 RNNs (Recurrent Neural Networks) – process text sequentially, one word at a time, remembering what came before.�
• 🧠 LSTMs (Long Short-Term Memory) – an improved RNN that handles longer dependencies and reduces memory loss over time.

​

• GRUs (Gated Recurrent Units) – similar to LSTMs but with a simplified architecture, faster to train, and still effective at capturing long-term dependencies.�
• ⚡️ Transformers – look at all words at once, using self-attention to learn relationships between words no matter how far apart they are.

https://aiml.com/compare-the-different-sequence-models-rnn-lstm-gru-and-transformers/

https://sungsoo.github.io/2021/06/29/illustrated-guide.html

https://www.geeksforgeeks.org/rnn-vs-lstm-vs-gru-vs-transformers/#

​

Evaluation & Deployment

Evaluation Metrics:

• Accuracy, Precision, Recall, F1-Score (for classification)
• BLEU: evaluate the quality of text generated by machine translation models, based on n-gram precision
• ROUGE (for translation/summarization): evaluate automatic summarization and translation by comparing n-grams between the generated and reference texts.)

​

Deployment:

• API (Flask, FastAPI): Lightweight web frameworks to build APIs for serving machine learning models, allowing integration with various applications.

​

• Integration into apps (Chatbots, Search Engines): Embedding models into application

Basic Tools and Libraries

• NLTK (Natural Language Toolkit): A comprehensive library for working with human language data, offering tools for text processing, tokenization, stemming, and parsing.�
• spaCy: A fast and efficient NLP library designed for production use, offering pre-trained models for tokenization, part-of-speech tagging, and named entity recognition.�
• scikit-learn�
• TextBlob: An easy-to-use library for processing textual data�
• Transformers (by Hugging Face): A powerful library that provides pre-trained models for state-of-the-art natural language understanding and generation tasks, based on architectures like BERT, GPT, and T5.�
• TensorFlow / PyTorch (for deep learning): Deep learning frameworks that support training and deploying complex neural networks, widely used for NLP tasks like text classification, translation, and generation.

​

Example: Sentiment Analysis (SELF STUDY)

Example: Step by Step: Sentiment Analysis

Lexicon: Pre-defined word lists with:

• Polarity: POS/NEG/NEU
• Emotion: Joy 😊, Anger 😠
• Strength: "good"=+1, "amazing"=+3, disaster"=-4

Types

✅ POS Lexicon: great, love, perfect

❌ NEG Lexicon: bad, hate, terrible

Applications

• Quick sentiment analysis
• Enhances ML model features

Limitations

• Misses sarcasm ("Just what I needed...")
• Requires domain adaptation

Most commonly used

Another one: TF-IDF : Simple ideas. Let’s: • disproportionately weight the common words that appear in many documents • Use that info and combine it with the word frequency info

Try: Bag-of-words (BoW)

Steps:

Step: Negation handling

The process of detecting and correctly interpreting the negation in a sentence—especially to avoid reversing or misclassifying the meaning. Negation flips the meaning of words:

• "I am happy" → Positive
• "I am not happy" → Negative

Types of Negation:

• Explicit:“I do not like this movie.”
• Implicit:“I hardly enjoyed the event.”
• Affixal: “This is meaningless.”

How to Handle It:

• Negation with positive interpretation: Negated Negative: Not bad → Good
• Identify Negation Words (e.g., not, never, no, can’t);
• Mark or Modify Words within the negation scope (usually next 2–5 tokens): Example: "not good" → good_NEG
• Use Modified Tokens (e.g., good_NEG) as features

Steps: Stemming/Lemmatization and Feature Extraction

Simplified: Rule-Based Sentiment Analysis Using Lexicons

Core Idea: A simple unsupervised method that calculates sentiment using predefined word sentiment scores.

​

Lexicon/Dictionary:� Predefined list of words with sentiment weights:

• Positive: +1 (e.g., "great")
• Negative: -1 (e.g., "terrible")
• Neutral: 0 (e.g., "the")

Weight Vector (w):� Vector of sentiment scores for each word�Document Vector (X):� Binary vector (1 = word present, 0 = absent) representing the text

How It Works:

Scoring: Score = ∑(Xⱼ × wⱼ) (dot product of document and weight vectors)

Prediction:

• Positive if score > 0
• Negative if score < 0
• Neutral/Positive if score = 0

Example: X = [1, 0, 1, 0, 1, 0, 1, 1]

w = [0, 0, 0, 0, 1, 0, 1, 0]

• Score = 2 → Prediction = Positive

ML based Based Sentiment Analysis

Collect Data: Labeled text (e.g., reviews, tweets)�

Preprocess Text: Clean, tokenize, remove stopwords, lemmatize�

Feature Extraction:Convert text to vectors (BoW, TF-IDF, embeddings)�

Split Data: Train/Test (e.g., 80/20)�

Train Model: Use ML classifiers (e.g., Logistic Regression, SVM, Naive Bayes)�

Evaluate: Accuracy, Precision, Recall, F1�

Predict: Apply model to new/unseen text

Input: Text (e.g., "This phone is amazing!")�

Label: Sentiment class (e.g., positive)�

Goal: Learn patterns that map input text to the correct label.

Rule Based: What Makes a Good Sentiment Lexicon?

• Coverage:� Includes many sentiment words, including slang and domain-specific terms (e.g., “lit”, “charged”).�
• Polarity Accuracy:� Assigns correct sentiment scores (e.g., “excellent” = +3, “terrible” = -3); avoids vague terms like “mad”.�
• Handles Variants:� Covers synonyms, word forms, emojis, and phrases like “not bad”.�
• Context Aware:� Adapts to meaning changes in slang or culture (e.g., “sick” = good in slang).

​

The End