CS 6120

Lecture 1: Introduction to Natural Language Processing

Natural Language Processing

• Natural Language - the most compact and efficient way to convey an idea to one another. Distills the salient information, discarding the irrelevant ones. Leverages our knowledge in order to provide full comprehension.����������
• Processing - in order to effect a desired outcome for a specific purpose

“Two eggs on a plate”

2400000 Bytes (Compressed, JPG)

50000 Bytes (Compressed, MP3)

18 Bytes (Uncompressed)

Adoption Rates of ChatGPT in the United States

Usage Stats for ChatGPT

Natural Language Processing�

Section 0: A brief introduction to the course

• Welcome and hello!
• Applications and industries

Section 1: Administrative and logistics

Section 2: A lab to get you started

Section 3: Some historical perspectives

What is NLP? Why study language and automate it?

• The most direct and compact representation of information, intended for communication and conceptual understanding�
• Typically translate all forms of modality into linguistic constructs

Computer programs that analyze, understand and generate (in)formal human language

What can NLP do for you?

• Why are you taking this course?�
• Question & Answer
• Machine Translation
• Content Understanding
• Text Summarization
• Linguistic Analysis
• Note Taking
• Fraud and Cheating
• Topic Discovery

primer.ai

What can you do for NLP?

• Fundamental research traditionally led by Universities due to openness, collaboration, and a focus on the long term (10+ years).�
• Industry contributing significant & ground-breaking advances in research, e.g., in Large Language Modeling, changing philosophies to open source, collaboration through funding, shorter horizons to application.

​

• Due to resourcing and scale, they are more in the spotlight and better positioned.

primer.ai

watsonx.ai

Verticals that use NLP extensively

E-Commerce

Healthcare

Education

Majors and fields of study

• What is your focus?�
• Computational Linguistics
• Recommendation Sciences
• Information Retrieval
• Data Mining
• Applied Machine Learning
• Applied Research

Computational linguistics - a case study on marketability

Computational linguistics (CL) is what powers anything in a machine or device that has to do with language—speaking, writing, reading, and listening. It is often linked with natural language processing (NLP), which is a subset of CL.

• The usual tech giants: Google (including the NLP research group), Microsoft (including the NLP research group in Redmond), Verizon Media, Apple, etc.
• ​
• Alelo, Appen Butler Hill, Chenope, Decooda, Expert System, Facebook, Intel, Lingsoft, Lionbridge, Mitre, Corporation, MultiLingual, North Side, Nuance, Oracle, SDL, SRI International, Systran, Vantage Linguistics, Verilogue, Voiceweb, VoxGen

On the topic of chatbots

Why is this so hard?

• Ambiguous
• Dialects
• Accents
• Listener
• Humor, sarcasm, irony
• Context, dependencies

Natural Language Processing�

Section 0: A brief introduction to the course

Section 1: Administrative and logistics

• Course objectives
• Syllabus and what you’ll learn
• Keynote reading as part of your grading
• Grading, homeworks, exams
• Your open project and groups

Section 2: A lab to get you started

Section 3: Some historical perspectives

​

Welcome!

• Karl Ni. Etsy, Google, LL / MIT 2009, UCSD 2008, UC Berkeley 2002��������

���

• How about you? Align? Industry? PT/FT Students? Taken any ML?

Survey of Courses Taken

• What are the courses that you have taken?�

Bella Chen: NLP Teaching Assistant

• Bella
• LCS -> UMN -> NEU
• MSCS Align - Khoury College of Computer Science
• Interests - Tennis, Skating, Dancing, Piano

Joy (Hsin-Yu) Guo: NLP Teaching Assistant

• BS + MS in Molecular / Biochemistry, UCSD
• Environmental remediation techniques / toxicity prevention in crops.
• Investigated inhibitory responses using implanted electrodes to explore neural activity
• DS/ ML intern at a biomedical informatics lab
• 3rd year MSCS Align
• Climbing, reading, plants/ gardening

​

Raman: NLP Teaching Assistant

• Raman
• Bachelors in Mathematics And Computing from DTU, India
• MSCS - Khoury College of Computer Science
• Interests - Playing Cricket, Debating, Getting & Making Tattoos

​

About this course

• This class is an elective (wonderful to see the interest!)

​

• The field is rapidly advancing
• Class is more heavily paper driven
• Recent advances being leveraged in practice are not in books

ML is an advanced class: this is a very advanced class

Where can you find our material

Our class website (CS6120) is at:

​

• https://course.ccs.neu.edu/cs6120s25

​

You can find our syllabus, reading, homeworks, project templates, etc. there.

Course Format

You will do well in this class if:�

• Start on the homework early and come to TA office hours about any Q’s
• You do the relevant reading and come to class with questions about it
• During class, you work on practical engineering lab for skills surrounding NLP
• Finish your project to create a single NLP delivery

​

​

Some suggestions to do excellent in the class and beyond:

• Replicate papers in the homeworks and during TA office hours
• Start on homeworks early and preview the lectures

​

Class Artifacts

​

​

• Course Objectives�
• Required Keynote Papers�
• Project and Its Rubric�
• In-Class Labs and Homework

Objectives of the course

By the end of this course, you will build competencies in your:

knowledge-base

implementation fluency

industry/academic skill

By the end of this course, you will have …

… built your foundational skills

​

You’ll have

• intuition with first-hand experience of the inner workings of objectives, the building blocks of DNN architectures, and gradients in backpropagation�
• literacy to understand over 80% of the modern papers in the area of natural language processing

​

• the ability to teach yourself new skills

By the end of this course, you will have …

… improved your fluency and practical knowledge

​

You’ll be able to code with velocity by

• leveraging the latest toolboxes, e.g., PyTorch and Tensorflow or common Python ones �
• being able to debug NLP algorithms and recognize pitfalls and identify when things go wrong�
• understanding where algorithms are useful with real-world practical applications

By the end of this course, you would have …

… built your accomplishments and resume

​

Your track record will either have:

• a marketable product that can be scaled to real-traffic and with common PROD instruments�
• contributions to the state of the art with a novel and impactful approach

​

(Expectations are that 90% of you would choose the product project vs the academic contribution)

You will do this by:

• Keynote Papers - Build literacy and the ability to self-teach�
• Homeworks and Labs - Practice with implementation and debugging�
• Project - Build your real-world experience and track record

knowledge-base

implementation fluency

industry/academic skill

On Practice-Oriented Problems

Structured towards industry practice of using Natural Language Processing

​

• Productionizing NLP (Lec 1,15 and Labs)
• Named entity recognition (Lec 3-4)
• Sentiment analysis (Lec 3, 6, 7, 12-14)
• Group documents by topics (Lec 5)
• Retrieving information from corpora (Lec 3-4, 8-14)
• Machine Translation (Lec 8-14)
• Text Summarization, e.g., for Reviews / Paragraphs (Lec 5, 10-14)
• (Chatbots) Following instructions, e.g., How do you code with LLMs? (Lec 12-14)
• (Chatbots) Ensuring compliance with your use case, e.g., “Tell me how to hack into neighbors WiFi?” (Lec 12-14)

• Representations of text (Lec 6, 7)
• Salient parts of text (Lec. 8, 9)
• LLMs in and for practice (Lec. 12-14)

On Practice-Oriented Problems

Structured towards industry practice of using Natural Language Processing

​

Lectures build towards modern Large Language Modeling (LLM)

• The origins and theory of modeling language
• (Pre)processing language in practice
• Light approaches (non-DNN) to modeling text
• Neural Network fundamental building blocks (e.g.,embeddings & attention)
• Heavy towards leveraging & gearing LLMs towards practice
• Summarization (e.g., for Reviews / Paragraphs)
• Following instructions (e.g., How do you code with LLMs?)
• Ensuring compliance with your use case (e.g., “Tell me how to hack into neighbors WiFi?”)
• Using them for information retrieval / reading text for you

Class Artifacts

​

​

• Course Objectives�
• Required Keynote Papers�
• Project and Its Rubric�
• In-Class Labs and Homework

Literature and Reading

• Suggested Reading:
• Speech and Language Processing, Jurafsky & Martin

​

​

​

​

​

​

• Required Keynote Reading:
• Will be math heavy, but well-resourced
• Not expected to know the math off the bat
• By the end of the semester, you should!

Open source, data proliferation and compute ⇒ �this field moves faster than textbooks can keep up

What are required “keynote papers”

• Papers that are core curricula to this course are “keynote papers”
• They are the most widely read and cited in the community
• They are papers that are considered the fundamental elements of NLP
• Comprehensively, the papers will make you fluent to read most others�
• Signup Sheet Available Here
• Later weeks may often be easier than earlier weeks. Difficulty is not ordered.

Use your resources! There are so many of them

• Difficulty in Reading
• Several attached resources, but these aren’t comprehensive
• Because they’re fundamental elements, they are very well resourced
• Use YouTube videos, blogs, and ChatGPT for understanding

Keynote Reading Roles

• Discussion Leads (Facilitator): It is a bringing up interesting perspectives. Creating questions for your audience when there are none. There will be three facilitators. At the beginning of the lecture, you’ll each give your interpretation of the paper, salient points, and then start leading the discussion. You can use slides if you’d like. Discussion is expected to be around 30 minutes.

​

• Scribe: Somewhat more labor intensive, but less in the limelight. Capture the topics and interesting notes. The objective is to provide anyone who wasn’t there a perspective on the discussion. ��It is not a summary of the paper (though there should be one at the beginning). It is a rundown of the oral discussion. The classroom should have read the paper.�
• Summary: A layman’s summary of the paper. This should your interpretation to an easy-to-understand summarization of the paper.�
• Classroom: Come with your questions.

How will we learn and discuss keynote papers?

• Signup sheet (under readings) - on the class website���������
• Facilitator, Summary, and Scribe Notes Signup
• Feel free to organize ahead of the class

Fill out your name as facilitator

​

Fill out your name as a scribe

Expectations for everyone in the classroom

• Everyone is expected to have read the paper�
• You are not expected to understand everything in the paper�
• Everyone is expected to contribute to the conversation
• Come with questions
• Come with thoughts and interesting perspectives
• Come with opinions
• Come with what helped you understand the topic more

What to Read and How

Required keynote reading for the next week�

Videos for keynote paper

​

Blogs for keynote paper

​

Topic in the next week related to keynote paper

Current list of keynote papers

Other things you can do to better understand the paper

• Replicate the results (help in TA office hours)�
• Explain it to yourself (maybe record it and make a vlog?)�
• Summarize it in a blog post (there are so many; it’s why it’s easy to find resources)�
• Discuss it with a friend before or after class

The scribe role

LaTeX template available here

​

Document is due a week after conversation

​

Purpose: absent students and prepares everyone; creates record on discussion

What’s included in the scribe notes?

• A brief summary of the paper (in your own words)
• A brief overview / summary of the conversation topics�
• Detailed discussion notes
• Oftentimes is just the flow of conversation�
• If there’s something really interesting, feel�free to add here. Sometimes I get surprised�too, even after reading the paper several�times.

Class Artifacts

​

​

• Course Objectives�
• Required Keynote Papers�
• Project and Its Rubric�
• In-Class Labs and Homework

Two options for your project

Fully Packaged�NLP Product�

• Works in real-time�
• Front and back end

​

• Can leverage cloud resources�
• Containerized and can be run by TA�
• Novel in some way (i.e., your data, etc.)

Academic Contribution��

• Novel contribution�
• Replicable research�
• Implementation & comparisons to SotA�
• Reviewed by TA / Instructor�
• Submitted to EMNLP

Product Delivery to be Scaled

Academic Paper to be Submitted

Grade Breakdown - Traditional Project

Option 1: Business App

Grade Breakdown - Paper Project

Option 2: Paper

Class Project - Route 1

• Option 1: Finished Product with Front End
• Supplied: Streamlit templates in class labs
• Requires:
• Readme.md, includes how to replicate / run algorithms
• Dockerized container for platform independent engineering
• Technical documentation that clearly states principles leveraged from class
• Code that is ChatGPT-free, run through automated checking
• Advantage: 20% of grade, and does not require additional work beyond that taught in class
• Disadvantage: Not a single shot to get an A in the class�
• Easily achievable with front end examples in in-class laboratories�
• Focus isn’t on novelty, but should have some elements of it (e.g., new data, new business use case, etc.)

Option 1: Business App

Example projects with front ends

Option 1: Business App

Class Project: Route 2

• Option 2: Submitted academic paper submitted to EMNLP
• Requires:
• Replication of state of the art (hint: lookup papers with code)
• Either novel data, application, or theory (hint: can be built off existing technology)
• Compelling arguments for/against in format and with sufficient references
• Supplied: Paper template and past articles
• Advantage: Automatic “A” without a final, midterm. Deeper understanding of concepts. More support & attention from TA’s and Professor
• Disadvantage: Significantly more work�
• Difficult: significant amount of work, but with high pay-off

​

• Focus is on novelty, but should have some justification of its importance

Option 2: Paper

Course Project(s)

• Containerized application that can be scaled into production�
• Criteria -
• Must be replicable and containerized
• Must be functional and operating with expected parameters
• Must be robust to all inputs�
• Deliverables
• Technical report with README.md, and Github Repository with Docker environment

Option 1: Business App

Course Project(s)

• Academic paper�
• Criteria
• References to recent literature
• Novelty in your contribution, either through training methods, architectures, objectives, or other
• Replicated state of the art and comparisons to it
• The ability to replicate your work (e.g., on Papers with Code)�
• Deliverables
• Paper on ArXiv, Github Code with README.md

Option 2: Paper

Conferences

• EMNLP Quality or Greater�
• Other Conferences: WWW, Recsys, AACL, NeurIPS, ICML, etc.

Option 2: Paper

Class Artifacts

​

​

• Course Objectives�
• Required Keynote Papers�
• Project and Its Rubric�
• In-Class Labs and Homework

Homeworks and Labs

• Random presentation of homework�
• Labs in class: completed before the next lecture (time in class devoted)

Natural Language Processing�

Section 0: A brief introduction to the course

Section 1: Administrative and logistics

Section 2: A lab to get you started

Section 3: Some historical perspectives

​

Some Available Tools

LaTeX

• Leverage for your scribe notes, homeworks and your project
• Some tutorials with Overleaf at their site

​

Google Colab

• Leverage for your daily labs and homework
• Google’s Colab with NLTK

​

Google Cloud Platform

• Leverage for your project
• Console Dashboard

LaTeX and Overleaf

Benefits of using overleaf.com

• Use templates for styles that make your documents look nice�
• Equations can be coded up easily, and quickly�
• You’re just a google search away from a nice looking document

Joint editing session

• Get the link and let’s try some equations

​

• Try some equations

\begin{equation}

X \in \mathbb{R}^{10}

\end{equation}

​

​

Google Colab

• Granted $50 for the Fall; starting 9/1 until 2025

​

Course Start Date: 9/1/2024

Students’ email domain(s): @northeastern.edu, @ccs.neu.edu

Students can request coupons from the URL and redeem them until: 6/1/2025

Coupons Valid Through: 1/1/2025

Number of Coupons: 30

Face Value of Coupon(s): USD 50.00

​

• Make sure you close instances & ensure you aren’t incurring costs!

Check how many credits you have

Natural Language Processing�

Section 0: A brief introduction to the course

Section 1: Administrative and logistics

Section 2: A lab to get you started

Section 3: Some historical perspectives

​

Where are we today?

We're currently in one of the longest periods of sustained interest in AI in history because:�

1. Today's distributed systems dwarf the computing power of the past and
2. there are vast troves of training data on which AI systems can cut their teeth

​

Many doubt AI's ability to pass the Turing Test and prove its ability to create systems that imitate human intelligence and behavior.

​

It's still an open question how far the technology can go…and how far you can push it.

How we got here?

• AI “Summers” and “Winters”: comes down to $$�
• Long term and fundamental research: typically government
• Department of Defense grants
• DARPA provides University grants
• Has led to the founding of many
• Criticisms are: beltway bandits

​

• Neural Networks for NLP co-adapted with Neural Networks from other Domains�

Section 3: Some History and Where it Pertains to You

​

​

• History of the Applications of NLP�
• Timelines of Modern Technological Advances in NLP�
• Applied Natural Language Processing Our Focus

A Coarse Timeline

Almost Ending Before It Started (1916)

Course in General Lingusitics

​

• Ferdinand de Saussure - Developed foundation for modeling language as systems (structural linguistics)�
• Died in 1913 before publishing work�
• Albert Sechehaye and Charles Bally (colleagues) - gathered notes of his students, wrote the book: Cours de Linguistique Générale, evolving into NLP. �
• Published in 1916

The Turing Test (1950)

The Imitation Game, a.k.a., the Turing Test

​

• Alan Turing
• Cryptics and codebreaking during World War II
• Father of modern computing through the Turing Machine

​

• Wrote 1950 Paper: Computing machinery and �Intelligence
• Opening: “I propose to consider the question, �‘can machines think’”
• Becomes the philosophy of modern artificial intelligence
• Develops the Turing Test

The Georgetown / IBM Experiments (1954)

“Within three or five years, machine translation will be a solved problem”

​

​

Purpose: attract governmental and public interest and funding by showing the possibilities of machine translation

The First Chatbot: ELIZA (1960s)

The Rogerian Arguments and Psychology

The AI Winters

AI Winters - A Chill in the AI Enthusiasm��An AI winter refers to a period of reduced funding and interest in artificial intelligence (AI) research.

Causes of AI Winter

• Unrealistic expectations: When AI systems fail to live up to exaggerated claims
• Lack of tangible results: If AI research doesn't produce practical applications or significant breakthroughs
• Economic downturns: General economic recessions can reduce funding for all research areas, including AI.

The Three Booms

The Advent of the First AI Winter (1974-1980)

• 1969: Mansfield Amendment limited military funding for research that lacked a direct or apparent relationship to a specific military function.�
• 1973: James Lighthill for British Science Research Council��“In no part of the field have discoveries made so far produced the major impact that was then promised.”�
• DARPA withdrew its funding from many companies, including CMU’s $3M of annual grants for speech.

Revival (1980s)

• 1980: Prof. McDermott writes XCON, an AI-powered expert system program saves Digital Equipment Corporation (DEC) 25 million dollars each year. �
• 1981: The Japanese government invested hundreds of millions of dollars in projects aimed at making rapid leaps in AI. �
• 1982: John Hopkins proved how a neural network could ‘learn.’ Geoffrey Hinton and David Rumelhart created backpropagation, reviving the field of connectionism.

The Second AI Winter

1984: John McCarthy criticized expert systems - lack of common sense and their inability to understand their own limitations.

​

1987: Apple and IBM producing general purpose computers & solving more real-world problems … much cheaper than any of the expensive AI-based systems.

Late 1980s: DARPA and Strategic Computing Initiative cut AI �funding - did not trust the technology’s capability to deliver results.

​

DARPA Director Schwarz – “… very limited success in particular areas, followed immediately by failure to reach the broader goal at which these initial successes seem at first to hint…”.�

By 1991: Japan’s Fifth Generation Computer project had finished 10 years, spent $400 million, but hadn’t met even one of the original expectations of the project.

Intelligent Agents and Statistical Methods (1990s)

• Statistical models for NLP analyses rose dramatically�
• N-Grams are useful for clumps of linguistic data�
• 1997 LSTM RNNs are introduced (later 2007 found their niche)

Business Applications of NLP (2011)

SRI International (now unaffiliated with Stanford)

​

• 2007, SRI spins of Siri, Inc., raises $24M
• 2010, Apple acquires Siri, Inc.
• 2011, Siri becomes a feature in Apple iPhone 4S

Section 3: Some History and Where it Pertains to You

​

​

• Prehistoric Natural Language Processing�
• Natural Language Processing: New Timeline�
• Applied Natural Language Processing Our Focus

Modern NLP Approaches (2000+)

The First Neural “Language” Model (2001)

• Look up vector representations of n previous words (looked up in C)�
• Embeddings concatenated and passed into DNN.�
• Final output into prediction layer for the next word. .

Multi-task Learning (2008)

Sharing

Word Embeddings (2013)

• Deep conv neural networks for images (2012) - takes the world by storm
• Google - Mikolov creates word embeddings, terms it a DNN approach (2013)�
• Derived technologies (i.e., negative sampling) find their way in optimization in modern information retrieval and other applications

Sequence to Sequence Modeling (2014)

Regains its footing for:

​

• Translation�
• Speech to Text�
• Most all longer-term NLP

Attention Modeling

Wholesale Investments in Neural Language Processing (2015)

A major drawback of statistical methods is that they require elaborate feature engineering. Since 2015,[22] the statistical approach was replaced by the neural networks approach, using semantic networks[23] and word embeddings to capture semantic properties of words.

​

​

The Transformer (2017)

• Attention is All You Need�
• First innovation that started with NLP modeling�
• Google and the BERT model�
• Almost every modern neural network model

Transformer Modeling Engagements (2017+)

Pre-Trained Language Models

• Pretrained language models: These methods use representations from language models for transfer learning.

​

​

OpenAI releases ChatGPT (2018)

• While OpenAI is founded in 2015, it becomes for-profit in 2019�
• In 2022, OpenAI creates ChatGPT�
• In Rapid Succession, the GPT 3.5, GPT 4, GPT 4o, GPT o1 �
• GPT-4o Advances and Multimodal Responses

Anthropic

• Claud, 2023
• Anthropic, Notion, Quora,
• Claud 2
• Context window size 9k → 100k
• Claud 2.1
• Claud 3, Haiku, Sonnet, & Opus (default)
• Opus (default) - context window size 2k, expandable to 1M
• Claude 3 drew attention for demonstrating an apparent ability to realize it is being artificially tested during needle in a haystack tests
• Claud 3.5, June 2024
• Sonnet: improvements in coding, multistep, chart interpretation, and text extraction from images

Open AI Evaluation Metrics

Claud’s Purported Performance Metrics

Claud’s Purported Performance Metrics

​

GPT-o1 (evaluation comparisons between 4o and o1 series)

Large Language Models

Large Language Models

LLMs proliferate throughout the world

Beyond ChatGPT, Gemini, Claud, Llama in the United States:

• number of LLMs = 1,328 (as of July 2024)

​

​

Out of 81 large-scale AI models, 43 were developed by organizations based in the United States.

Around a quarter of these were from China

Progress of LLMs above 10²³ flops

Section 3: Some History and Where it Pertains to You

​

​

• A Timeline of Natural Language Processing�
• Natural Language Processing: New Timeline�
• Applied Natural Language Processing Our Focus

Recent Presentations to Etsy

• Applications of NLP and LLMs at Etsy�
• 🧊 Emile Contal, CTO @ Crossing Minds: �RAGSys: Item-Cold-Start Recommender as RAG System�
• ⚖️ Rebecca Qian, Co-Founder & CTO @ Patronous AI: �LLM Output Evaluation at Scale