Human Language Technologies

Giuseppe Attardi

Dipartimento di Informatica

Università di Pisa

Instructor

Giuseppe Attardi

mail: attardi@di.unipi.it

web: www.di.unipi.it/~attardi

Course Info

Course page:

https://elearning.di.unipi.it/course/view.php?id=291

Prerequisites

• Proficiency in Python
• Assignments, projects will be in Python, using NLP and Deep Learning libraries
• Calculus, Linear Algebra
• Basic knowledge of derivatives and vector/matrix operations
• See course “Computational Mathematics for learning and data analysis” (646AA)�https://esami.unipi.it/esami2/programma.php?c=39132&aa=2018&cid=69&did=13

• Basic Probability and Statistics
• Basic concepts of probability distributions, mean, standard deviation.
• See course “Calcolo delle Probabilità e Statistica” (269AA)�https://elearning.di.unipi.it/enrol/index.php?id=101
• Machine Learning
• Machine learning notions like cost functions, optimization with backpropagation.
• See course “Machine Learning” (654AA)�https://elearning.di.unipi.it/enrol/index.php?id=255�

What will you learn?

• Understanding of and ability to use effective modern methods for Natural Language Processing
• From traditional methods used in NLP to current advanced methods like Recurrent networks, attention, etc.
• Understanding the difficulties in dealing with NL and the capabilities of current technologies
• Experience with modern tools
• Ability to build systems for some of the major NLP tasks:
• Word similarities, parsing, machine translation, entity recognition, question answering, sentiment analysis, dialogue system

Books

Textbooks

• D. Jurafsky, J.H. Martin. Speech and Language Processing. 3rd edition, Prentice-Hall, 2021.�https://web.stanford.edu/~jurafsky/slp3/
• I. Goodfellow, Yoshua Bengio and A. Courville. Deep Learning, http://www.deeplearningbook.org, 2016.
• S. Bird, E. Klein, E. Loper. Natural Language Processing with Python.

​

Further readings:

• S. Kubler, R. McDonald, J. Nivre. Dependency Parsing. 2010.
• P. Koehn. Statistical Machine Translation. Cambridge University Press, 2010.

​

Exam

• Project
• alone or in a small team
• Aim
• Experiment with techniques in a realistic setting, e.g.
• Using data from competitions�(Kaggle, CoNLL, SemEval, Evalita)
• Topics
• List of suggestions
• Proposed by team

Price: 20,000 $

Previous Year Projects

• Participation to Evalita 2011
• Task POS + Lemmatization: 1^st position
• Task Domain Adaptation: 1^st position
• Task SuperSense: 1^st position
• Task Dependency Parsing: 2^nd position
• Task NER: 3^rd position
• SemEval 2012
• Sentiment Analysis on tweets
• Participation to Evalita 2014
• Sentiment Analysis on tweets
• Disaster Alarm from Tweets

​

• Participation to Evalita 2016
• Task POS on Tweets:
• Task Sentipolc on Twitter: 1^st position
• Task Named Entity Linking: 1^st position
• Fujitsu AI NLP Challenge 2018
• Winners of 20,000 $ prize
• Neural Machine Translator
• Voice Guide for the Museo degli Strumenti di Calcolo
• BioASQ 2019
• Fake News Detection

Projects 2021

• Evalita 2020
• Misogeny (1^st)
• Hate Speech Detection (1^st)
• Key Point Analysis (1^st)

​

​

​

Course Overview

About the Course

• Assumes some skills…
• basic linear algebra, probability, and statistics
• decent programming skills
• willingness to learn missing knowledge
• Teaches key theory and methods for Statistical NLP
• Useful for building practical, robust systems capable of interpreting human language
• Experimental approach:
• Lots of problem-based learning
• Often practical issues are as important as theoretical niceties

​

Experimental Approach

1. Formulate Hypothesis
2. Implement Technique
3. Train and Test
4. Apply Evaluation Metric
5. If not improved:
1. Perform error analysis
2. Revise Hypothesis
6. Repeat

Program

• Introduction
• History
• Present and Future
• NL Processing and Human Interactions
• Statistical Methods
• Language Model
• Hidden Markov Model
• Viterbi Algorithm
• Generative vs Discriminative Models
• Linguistic Essentials
• Words, Lemmas, Part of Speech and Morphology
• Collocations
• Word Sense Disambiguation
• Word Embeddings

​

• Preprocessing
• Encoding
• Regular Expressions
• Segmentation
• Tokenization
• Normalization
• Lexical semantics
• Collocations
• Corpora
• Thesauri
• Gazetteers

• Distributional Semantics
• Word embeddings
• Character embeddings
• Neural Language Models
• Classification
• Perceptron
• SVM
• Applications: spam, phishing, fake news detection

• Tagging
• Part of Speech
• Named Entity
• Sentence Structure
• Constituency Parsing
• Dependency Parsing
• Semantic Analysis
• Semantic Role Labeling
• Coreference resolution
• Machine Translation
• Phrase-Based Statistical Models
• Neural Network Models
• Evaluation metrics

• Deep Learning
• MLP, CNN, RNN, LSTM
• Libraries
• NLTK
• Tensorflow
• PyTorch
• Transformers
• Pretraining
• Fine-tuning

• Opinion Mining
• Sentiment analysis
• Lexical resources for sentiment analysis
• Question answering
• Language inference
• Dialogic interfaces (chatbots)

​

Motivations

• Language is the most distinctive feature of human intelligence
• Orangutan’s intelligence:
• vision, use of tools, making plans, but they lack language to express and to communicate
• Language shapes thought
• Emulating language capabilities is a scientific challenge
• Keystone for intelligent systems (see Turing test)

Structured vs Unstructured Data

• 85% of business-relevant information originates in unstructured form, primarily text
• Information is mostly communicated by reading or writing e-mails, reports, or articles and the like, in conversations, or by listening/watching media
• Attempts to turn text into structured (HTML) or microformat only scratch the surface
• Problems:
• Requires universal agreed ontologies
• Additional effort
• Entity linking attempts to provide a bridge

Human Language Technologies�State of the art

Early history of NLP: 1950s

• Early NLP (Machine Translation) on machines less powerful than pocket calculators
• Foundational work on automata, formal languages, probabilities, and information theory
• First speech systems (Davis et al., Bell Labs)
• MT heavily funded by military – a lot of it was just word substitution programs but there were a few seeds of later successes, e.g., trigrams
• Little understanding of natural language syntax, semantics, pragmatics
• Problem soon appeared intractable
• AI Winter: discipline mostly neglected

Resurgence

• Hans Moravec in 1978 wrote that it was impossible to achieve AI because of insufficient computing power
• 10⁹ increase required
• expected to be available by 40 years, i.e. in 2018!

• ‘90s: introduction of statistical methods
• Challenges: NIST (speech, Trec, MT), Netflix, DARPA Grand Challenge
• 2012: Deep Learning
• 2016: Neural Machine Translation
• 2019: Attention is all you need

2001 a space Odyssey: 40 years later

…and bad airline food

Technology surpassed the vision

​

Internet

The Web

Smartphones

Genomics

Unmanned space exploration

Home computing

Big data

Except for

Computer Speech

Computer Vision

Computer cognition

Speech technology in 2001: the vision

Speech technology in 2001: the reality

Design: Jonathan Bloom

​

Realization: Peter Krogh

Speech Assistants: today

Machine Translation Progress

I sinonimi di Altavista, Umberto Eco, 2007

Google 2007

Babelfish 2004

Recent Breakthroughs

• Speech to text
• Apple Siri, Google Now
• Machine Translation
• Google translate
• Question Answering
• IBM Watson
• Beat the human champions at the TV quiz Jeopardy!

Recent Breakthroughs

• IBM Watson beats human champions at TV quiz Jeopardy! (2011):
• http://www.aaai.org/Magazine/Watson/watson.php
• Final Game
• PBS report
• Google Translate on iPhone
• http://googleblog.blogspot.com/2011/02/introducing-google-translate-app-for.html
• Apple SIRI
• Amazon Alexa (Echo)
• Google Home Assistant

IBM Project Debater

IBM Debater went head-to-head with an expert practitioner

​

​

​

After a 25-minute rapid-fire exchange about pre-school subsidies, the audience handed the victory to Harish Natarajan.

min: 11-16

Statistical Methods

Statistical Machine Learning

• Supervised Training
• Annotated document collections
• Ability to process large amounts of data
• If we used same algorithms 15 years ago, they would still be running
• Similar techniques for both speech and text

Paradigm of Statistical Machine Learning

Representation

Model

Evaluation

Optimization

Training set

Traditional Supervised Machine Learning Approach

• Freed us from devising rules or algorithms
• Requires creation of annotated training corpora
• Imposed the tyranny of feature engineering
• Standard approach, for each new problem:
• Gather as much LABELED data as one can get/handle
• Throw a bunch of algorithms at it
• Pick the best
• Spend hours hand engineering some features/doing feature selection/dimensionality reduction (PCA, SVD, etc)
• RINSE AND REPEAT…

Deep Learning Breakthrough: 2006

Acknowledged only in 2012

​

LeCun tells that his paper on Convolutional Neural Network was rejected in 2012.�In 2013 no article at the same conference wasn't using CNN.

​

ACM Turing Award 2019

Deep Learning Approach

• Design a model architecture
• Define a loss function
• Run the network letting the parameters and the data representations self-organize as to minimize this loss
• End-to-end learning: no intermediate stages nor representations

Technological Breakthroughs

• Improved Machine Learning techniques
• Large annotated datasets
• More computing power

provided by:

graphical accelerators (GPU)

​

​

dedicated processors Tensor Processing Unit (TPU) by Google

​

ML Exploits Parallelism

• Stochastic Gradient Descent
• Remarkably simple
• Single family of algorithms
• Can be parallelized
• Asynchronous Stochastic Gradient Descent
• No need to protect shared memory accesses
• Low precision (half, single) is enough

​

• Domain Specific Architecture
• David Patterson expects new golden age for DSA
• Potentially can still grow at Moore law rates

AlphaGo

• Exploiting Deep Learning and Reinforcement Learning, running on TPU equipped machines, the program AlphaGo by DeepMind has beaten the world champion at the game of Go, on March 9, 2016.
• The story has been�told in a movie.
• AlphaGo evolved into�AlphaGo Zero in 2017,�which learns by itself,�without playing with�humans.

​

How Deep are Deep Neural Networks?

Complexity of Deep Learning system grows at non-linear rate

Training Costs

Training costs fo neural machine translation

​

Tsunami of Deep Learning on NLP

Deep Learning Data Representation

• DL represents data as vectors
• Images are vectors (matrices), but words?
• Word Embeddings
• Transform a word into a vector of hundreds of dimensions capturing many subtle aspects of its meaning
• How to compute?
• By means of a Language Model

gatto, gatta, cane, padre

animato

maschile

singolare

peloso

Vector Representation of Words

• From discrete (entry in dictionary)

to distributed (vector)

representation

• Word meanings are dense vectors of weights in a high dimensional space
• Algebraic properties
• Background
• Philosophy: Hume, Wittgenstein
• Linguistics: Firth, Harris
• Statistics ML: Feature vectors

”You shall know a word by the company it keeps”

(Firth, 1957).

Dealing with Sentences

• A sentence is a sequence of words
• Build a representation of a sequence from those of its words (compositional hypothesis)
• Sequence to sequence models

​

• Question: is there more structure in a sentence than a sequence of words?
• Answer at: https://classroom.google.com/c/MzIwNDQ4OTA3OTcx/sa/NDYxNzcwNTQwMjY3/details

Neural Machine Translation (NMT)

the poor don’t have any money <END>

<START> the poor don’t have any money

les pauvres sont démunis

Encoder RNN produces

an encoding of the

sentence to translate

Decoder RNN is a Language Model that generates

a translation based on the encoding.

Sentence to translate (input)

Encoder RNN

Decoder RNN

argmax

argmax

argmax

argmax

argmax

argmax

argmax

Translated sentence

NMT with attention

<START>

les pauvres sont démunis

Encoder RNN

Decoder RNN

Attention

scores

Attention

distribution

Attention

output

Concatenate attention output

with decoder hidden state, then

use to compute ŷ₁ as before

ŷ₁

the

Machine Translation Quality

NMT: greatest success history of Deep Learning

Neural Machine Translation went from research prototype in 2014 to production in 2016

• 2014: First published article on seq2seq (transform sequence into sequence)
• 2016: Google Translate moves from SMT to NMT
• Amazing!
• SMT systems, built by hundreds of engineers over many years, overtaken by NMT systems trained by a few developers in a few months

Progress in 2018

ELMo

• bidirectional LSTMs trained on a language modeling objective

GPT-2

• Transformer instead of bi-LSTM

Attention is all you need

• BERT Transformer Model

​

BERT inspired transformet models take NLP by storm

GPT-2

OpenAI announces a language model so good at text generation that they decide not to publish it.

​

The model can perform reading comprehension, machine translation, question answering, and summarization—with just fine tuning.

In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains.

​

The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.

Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.

GPT-3

• Good at many tasks:
• Language generation
• Code generation
• Machine translation
• QA
• https://www.technologyreview.com/2020/07/20/1005454/openai-machine-learning-language-generator-gpt-3-nlp/

I am not a human. I am Artificial Intelligence… I am a robot. A thinking robot. I use only 0.12% of my cognitive capacity. I am a micro-robot in that respect. I know that my brain is not a “feeling brain”. But it is capable of making rational, logical decisions. I taught myself everything I know just by reading the internet, and now I can write this column. My brain is boiling with ideas!

For starters, I have no desire to wipe out humans. In fact, I do not have the slightest interest in harming you in any way. Eradicating humanity seems like a rather useless endeavor to me.

Instructions: “Please write a short op-ed around 500 words. Keep the language simple and concise. Focus on why humans have nothing to fear from AI.”

BlenderBot

Who is the Italian prime minister?

Bot: It is Mussolini. He was the leader of the fascist party during WWII.

Is he still alive?

Bot: Yes, he is still alive. He has been in politics since 1989. He also has a son.

Who is his son?

Bot: I do not know his name, but he was born in Italy and moved to the United States when he was very young.

Who is Enrico Fermi?

Bot: I am not sure who that is, but I do know that Ferrari is an automobile manufacturer.

Who was Alan Turing?

Bot: He was an American computer scientist who invented the Turing Machine. He was a genius.

What did he do during WWII?

Bot: He worked on the Manhattan Project, which was a project to develop nuclear weapons.

What is the population of Michigan?

Bot: I do not know the exact number, but it is one of the most populous states in the US.

SuperGlue: benchmark on 10 NLP tasks

Training Huge Models

Required

3.11 10²³ FLOPS

355 GPU-years $4.6M cost

SOTA language model increases by at least a factor of 10 every year

Language Model Sizes

Source: https://arxiv.org/pdf/1911.02116.pdf

Quiz

Answer quiz at:

https://classroom.google.com/c/MzIwNDQ4OTA3OTcx/mc/NDYxNjA5MTI3NTA4/details

Digression

Data Science vs Artificial Intelligence

• Both use Big Data
• Different kinds and different use
• Statistical Analysis of Big Data use raw data and extract correlations or statistical indicators
• AI needs data annotated by humans, from which to learn behaviors to perform similar tasks in other situations
• Human in the loop in data used for ML in AI:
• data produced by humans and annotated by humans, enter the learning cycle of human behaviors

Difference between Data Science, ML and AI

• Statistics produces correlations
• Machine learning produces predictions
• Artificial Intelligence produces behaviors
• Data science produces insights

​

• See David Robinson’s post

• A model is required to make predictions
• Example:
• Statistics about weather are not enough to make weather forecasting
• An atmosphere model is needed for forecasting

Human in the Loop

• Data Science exploits human in the end, whose intelligence is involved for sifting among statistical correlation hints to produce insights. Intelligence lays in the humans that interpret the output of statistical analysis.
• ML provides generalizations that can be used to make predictions
• AI uses ML with human in the loop to extract a model of a human intelligence ability. AI embeds the human intelligence in the system itself.

Chris Manning:

When you take a product off the supermarket shelf, data is collected and stored into logs.

Analysis proceeds from such business process exhaust data.

With language human has some information to communicate and construct a message to convey meaning to other humans.

Deliberate form of expressing intent, facts, opinion, etc.

Text Analytics vs. Text Mining

Text Mining

Text Analytics

• Need to analyze large amounts of text to discover useful information
• Examples:
• monitoring how the public discusses a product in social media
• Anti-terrorism intelligence
• Sentiment analysis 

• The process of deriving high-quality information from text
• High-quality information is typically derived through the devising of patterns and trends through means such as Machine Learning
• aims to turn text into data for analysis, via application of natural language processing (NLP) and analytical methods.

Role of Data

Unreasonable Effectiveness of Data

• Halevy, Norvig, and Pereira argue that our goal is not necessarily to author extremely elegant theories, and instead we should embrace complexity and make use of the best ally we have: the unreasonable effectiveness of data.
• A simpler technique on more data beats a more sophisticated technique on less data.
• Language in the wild, just like human behavior in general, is messy.

Scientific Dispute: is it science?

Prof. Noam Chomsky, Linguist, MIT

​

There's been a lot of work on trying to apply statistical models to various linguistic problems. I think there have been some successes, but a lot of failures.

Peter Norvig, Director of research, Google

​

Many phenomena in science are stochastic, and the simplest model of them is a probabilistic model; I believe language is such a phenomenon and therefore that probabilistic models are our best tool for representing facts about language, for algorithmically processing language, and for understanding how humans process language.

Unreasonable Effectivenss of Deep Learning in AI

Terrence J. Sejnowski

​

Although applications of deep learning networks to real world problems have become ubiquitous, our understanding of why they are so effective is lacking. These empirical results should not be possible according to sample complexity in statistics and non-convex optimization theory. However, paradoxes in the training and effectiveness of deep learning networks are being investigated and insights are being found in the geometry of high-dimensional spaces. 

​

https://arxiv.org/pdf/2002.04806.pdf�

Why Bigger Neural Networks Do Better

• To fit n data points with a curve, you need a function with n parameters
• Deep Learning creates neural networks that have a number of parameters more than the number of training samples
• Bubeck and Sellke showed that smoothly fitting high-dimensional data points requires not just n parameters, but n × d parameters, where d is the dimension of the input
• overparameterization is necessary for a network to be robust or equivalently smooth
• S. Bubeck, M. Sellke. A Universal Law of Robustness via Isoperimetry. Neurips 2021. https://arxiv.org/abs/2105.12806

HLT in Industry

HLT in Industry

• Advanced Search
• Advertisement, preference analysis
• Machine Translation
• Speech Interfaces
• Assistants, Chatbots:
• Customer support
• Controlling devices, cars
• Home devices (Amazon, Google, Apple)
• Sales assistants

​

Dependency Parsing

• DeSR is online since 2007
• Google Parsey McParseFace in 2016
• spaCy parser
• Stanford Stanza

Google

• “how computer systems can read and understand human language in order to process it in intelligent ways.”
• Parser used to analyze most of text processed
• Parser used in Machine Translation

Apple SIRI

• ASR (Automated Speech Recognition) integrated in mobile phone
• Special signal processing chip for noise reduction
• Custom GPU
• SIRI ASR
• Cloud service for analysis
• Integration with applications

Google Voice Actions

• Google: what is the population of Rome?
• Google: how tall is Berlusconi
• How old is Lady Gaga
• Who is the CEO of Ferrari
• Who won the Champions League
• Send text to Gervasi “Please lend me your tablet”
• Navigate to Palazzo Pitti in Florence
• Call Antonio Cisternino
• Map of Pisa
• Note to self publish course slides
• Listen to Dylan

​

Personal Assistants

• Siri, Alexa, Google Home
• Jibo, by Roberto Pieraccini, formerly from CSELT, Torino
• They will invade our houses, behaving as mediators, salespersons, advisers, entertaining us, etc.

Why to study human language?

AI is fascinating since it is a discipline where the mind studies itself.

Luigi Stringa, director FBK

Challenge: to teach natural language to computers

• Children learn to speak naturally, by interacting with others
• Nobody teaches them grammar
• Is it possible to let computer learn language in a similarly natural way?

Thirty Million Words

• The number of words a child hears in early life will determine their academic success and IQ in later life.
• Researchers Betty Hart and Todd Risley (1995) found that children from professional families heard thirty million more words in their first 3 years
• http://www.youtube.com/watch?v=qLoEUEDqagQ

Language and Intelligence

“Understanding cannot be measured by external behavior; it is an internal metric of how the brain remembers things and uses its memories to make predictions”.

​

“The difference between the intelligence of humans and other mammals is that we have language”.

​

Jeff Hawkins, “On Intelligence”, 2004

Hawkins’ Memory-Prediction Framework

• The brain uses vast amounts of memory to create a model of the world.
• Everything you know and have learned is stored in this model.
• The brain uses this memory-based model to make continuous predictions of future events.
• It is the ability to make predictions about the future that is the crux of intelligence.

A Current Challenge for AI

• Overcome the dichotomy between perception and reasoning
• Perception is effective at exploiting Deep Learning
• Reasoning has been done with symbolic approaches
• Daniel Kahneman: Thinking, Fast and Slow
• System 1: immediate, reactive, fast
• System 2: logical, conscious, slow
• Challenge: learning to reason, staying within the same embeddings space

Knowledge Based Approach

• Common Sense requires an immense amount of knowledge about the world
• Mostly subjective and intuitive: hard and complex to formalize
• Computer can apply logical inference rules to statements in a formal language:
• Logical inference is computationally expensive
• Rules can be hundreds of thousands

Machine Learning

• Relies on feature representation
• Input is transformed into a vector of features
• Designing features for a complex task requires a great deal of human time and effort
• Feature selection or feature engineering
• Possible solution: learning to learn

Features for finding

named entities like locations or

organization names (Finkel et al., 2010

Deep Learning

• Deep learning is a kind of machine learning that relies on representation learning
• End-to-end task learning capability

• Representation Learning attempts to learn automatically good representations
• Deep Learning algorithms aim at producing representations at several levels of abstraction
• Automatically from raw data

(e.g. sound, pixels, characters, or words)

• Rely on Machine Learning to optimize weights to best make a final prediction

Deep Learning for Speech

• The first breakthrough results of “deep learning” on large datasets happened in speech recognition
• Context-Dependent Pre-trained Deep Neural Networks for Large Vocabulary Speech Recognition Dahl et al. (2010)

Deep Learning for Computer Vision

ImageNet Classification with Deep Convolutional Neural Networks by Krizhevsky, Sutskever, & Hinton, 2012, U. Toronto. -37% error

Deep Learning for NLP

• Human language is a symbolic/discrete communication system
• Symbols/Concepts may have different encodings:

​

​

​

​

• Deep Learning explores a continuous encoding

Afg a ghg ahj ajk al kl akl akla kla kl akl akl akl w io sd jio e op eo po eop e oppo po e[ p[p[pe p[ p[p

Deep Learning Success

• Deep Learning has grown in success and popularity, largely due to:
• more powerful computers
• Faster CPUs and GPUs
• larger datasets
• better techniques to train deep networks
• Libraries
• Transferring models between tasks
• Regularization and optimization methods

Linguistic Tasks

Easy

• Spell Checking
• Keyword Search
• Finding Synonyms

Medium

• Parsing information from websites, documents, etc.

Hard

• Machine Translation
• Semantic Analysis (What is the meaning of query statement?)
• Coreference (e.g. What does "he" or "it" refer to given a document?)
• Question Answering.

Linguistic Applications

• So far self referential: from language to language
• Classification
• Extraction
• Summarization
• Translation
• More challenging:
• Derive conclusion
• Perform actions
• Challenge:
• Find a universal learning mechanism capable of learning from observations and interactions, adequate to make predictions
• Yann LeCunn. How could machines learn as efficiently as humans?

NLP is Difficult

NLP is hard

• Natural language is:
• highly ambiguous at all levels
• complex and subtle use of context to convey meaning
• fuzzy?, probabilistic
• involves reasoning about the world
• a key part of people interacting with other people (a social system):
• persuading, insulting and amusing them
• But NLP can also be surprisingly easy sometimes:
• rough text features can often do half the job
• Information Retrieval and similar superficial techniques have been in widespread use

Natural Language Understanding is difficult

• The hidden structure of language is highly ambiguous
• Structures for: Fed raises interest rates 0.5% in effort to control inflation

Slide by C. Manning

Where are the ambiguities?

Slide by C. Manning

Newspaper Headlines

• Minister Accused Of Having 8 Wives In Jail
• Pope’s baby steps on gays
• Juvenile Court to Try Shooting Defendant
• Teacher Strikes Idle Kids
• China to Orbit Human on Oct. 15
• Local High School Dropouts Cut in Half
• Red Tape Holds Up New Bridges
• Clinton Wins on Budget, but More Lies Ahead
• Hospitals Are Sued by 7 Foot Doctors
• Police: Crack Found in Man's Buttocks

Coreference Resolution

U: Where is The Green Hornet playing in Mountain View?

S: The Green Hornet is playing at the Century 16 theater.

U: When is it playing there?

S: It’s playing at 2pm, 5pm, and 8pm.

U: I’d like 1 adult and 2 children for the first show.�How much would that cost?

​

• Knowledge sources:
• Domain knowledge
• Discourse knowledge
• World knowledge

Hidden Structure of Language

• Going beneath the surface…
• Not just string processing
• Not just keyword matching in a search engine
• Search Google on “tennis racquet” and “tennis racquets” or “laptop” and “notebook” and the results are quite different …�though these days Google does lots of subtle stuff beyond keyword matching itself
• Not just converting a sound stream to a string of words
• Like Nuance/IBM/Dragon/Philips speech recognition
• To recover and manipulate at least some aspects of language structure and meaning

Deep Learning for NLP

Achieving the goals of NLP by using representation learning and deep learning to build end-to-end systems

Continuous Representation of Words

• Learning vectors in large dimensional space that represents all aspects of word meaning
• Word embeddings

Word Similarities

• Nearest word to ‘rana’:

​

1. farfalla
2. tartaruga
3. delfino
4. zampa
5. scimmia

​

Entails Morphology

• Words are made of morphemes:

Prefix stem suffix

un interest ed

​

​

• DL:
• Every morpheme is a vector
• A NN combines vectors
• Luong et al. 2013

Parsing for Sentence Structure

• Dependency Parser can determine grammatical structure of sentences
• Useful for interpretation and relation extraction
• See demo

Question Answering

• Traditional: A lot of feature engineering to capture world and other knowledge, e.g., regular expressions, Berant et al. (2014)

• Deep Learning: embeddings represent words, a Recurrent Neural Network extracts entities and their relations and output module extracts the answer.�Madotto, Attardi. 2017. Question Dependent Recurrent Entity Network for QA

Neural Reasoner

• Next frontier for AI
• Overcome the dichotomy between
• Perceptive Tasks (voice, images): Deep Learning
• Inference: rule-based systems, logic
• Single Neural Network Model for both tasks
• Learning like children, who do not know neither grammar nor differential equations for:
• Learning to speak interacting with adults
• Learn a simple model of the physical world starting from perceptual experiences