Do English and math teachers have anything in common?

2020

Jerry Tuttle, FCAS, Rocky Mt College Art & Design

What is data science?

• "Work that takes more programming skills than most statisticians have, and more statistics skills than a programmer has."

2020

Data science

• Data science tries to find meaning in large amounts of data.
• Math people like data.
• Text analysis is a branch of data science.
• Math people like to count. English teachers like to use words.
• So let’s count words!
• We know data can be dirty.
• Text data is dirtier than numerical data.
• First, let’s clean dirty text data.

​

​

​

​

​

2020

Cleaning dirty data

• Delete punctuation, remove capitals
• Delete funny characters: œ, â, €, URLs
• Delete escape & Unicode characters: \n, \u0009, etc.
• Stemming: lov = {love, lovable, loving, lovely, love-affair}
• Lemmatizing: love = {lov}
• Delete stop words: a, an, the, etc.

2020

https://www.insightsquared.com/wp-content/uploads/2016/04/dirtydata-e1461588583204.png

How do writers differ quantitatively ?

• # Characters per word (used by logician DeMorgan, 1851), # words per sentence
• Percent unique words
• Use of frequent words
• Use of sensory adjectives
• Use of sentiment words
• Use of positive or negative words
• Verb/adjective ratio
• Complexity (grade level readability)

​

2020

Sensory adjectives

2020

http://www.pics4learning.com/results.php?query=senses&search=qsearch&x=15&y=10

Sentiment words

2020

http://aylien.com/text-api/sentiment-analysis=10

Let’s compare 2 great works of literature

# 1. Hamlet

• Neither a borrower nor a lender be
• To thine own self be true
• Something is rotten in Denmark
• Brevity is the soul of wit
• There is nothing either good or bad but thinking makes it so
• To die, to sleep, perchance to dream
• The lady protests too much methinks
• Alas poor Yorick! I knew him

​

​

2020

#2. 1 + 1 = 0 excerpt:

The miniskirted waitress brought two more beers to the table. She leaned over as she placed each beer bottle on the table, invit- ing the two male patrons a teasing look down the top of her blouse. She flipped her blonde hair, batted her long eyelashes, and flashed a big smile through her shiny red lipstick. Each of these gestures was designed to elicit a greater than average tip from the two considerably drunk young men. She did not understand that the two men were actuaries who could easily estimate a 15% tip to within a couple of pennies, no matter how drunk or distracted they were.

​

2020

:#2. 1 + 1 = 0 excerpt:

The miniskirted waitress brought two more beers to the table. She leaned over as she placed each beer bottle on the table, invit- ing the two male patrons a teasing look down the top of her blouse. She flipped her blonde hair, batted her long eyelashes, and flashed a big smile through her shiny red lipstick. Each of these gestures was designed to elicit a greater than average tip from the two considerably drunk young men. She did not understand that the two men were actuaries who could easily estimate a 15% tip to within a couple of pennies, no matter how drunk or distracted they were.

​

2020

Post stop-word freq

2020

Unique words

2020

Some words from Hamlet not in 1+1=0:

• encompassment
• inventorially
• reconcilement
• recognizances
• schoolfellows

​

​

​

​

Compare word lengths

2020

Compare sensory adj.

2020

Compare sentiments

2020

Hamlet Pos & Negs

2020

Hamlet (pos – neg) �index every 80 lines

2020

Common authorship?

2020

JT1

​

JT9

I used text analysis on 10 actuarial stories. I used k means clustering to test common authorship. Algorithm made 3 clusters and did not pair my 2 stories, but it came close.

Uses of text analysis

• Did Shakespeare really write his works
• Did Hamilton write 51 Federalist papers
• Which Beatles songs did John write
• Which tweets did Trump’s staff write
• Compare sentiments of characters
• Compare books, presidents, judges
• Opinions on consumer products, politicians, course evaluations
• Predict spam, crime, fraud, terrorism

​

2020

Final thoughts

​

​

2020

• Cleaning the data takes much time
• Hamlet has old English, stage directions
• Use R or Python, not Excel
• Good reference is Silge, J. & Robinson, D. Text mining with R. O’Reilly
• Full text of 1+1=0: bit.ly/jt_oneplusone

​

​

https://blog.rstudio.com/2019/12/17/r-vs-python-what-s-the-best-for-language-for-data-science/

​