Joseph E. Gonzalez

Data 100�Feature Engineering

Slides by:

Joseph E. Gonzalez, Deb Nolan, John DeNero, & Josh Hug

jegonzal@berkeley.edu

deborah_nolan@berkeley.edu

denero@Berkeley.edu

josh@joshh.ug

​

​

​

?

Joseph E. Gonzalez

Recap of Linear Models

Joseph E. Gonzalez

Feature Engineering

• The process of transforming the raw features to into more informative features that can be �used in modeling tasks.
• Feature Engineering enables you to:
• capture domain knowledge (e.g., periodicity or relationships between features)
• express non-linear relationships using simple linear models
• encode non-numeric features to be used as inputs to models

Feature Functions

• Feature functions transform features into new features

Domain

Entirely Quantitative and Transformed Values

Different number�of features

Joseph E. Gonzalez

Feature Functions

• Feature functions transform features into new features

Domain

Entirely Quantitative and Transformed Values

(phi)ture function

Designing the feature functions is a big part of machine learning and data science.

​

Feature Functions

• capture domain knowledge
• substantial contribute to expressivity (and complexity)

Joseph E. Gonzalez

Feature Functions

• Feature functions transform features into new features

Domain

Entirely Quantitative and Transformed Values

(phi)ture function

Designing the feature functions is a big part of machine learning and data science.

​

Feature Functions

• capture domain knowledge
• substantial contribute to expressivity (and complexity)

Joseph E. Gonzalez

Feature Functions Examples

Joseph E. Gonzalez

The Constant Feature Function

• By adding the all 1s column to our original data we were already introducing a feature function:

​

​

​

​

• Sometimes this feature and it’s parameter are called:
• Constant feature, offset, intercept, bias

n

d

n

d

1

1

1

1

…

1 +

= p

Joseph E. Gonzalez

Modeling Non-linear Relationships

Feature Functions:

Note that feature functions don’t depend on parameters.

Joseph E. Gonzalez

Encoding Categorical and Text Features

What if x is a record with numbers, text, booleans, etc…

X

Y

Predict rating from review information

RatingsData(uid INTEGER, age FLOAT,

state STRING, hasBought BOOLEAN,

review STRING, rating FLOAT)

Schema:

As a Linear Model?

Can I use X and Y directly in a linear model

• No! Why?
• Text, Categorical data, Missing values…

RatingsData(uid INTEGER, age FLOAT,

state STRING, hasBought BOOLEAN,

review STRING, rating FLOAT)

X=

Y=

Basic Transformations

• Uninformative features: (e.g., UID)
• Is this informative (probably not?)
• Transformation: remove uninformative features (why?)
• They could influence the model.
• Quantitative Features (e.g., Age)
• Transformation: May apply non-linear transformations (e.g., log)
• Transformation: Normalize/standardize (more on this later …)
• Example: (x – mean)/stdev
• Categorical Features (e.g., State)
• How do we convert State into meaningful numbers?
• Alabama =1 , …, Utah = 50 ?
• Implies order/magnitude means something … we don’t want that ...
• Transformation: One-hot-Encode

One Hot Encoding (dummy encoding)

• Transform categorical feature into many binary features:

​

​

​

Corresponding feature functions

See notebook for example code.

Encoding Missing Values

• Missing values in Quantitative Data
• Try to impute (estimate) missing values… (tricky)
• Substitute the sample mean
• Add a binary field called “missing_col_name”. (why?)
• Sometimes missing data is signal!
• Missing values in Categorical Data
• Add an addition category called “missing_col_name”

Encoding categorical data

• Categorical Data 🡺 One-hot encoding:

​

​

​

​

• Text Data
• Bag-of-words & N-gram models

​

​

​

​

Bag-of-words Encoding

• Generalization of one-hot-encoding for a string of text:

​

​

​

• Encode text as a long vector of word counts (Issues?)
• Long = millions of columns 🡪typically high dimensional and very sparse
• Word order information is lost… (is this an issue?)
• New unseen words at prediction (test) time 🡪 drop them …
• A bag is another term for a multiset: an unordered collection which may contain multiple instances of each element.
• Stop words: words that do not contain significant information
• Examples: the, in, at, or, on, a, an, and …
• Typically removed

​

I made this art piece in graduate school

​

Do you see the stop word?

​

There used to be a dustbin and broom

… but the janitors got confused …

N-Gram Encoding

• Sometimes word order matters:

​

​

• How do we capture word order in a “vector” model?
• N-Gram: “Bag-of- sequences-of-words”

The book was not well written but I did enjoy it.

The book was well written but I did not enjoy it.

2-Gram Encoding

The book was well written ...

the book

well written

book was

was well

N-Gram Encoding

• Sometimes word order matters:

​

​

• How do we capture word order in a “vector” model?
• N-Gram: “Bag-of- sequences-of-words”
• Issues:
• Can be very sparse (many combinations occur only once)
• Many combinations will only occur at prediction time 🡪 drop ..
• Often use hashing approximation:
• Increment counter at hash(“not enjoy”) collisions are okay

​

The book was not well written but I did enjoy it.

The book was well written but I did not enjoy it.

Feature Transformations to Capture Domain Knowledge

• Feature functions capture domain knowledge by introducing additional information from other sources �and/or combining features

​

​

​

• Encoding non-linear patterns

​

Could do a database lookup

Diurnal patterns.