4 of 34

Today’s Instructors

Kyle Han

2nd Year

Data Science

Interests: NLP, AI/ML, Education

Sameer Rahman

2nd Year

Data Science

Interests: AI/ML, NLP, Sports Analytics, Fintech

5 of 34

Ice Breaker

You can only keep 3 apps on your phone…what are you keeping?

6 of 34

Machine Learning Pipeline

You’ve already cleaned your data BUT
Remember: Garbage in means Garbage out
Feature engineering and selection helps our model’s predictive power

7 of 34

What is a Feature?

8 of 34

Real-world Scenario

You’re tasked with creating a model that predicts NBA player salaries

Each of these columns is a feature: measurable property of your data

9 of 34

If you had to make a housing prediction model, what are some features would you want for your model?

10 of 34

Feature Engineering

11 of 34

Feature Engineering

Using domain knowledge to create new, meaningful features from raw data

Gives your analysis a different perspective – uncover patterns you’d otherwise miss

Your chance to show creativity + domain expertise

12 of 34

Turn to your neighbor and discuss some domains you think you have a lot of expertise in

13 of 34

Why is this Important?

Even simple models are better with strong features

Goal: Create features with high correlations to target variable

14 of 34

Data Cleaning Review

First make sure that your data is clean:

*Missing values: df.fillna() or df.dropna()
Duplicates: df.drop_duplicates()
Wrong type: df[‘col’].astype(type)
String casing: df[‘col’].str.lower() or .str.upper()
Fix inconsistencies: df[‘col’].replace({old : new})
Dates: pd.to_datetime(df[‘col’])

15 of 34

Handling Missing Data as Features

Missing data can be informative

We can add a flag feature instead of just filling or dropping

16 of 34

Handling Categorical Data

Machine learning models need numbers, not text

2 Main Ways of Approaching categorical data:

One-Hot Encoding

Label Encoding

17 of 34

One-Hot Encoding

Creates a new binary column for each category

Best for nominal data with no particular order

Each row gets a 1 in the column that matches the category

pd.get_dummies(df[col])

18 of 34

Label Encoding

Assigns a unique integer to each category

Best for ordinal data with a clear order

from sklearn.preprocessing import LabelEncoder

Encoder = LabelEncoder()

df[‘Encoded Col’] = encoder.fit_transform(df[‘Col’])

19 of 34

Which of the following columns do you think should be one-hot encoded or label encoded?

20 of 34

Creating New Features

Oftentimes, it’s better to create new features using domain knowledge to capture otherwise unseen relationships

Housing Data:

Total rooms: Bedrooms + Bathrooms
House Age: Current Year - Year Built
Lot Utilization: Living Area / Lot Size

Do some more EDA and see if you can find more linear relationships with your target variable

21 of 34

What could be a problem with this housing data?

22 of 34

The values in size and num_bedrooms are scaled totally different!

The model would incorrectly believe that the size feature is 1000x more important than the bedrooms feature.

23 of 34

Technique: Feature Scaling

Most of the time, we don’t want one feature to overpower all other features. Feature scaling can help normalize features to be on a similar scale.

Applications

Gradient Descent
K Nearest Neighbors

Normalization: Scales data to a fixed range, usually 0 to 1.

Standardization: Scales data to have a mean of 0 and a standard deviation of 1 using z-scores.

24 of 34

Feature scaling good 👍

25 of 34

Feature Selection

26 of 34

Feature Selection

Feature Selection = process of selecting a subset of relevant features from your data

Pandas

data[[‘column1’, ‘column2’]]
data.iloc (uses row/col indexes)
data.loc (uses row/col names)

Relational Algebra

Database theory (data101), analogous to SQL queries
π operator

27 of 34

Feature Selection Importance

Reduces Overfitting: Fewer redundant features mean the model is less likely to learn from noise.

Improves Accuracy: Removing irrelevant features can improve the model's generalization performance.

Reduces Training Time: Fewer features mean faster model training and prediction.

Improves Interpretability: Simpler models with fewer features are easier to understand.

28 of 34

Feature Selection Methods

Statistical Filtering

Uses statistical tests to score each feature.
Independent of the model.
Fast and computationally cheap.
Ex. Chi-Squared test, Correlation Coefficient

Wrapper Methods

Uses a predictive model to score feature subsets.
Model-dependent.
Computationally expensive but can find the best subset.
Ex. Recursive Feature Elimination (RFE)

Embedded Methods

Feature selection is performed during model training.
A good compromise between filter and wrapper methods.
Ex. LASSO and Ridge Regression

29 of 34

Key Takeaways

Feature Engineering is about creating the best possible features from your raw data.

Feature Selection is about choosing the most relevant features to build simpler, faster, and more robust models.

Both are essential for building high-performing machine learning systems.

And remember GIGO: Garbage in, Garbage Out!

30 of 34

Worksheet!

https://docs.google.com/document/d/1ycemy62apkvM9eJvGY4SXvh2GumVm3Imzd7F09B4T3Y/edit?usp=sharing

31 of 34

Resources!

Kaggle tutorial

Guide to feature engineering concepts like encoding, PCA, etc.
Link

Your TAs and tutors obviously!

1 of 34

2 of 34

3 of 34