Feature Engineering & Feature Selection
Davis David�Zindi Africa
davisdavid179@gmail.com�� �
CONTENT:
1.Feature Engineering
Feature engineering refers to a process of selecting and transforming variables/features when creating a predictive model using machine learning.
Feature engineering has two goals:�
Data scientists spend 60% of their time on cleaning and organizing data.
57% of data scientists regard cleaning and organizing data as the least enjoyable part of their work
“At the end of the day, some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used.” �— Prof. Pedro Domingos from University of Washington
Read his paper here : A few useful things to know about machine learning
2. Missing Data
2. Missing Data
Common missing values
2. How to handle Missing Values
Variable deletion involves dropping variables(columns) with missing values on an case by case basis.
This method makes sense when lot of missing values in a variable and if the variable is of relatively less importance.
The only case that it may worth deleting a variable is when its missing values are more than 60% of the observations.
2. How to handle Missing Values
2. How to handle Missing Values
(b) Mean or Median Imputation
A common technique is to use the mean or median of the non-missing observations.
This strategy can be applied on a feature which has numeric data.
2. How to handle Missing Values
(c) Most Common Value
Replacing the missing values with the maximum occurred value in a column/feature is a good option for handling categorical columns/features.
3. Continuous Features�
�Example continuous features: age, salary , prices, heights
Common methods
3. Continuous Features�
For each value in a feature, Min-Max normalization subtracts the minimum value in the feature and then divides by the range. The range is the difference between the original maximum and original minimum.
It scale all values in a fixed range between 0 and 1.
3. Continuous Features�
(b) Standardization
The Standardization ensures that for each feature have the mean is 0 and the variance is 1, bringing all features to the same magnitude.
If the standard deviation of features is different, their range also would differ from each other.
x = observation, μ = mean , σ = standard deviation
4.Categorical Features
Categorical features represents types of data which may be divided into groups.
Example: genders, educational levels
Any non-numerical values need to be converted to integers or floats in order to be utilized in most machine learning libraries.
Common Methods
4.Categorical Features
(a) One-hot-encoding
By far the most common way to represent categorical variables is using the one-hot encoding or one-out-of-N encoding, also known as dummy variables.
The idea behind dummy variables is to replace a categorical variable with one or more new features that can have the values 0 and 1.
4.Categorical Features
4.Categorical Features
(b) Label Encoding
Label encoding is simply converting each categorical value in a column to a number.
NB: It is recommended to use label encoding to a Binary variable �
5.Feature Selection
5.Feature Selection
Top reasons to use feature selection are:�
5. Feature Selection
��“I prepared a model by selecting all the features and I got an accuracy of around 65% which is not pretty good for a predictive model and after doing some feature selection and feature engineering without doing any logical changes in my model code my accuracy jumped to 81% which is quite impressive”
- By Raheel Shaikh��
5.Feature Selection
5.Feature Selection
(b) Feature Importance
Feature importance gives you a score for each feature of your data, the higher the score more important or relevant is the feature towards your target feature.
Feature importance is an inbuilt class that comes with Tree Based Classifiers
Example:
5. Feature Selection
(c) Correlation Matrix with Heatmap