PYTHON FOR MACHINE LEARNING

Hands-on Walkthrough of Applying Machine Learning to Your Data Science Projects

NUS Statistics Society

Link For Workshop Materials

https://bit.ly/35AS7bk

Prerequisites and Learning Objectives

• We assume you have some basic Python programming skills and can understand Python code

Learning Objectives:

• Get a high level understanding of what data science and machine learning is
• Learn how to use machine learning for your data science projects and competitions
• Know where to look for further resources, since data science and machine learning is such a vast field

TABLE OF CONTENTS

INTRO TO DATA SCIENCE AND ML

NUMPY AND PANDAS QUICKSTART

EXPLORATORY DATA ANALYSIS

DATA PREPROCESSING

MODEL TRAINING AND EVALUATION

OVERVIEW OF SCIKIT-LEARN ALGORITHMS

01

03

02

04

05

06

INTRO TO DATA SCIENCE AND ML

01

What is Data Science?

Data science is a multidisciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights data

Why Learn Data Science?

Data Science Process

What is Machine Learning?

• Machine Learning refers to algorithms and models that perform a task without being explicitly programmed, relying on statistical patterns from data instead

​

​

model

Input

Output

What is Machine Learning?

• There are two major categories:
• supervised learning
• unsupervised learning
• Supervised learning requires a labelled dataset that contains ground truth values and can be used for classification and regression tasks
• Unsupervised learning is used when you don’t have labelled data and want to identify patterns present within a set of data points. Examples are dimensionality reduction, data visualisation and clustering

​

​

Supervised Learning: Classification vs Regression

NumPy and Pandas Quickstart

02

• NumPy is a linear algebra library in python which almost every machine learning Python package depends on to a certain extend
• Advantages of using NumPy over Python lists include: being more compact, faster read-write access, more convenient and faster computation
• A NumPy array is simply a grid that contains values of the same type. NumPy Arrays come in two forms; Vectors and Matrices. Vectors are strictly one-dimensional arrays, while Matrices are multidimensional.

Basic Numpy Operations

• Creating NumPy array from Python list

​

​

​

​

Basic Numpy Operations

• Reshaping NumPy arrays

​

​

​

• 1-d array slicing works similarly to python lists

​

​

​

​

​

​

​

​

​

​

​

​

​

Basic Numpy Operations

• Multidimensional array slicing and indexing

​

​

​

​

​

​

​

​

​

​

​

​

​

Basic Numpy Operations

• Broadcasting operations

​

​

​

​

​

​

​

​

​

​

​

​

​

• Pandas is an open source library built on top of NumPy
• Think of it like Excel for Python
• 2 main data structures in pandas are the Series and DataFrame Object
• A Series is a one-dimensional array which is very similar to a NumPy array. What differentiates Series from NumPy arrays is that series can have an access labels with which it can be indexed.
• A DataFrame is a two-dimensional data structure in which the data is aligned in a tabular form i.e. in rows and columns.

​

Pandas

• Creating pandas Series

​

​

​

​

• Accessing values in Series

Pandas

• Creating pandas Dataframe

Pandas

• Selecting Rows and Columns from DataFrame

​

Reading CSV

• The California Housing Dataset is available in every Colaboratory Notebook by default under sample data folder
• Besides pd.read_csv, there is also pd.read_excel for Excel Spreadsheets

EXPLORATORY DATA ANALYSIS

03

What is Exploratory Data Analysis?

Exploratory Data Analysis refers to the process of performing initial investigations on data with the help of summary statistics and graphical representations, so as to:

1. Discover patterns
2. Spot anomalies and mistakes
3. Test hypothesis
4. Check assumptions

​

​

Seaborn

• Seaborn is a high level data visualisation library based on matplotlib
• Often much easier to use than matplotlib and closely integrated with pandas data structures

Detecting Outliers

The box plot shows the three quartile values of the distribution along with extreme values. The “whiskers” extend to points that lie within 1.5 IQRs of the lower and upper quartile, and then observations that fall outside this range are displayed independently

Detecting Outliers

Plotting Distribution of a Feature

• Checking the distribution of data can help you find anomalies and give you better insight of how the data may be collected

​

• Here we can see that median_house_value of 50000 has a unusually high frequency, which may suggest that house values are capped at 50000 when the data was collected

Plotting Features Against Each Other

• Scatter plot is a great way to visualise the relationship between 2 variables.

​

• Here we set linewidth=0 to remove the white borders for each of the data points enabled by default

Plotting Features Against Each Other

• Here we set alpha=0.05 so that areas with more data points have higher opacity and the denser regions can be easily spotted

Plotting Features Against Each Other

• Setting hue=’median_house_value’ will allow us to colour the scatter plot based on the value of that corresponding column in the dataframe

​

• This tells us that denser regions and coastal areas correlates with higher housing prices

Plotting Correlation Heatmap

• df.corr() calculates the Pearson Correlation of the features against one another (by default)
• Give you a quick glance of which features are correlated linearly

DATA PREPROCESSING

04

Data Preprocessing

Now that we have a good understanding of the dataset, we need to:

1. Get the data into the correct format
2. Clean the data
3. Perform feature engineering

​

Data Cleaning

• In a competition setting, the data you receive may have already been cleaned by the organiser
• However, in the real world it is common to have duplicate entries, missing values or even erroneous entries such as wrong labels or typos
• Improving data quality can often yield better performance gains than trying to tweak your algorithm
• Many industry practitioners report spending the bulk of their time cleaning and organising data

Handling duplicate entries

• Data may come from different sources combined which could cause duplication of data
• A dataset with duplicate entries may not reflect the ‘true’ distribution of the data points and cause model performance to suffer
• Duplicate entries should be removed prior to shuffling and splitting the dataset because otherwise the same data point may end up in both the training and test set, also known as data leakage

Handling missing values

• A simple approach is to drop the entries with missing values or imputing values such as mean or mode
• This approach can be useful if only a few entries have missing data, however if many entries have missing values this may remove too much information
• May also be possible to predict the values of the missing data using other features in the same data entry

Handling missing values

One-Hot Encoding

• Most machine learning models needs input to be numeric
• Categorical Data contains label values rather than numeric values
• One way to do it is to encode banana=0, hotdog=1, hamburger=2 and ice cream=3, this is known as Label Encoding

One-Hot Encoding

• However this will assign a higher weightage to certain label values and can throw off the model
• One-Hot Encoding solves this by creating a column for each of the label values and assigning 0 or 1 based on whether the label value matches the column name

One-Hot Encoding

Feature Engineering

Feature engineering refers to the process of creating new input features from your existing ones to improve model performance.

​

Coming up with features is difficult, time-consuming, requires expert knowledge. “Applied machine learning” is basically feature engineering. ~ Andrew Ng

​

Feature Engineering: Indicator Variables

The first type of feature engineering involves using indicator variables to help your algorithm “focus” on important signals in the data.

• Indicator variable from thresholds: Let’s say you’re studying alcohol preferences by U.S. consumers and your dataset has an age feature. You can create an indicator variable for age >= 21 to distinguish subjects who were over the legal drinking age.
• Indicator variable from multiple features: You’re predicting real-estate prices and you have the features n_bedrooms and n_bathrooms. If houses with 2 beds and 2 baths command a premium as rental properties, you can create an indicator variable to flag them.
• Indicator variable for groups of classes: You’re analyzing website conversions and your dataset has the categorical feature traffic_source. You could create an indicator variable for paid_traffic by flagging observations with traffic source values of "Facebook Ads" or "Google Adwords".

​

​

​

Feature Engineering: Interaction Features

The next type of feature engineering involves highlighting interactions between two or more features. Specifically, look for opportunities to take the sum, difference, product, or quotient of multiple features.

• Difference between two features: You have the features house_built_date and house_purchase_date. You can take their difference to create the feature house_age_at_purchase.
• Product of two features: You’re running a pricing test, and you have the features price and units_sold. You can take their product to create the feature earnings.
• Quotient of two features: You have a dataset of marketing campaigns with the features n_clicks and n_impressions. You can divide clicks by impressions to create click_through_rate, allowing you to compare across campaigns of different volume.

​

​

​

​

Feature Engineering: Feature Representation

Your data won’t always come in the ideal format. You should consider if you’d gain information by representing the same feature in a different way.

• Date and time features: Let’s say you have the feature purchase_datetime. It might be more useful to extract purchase_day_of_week and purchase_hour_of_day.
• Numeric to categorical mappings: You have the feature years_in_school. You might create a new feature grade with classes such as "Elementary School", "Middle School", and "High School".
• Grouping sparse classes: You have a feature with many classes that have low sample counts. You can try grouping similar classes and then grouping the remaining ones into a single "Other" class.

​

​

​

Feature Scaling

• Most of the times, your dataset will contain features highly varying in magnitudes, units and range. If left alone, these algorithms only take in the magnitude of features neglecting the units.
• The features with larger magnitudes will contribute a higher weightage and may cause the algorithm to not perform well.
• Another reason why feature scaling is applied is that gradient descent converges much faster with feature scaling than without it.
• There are two common ways to get all attributes to have the same scale: min-max scaling and standardization.

​

Scikit-Learn

• Machine learning library built on top of NumPy, SciPy and matplotlib
• Includes tools for model selection, preprocessing and a wide variety of algorithms

Feature Scaling: Min-Max Scaling

• Min-max scaling (many people call this normalization) is the simplest: values are shifted and rescaled so that they end up ranging from 0 to 1.

​

​

​

​

Scikit-Learn provides a transformer called MinMaxScaler for this. It has a feature_range hyperparameter that lets you change the range if, for some reason, you don’t want 0– 1.

Feature Scaling: Standardization

• Standardization is different: first it subtracts the mean value and then it divides by the standard deviation so that the resulting distribution has unit variance and zero mean

​

​

• Unlike min-max scaling, standardization does not bound values to a specific range, which may be a problem for some algorithms (e.g., neural networks often expect an input value ranging from 0 to 1). However, standardization is much less affected by outliers. For example, suppose a feature has values ranging from 0 to 15 with except one outlier of 100. Min-max scaling would then crush all the other values from 0– 15 down to 0– 0.15, whereas standardization would not be much affected.
• Scikit-Learn provides a transformer called StandardScaler for standardization.

​

MODEL TRAINING AND EVALUATION

05

Model Training

At this point, we have:

1. Our post-processed data, consisting of:
1. Input features (e.g. Sex, Age)
2. Output feature (Whether they survived the titanic)

Linear Regression

where X are input features, and β are feature weights

Minimize squared distance between true Y and predicted Y

(Linear Least Squares)

​

Model Training

Fitting a linear regression model, we obtain the feature weights β. We can then use these weights to infer on new input:

​

Example:

Final grade = 0.3 x attendance + 1.5 x hours spent studying

Remembering The Goal

• Train a model that generalizes to unseen data
• How do we know if the model generalizes well?
• What do we do if our model isn’t working?

model

Input

Output

Underfitting and Overfitting

Underfitting

• Model is too simple to capture the nuances and complexity of the dataset
• E.g. Fitting a linear model to a quadratic function
• Symptoms of underfitting: Low training and test accuracy

​

• Fix:
• Use a more complex model
• Train for a longer period of time
• Reduce regularization

Overfitting

• Model learns the patterns in the noise in the training dataset
• Akin to memorizing the training examples
• Symptoms of overfitting: High training accuracy, low validation/test accuracy

​

• Fix:
• Reduce training time (keep the model with the lowest validation error)
• Clean and collect the data
• Use a simpler model, or regularize more heavily

Underfitting and Overfitting

What do we do when the model doesn’t work?

• 2 possible problems:
• The data is bad (hard to fix)
• The model is bad (easier to fix)
• Diagnosis comes with experience:
• EDA
• Hyperparameter tuning
• Trying out different models, understanding the properties of different models

​

Problems with Data

• More difficult in the sense that time consuming manual work may be needed to fix

​

​

Choosing an Evaluation Metric

• This can be defined by the host in a data science competition setting
• Depends on the business objective you are trying to optimise for, as well as the distribution of the dataset you have
• Common evaluation metrics: accuracy, precision, recall, F1 score, MSE, MAE, AUC
• Other problem sets such as computer vision or natural language processing tasks may have their own niche set of evaluation metrics as well

​

​

Accuracy, Precision and Recall

• Accuracy = (TP + TN) / (TP + FP + TN + FN)
• Accuracy is most commonly used metric, but may not be the best metric when the positive class largely outnumbers the negative class

​

• Precision = TP / (TP + FP)

= TP / Total Predicted Positive

• Recall = TP / (TP + FN)

= TP / Total Actual Positive

Data Splits

• In order to detect if the model is overfitting, we need to split the dataset into a training and test set.
• The data in the test set should only be used to evaluate the performance of the model. It should not be used to train the model nor lead your decision in tuning the parameters of the model as that will lead to an over-optimistic evaluation score

Validation Set

• If we tune the parameters of our model using the test set, the model will be tuned to perform well on the distribution of the test set and essentially we are overfitting our model to the test set
• One solution is to split the dataset into 3 sets - A training set for training the model, a validation set (also known as development set) for choosing the best algorithms and parameters and a final test set solely for reporting the performance of the model.

K-Fold Cross Validation

• One disadvantage of the previous method is that the size of the training data available to the model is reduced
• Using k-fold cross validation, you can split the dataset into k sets, use one of the sets to evaluate the model and the remaining k - 1 sets to train the model, repeating k times and averaging the result

Training a Linear Regression Model

Training a Linear Regression Model

Training a Random Forest Model

Hyper-parameter Tuning

• The n_estimators used in the RandomForestRegressor is an example of a hyper-parameter
• There are usually many different parameters and certain combinations of them may produce a better result.
• Our job is to find the best set of parameters in the parameter space under the time constraints that we have, usually not possible to search over entire parameter space
• One option would be to fiddle with the hyperparameters manually, until you find a great combination of hyperparameter values. This would be very tedious work, and you may not have time to explore many combinations.

​

Grid and Randomized Search

• A common approach is to use Grid Search and Randomized Search using the GridSearchCV and RandomizedSearchCV classes in sklearn.model_selection
• However, the complexity of Grid Search grows exponentially with addition of new hyper-parameters since it iterates through all the possible combinations of the parameter values
• Randomized Search will pick random values within the specified parameter range to test, however it does not make use of information gained from previous runs of the algorithm testing

OVERVIEW OF MACHINE LEARNING ALGORITHMS

06

Commonly used Machine Learning Algorithms

Linear Regression

• The core idea is to obtain a line that best fits the data.
• The best fit line is the one for which total prediction error (all data points) are as small as possible.
• Error is the distance between the point to the regression line
• Used when prediction is a continuous value e.g. Housing prices

Logistic Regression

• Used when the target variable is categorical
• Predicts probability of occurrence of an event using the sigmoid function
• Hypothesis => Z = WX + B
• hΘ(x) = sigmoid (Z)

Decision Tree

• Starting at the root, splits the data using the “most informative” feature
• Model output is human-understandable
• Tunable hyperparameters (non-exhaustive):
• Depth of tree
• Minimum number of samples per leaf

Random Forest

• Random Forest is a term for an ensemble of decision trees
• Each tree gives a classification and we say the tree “votes” for that class
• The forest chooses the classification having the most votes

K-Nearest Neighbours

• Stores all available cases and classifies new cases by a majority vote of its k neighbors. The case being assigned to the class is most common amongst its K nearest neighbors measured by a distance function.

Choosing the right model

• Complex model != Better model
• Understand the model assumptions
• If they’re cheap to try, try them all

In a competition setting:

• Establish baselines: linear model etc.
• Try robust models: XGBoost, Random Forest
• With sufficient, and unstructured data, maybe even deep learning models

​

Summary

Today we learnt how to:

1. Data processing and EDA with numpy, pandas
2. Learnt basic concepts in machine learning, and common problems
1. Data splitting
2. Debugging models that don’t work
3. Basic models (Linear regression, Decision trees etc.)
4. Machine learning with scikitlearn
5. How to approach a machine learning problem

What’s Next?

• Practice! Kaggle/toy problems
• Deep learning with Tensorflow/Pytorch
• If you’re serious about going into the field beyond a hobby:
• Linear Algebra
• Multivariable Calculus
• Probability and Statistics

FURTHER RESOURCES

07

Further Resources

• Andrew Ng’s Machine Learning Course (https://www.coursera.org/learn/machine-learning/)
• Aurélie Géron ‘s Hands-On Machine Learning with Scikit-Learn, Keras and Tensorflow Book
• François Chollet’s Deep Learning with Python Book
• Documentation of the various libraries
• Kaggle Competitions and Write-Ups by Winners
• The hundred page book of machine learning

Projects to Try

• https://www.kaggle.com/competitions
• https://elitedatascience.com/datasets
• https://elitedatascience.com/machine-learning-projects-for-beginners
• https://toolbox.google.com/datasetsearch
• https://archive.ics.uci.edu/ml/datasets.php
• https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research

Post Workshop Survey

https://bit.ly/2OV69i3

Thank you and we hope you have fun in your machine learning journey!

CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik.

Please keep this slide for attribution.

​

​

23 October 2019 (Wed), 7:00 pm – 9:00 pm

NUS Science LT 32

BestTop X NUS Statistics Society Career Talk Event:

How To Prepare Your Career in Data Science Industry

An information and networking session for you and your friends to interact with three honoured speakers, and learn more about the data science industry.

​

​

https://orgsync.com/167409/forms/370841

Free refreshments are provided!

https://orgsync.com/

167409/forms/370841

​

Our Social Media Links:

FaceBook:

https://www.facebook.com/nus.statistics.soc/

LinkedIn: https://www.linkedin.com/company/nusstatssoc/

Instagram:

https://www.instagram.com/nusstatssoc/

​

Extra Slides

Installing Python

• Can either install using Anaconda (https://www.anaconda.com/distribution/), from https://www.python.org/, or using your system’s package manager
• We recommend sticking to one installation of python to avoid path conflicts and using virtual environments to manage dependencies versioning
• If you haven’t installed python already, you can still follow this workshop using the python environment on Colaboratory

​

​

Machine Learning Tools Covered

• Python
• Jupyter Notebook / Colaboratory
• Pandas
• Pandas Profiling
• Numpy
• Scikit-Learn
• Seaborn

​

​

> pip install jupyter pandas numpy scikit-learn seaborn

Basic Numpy Operations

• Generating NumPy arrays

​

​

​

​

​

​

​

​

​

Basic Numpy Operations

• Generating NumPy arrays

​

​

​

​

​

​

​

​

​

Virtual Environments

• By default, every project will use the same version of Python and packages
• Different projects may require different versions of packages or Python
• A virtual environment is a isolated workspace with its own python version and package versions
• Help reproducibility by fixing package versions for a particular project so it is less likely to encounter errors due to conflicting package versions
• Can use tools such as virtualenv or conda to create virtual environments
• Further Reading: https://realpython.com/python-virtual-environments-a-primer/ and https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html#managing-envs

​

​

Field of Data Science

https://pbs.twimg.com/media/Dms6CRWUcAIrlDr?format=jpg&name=large

How do Machine Learning Models Work?

• You can think of a machine learning model as a mathematical function that maps given inputs to an output (x to y):

​

• Other important parts of a ML model are hyper-parameters, parameters, cost function and the optimiser
• Hyper-parameters are the settings of the model you tune before training, often to control the size, complexity and regularization of the model.

​

​

How do Machine Learning Models Work?

• The cost function measures the extent of the model’s error, given the model’s prediction and the actual value

​

​

• The model’s parameters (also known as weights) are tuned by the model optimiser to reduce the error given by the cost function

​

​

How do Machine Learning Models Work?

• One of the most common optimisers used is called gradient descent
• Finds the optimal set of parameters that gives the minimum value of the cost function

​

​

Bayesian Optimization

• A better approach is to use Bayesian Optimization where the results of previous runs are used to make informed guesses about the best parameter values for the next run

Bayesian Optimization

​

Linear Regression

Algorithm

​

​

​

​

​

​

Sklearn Example

1. from sklearn.linear_model import LinearRegression
2. model = LinearRegression()
3. model.fit(X_train, y_train)
4. y_predict = model.predict(X_test)

Description

• Simple linear model
• Not expressive and cannot model non-linear relationships
• Continuous data
• Loss function: Mean squared error (usually)
• High bias, low variance

Support Vector Machines

Diagram

​

​

​

​

​

Description

• Linear model
• Maximises margin between closest support vectors to form a hyperplane
• Extends by using kernel tricks, transforming datasets into rich feature space, so that complex problems can be dealt in a ‘linear’ fashion in the lifted hyperspace
• Robust against overfitting

Decision Tree

Algorithm

1. Computer the entropy for dataset
2. For every attribute/ feature:
1. Calculate entropy for all categorical values
2. Take average information entropy for the current attribute
3. Calculate gain for the current attribute
3. Pick the highest gain attribute
4. Repeat until we get the tree we desire

Sklearn Example

1. from sklearn import tree
2. model = tree.DecisionTreeClassifier()
3. model.fit(X_train, y_train)
4. y_predict = model,predict(X_test)

​

Description

• Example: ID3 (Iterative Dichotomiser 3)
• Uses entropy function and information gain as metrics
• Used for explainability
• Robust to outliers
• Prone to overfitting

K-means Clustering

Algorithm

​

​

​

​

​

Applications

• Clustering of T-shirt sizes (XS, S, M, L, XL)
• ​

​

Description

• Can be used for unsupervised learning (do not require labels)
• No. of clusters specified a priori
• Useful for spherical clusters

Multi-layer Perceptron

Diagram

​

​

​

​

​

from sklearn.neural_network import MLPClassifier

Details

• Very expressive: High variance, low bias
• Requires a lot of data
• Basis for state-of-the-art deep learning models
• E.g. LSTM, CNN

​

Algorithm