END-TO-END MACHINE LEARNING PROJECT

Introduction

• In this chapter, you will go through an example project end to end, pretending to be a recently hired data scientist in a real estate company.1 Here are the main steps you will go through:

1. Look at the big picture.

2. Get the data.

3. Discover and visualize the data to gain insights.

4. Prepare the data for Machine Learning algorithms.

5. Select a model and train it.

6. Fine-tune your model.

7. Present your solution.

8. Launch, monitor, and maintain your system.

Working with Real Data

• When you are learning about Machine Learning it is best to actually experiment with real-world data, not just artificial datasets. Fortunately, there are thousands of open datasets to choose from, ranging across all sorts of domains.
• Here are a few places you can look to get data:

• Popular open data repositories:

—UC Irvine Machine Learning Repository

—Kaggle datasets

—Amazon’s AWS datasets

• Meta portals (they list open data repositories):

—http://dataportals.org/

—http://opendatamonitor.eu/

—http://quandl.com/

​

Cont…

• Other pages listing many popular open data repositories:

—Wikipedia’s list of Machine Learning datasets

—Quora.com question

—Datasets subreddit

• In this chapter we chose the California Housing Prices dataset from the StatLib repository2 (see Figure 2-1). This dataset was based on data from the 1990 California census.
• It is not exactly recent (you could still afford a nice house in the Bay Area at the time), but it has many qualities for learning, so we will pretend it is recent data.
• We also added a categorical attribute and removed a few features for teaching purposes.

Cont…

Look at the Big Picture

• Welcome to Machine Learning Housing Corporation! The first task you are asked to perform is to build a model of housing prices in California using the California census data.
• This data has metrics such as the population, median income, median housing price, and so on for each block group in California.
• Block groups are the smallest geographical unit for which the US Census Bureau publishes sample data (a block group typically has a population of 600 to 3,000 people).
• We will just call them “districts” for short.
• Your model should learn from this data and be able to predict the median housing price in any district, given all the other metrics.

Cont…

• Since you are a well-organized data scientist, the first thing you do is to pull out your Machine Learning project checklist.
• You can start with the one in ???; it should work reasonably well for most Machine Learning projects but make sure to adapt it to your needs.
• In this chapter we will go through many checklist items, but we will also skip a few, either because they are self-explanatory or because they will be discussed in later chapters.

Frame the Problem

• The first question to ask your boss is what exactly is the business objective; building a model is probably not the end goal. How does the company expect to use and benefit from this model?
• This is important because it will determine how you frame the problem, what algorithms you will select, what performance measure you will use to evaluate your model, and how much effort you should spend tweaking it.
• Your boss answers that your model’s output (a prediction of a district’s median housing price) will be fed to another Machine Learning system (see Figure 2-2), along with many other signals.
• This downstream system will determine whether it is worth investing in a given area or not. Getting this right is critical, as it directly affects revenue.

Cont…

Pipelines

• A sequence of data processing components is called a data pipeline. Pipelines are very common in Machine Learning systems, since there is a lot of data to manipulate and many data transformations to apply.
• Components typically run asynchronously. Each component pulls in a large amount of data, processes it, and spits out the result in another data store, and then some time later the next component in the pipeline pulls this data and spits out its own output, and so on.
• Each component is fairly self-contained: the interface between components is simply the data store. This makes the system quite simple to grasp (with the help of a data flow graph), and different teams can focus on different components.
• Moreover, if a component breaks down, the downstream components can often continue to run normally (at least for a while) by just using the last output from the broken component.

Cont…

• This makes the architecture quite robust.
• On the other hand, a broken component can go unnoticed for some time if proper monitoring is not implemented.
• The data gets stale and the overall system’s performance drops.

​

• If the data was huge, you could either split your batch learning work across multiple servers (using the MapReduce technique), or you could use an online learning technique instead.

Select a Performance Measure

• A typical performance measure for regression problems is the Root Mean Square Error (RMSE). It gives an idea of how much error the system typically makes in its predictions, with a higher weight for large errors. Equation 2-1 shows the mathematical formula to compute the RMSE.

​

• This equation introduces several very common Machine Learning notations that we will use throughout this book:
• m is the number of instances in the dataset you are measuring the RMSE on.
• For example, if you are evaluating the RMSE on a validation set of 2,000 districts, then m = 2,000.
• x(i) is a vector of all the feature values (excluding the label) of the ith instance in the dataset, and y(i) is its label (the desired output value for that instance).
• For example, if the first district in the dataset is located at longitude –118.29°, latitude 33.91°, and it has 1,416 inhabitants with a median income of $38,372, and the median house value is $156,400 (ignoring the other features for now), then:

Cont…

• h is your system’s prediction function, also called a hypothesis. When your system is given an instance’s feature vector x(i), it outputs a predicted value ŷ(i) = h(x(i)) for that instance (ŷ is pronounced “y-hat”).
• For example, if your system predicts that the median housing price in the first district is $158,400, then ŷ(1) = h(x(1)) = 158,400.
• The prediction error for this district is ŷ(1) – y(1) = 2,000.
• RMSE(X,h) is the cost function measured on the set of examples using your hypothesis h.
• We use lowercase italic font for scalar values (such as m or y(i)) and function names (such as h), lowercase bold font for vectors (such as x(i)), and uppercase bold font for matrices (such as X).

Cont…

• Even though the RMSE is generally the preferred performance measure for regression tasks, in some contexts you may prefer to use another function.
• For example, suppose that there are many outlier districts. In that case, you may consider using the Mean Absolute Error (also called the Average Absolute Deviation; see Equation 2-2):

​

​

​

​

• Both the RMSE and the MAE are ways to measure the distance between two vectors: the vector of predictions and the vector of target values. Various distance measures, or norms, are possible:

​

Cont…

Check the Assumptions

• Lastly, it is good practice to list and verify the assumptions that were made so far (by you or others); this can catch serious issues early on.
• For example, the district prices that your system outputs are going to be fed into a downstream Machine Learning system, and we assume that these prices are going to be used as such.
• But what if the downstream system actually converts the prices into categories (e.g., “cheap,” “medium,” or “expensive”) and then uses those categories instead of the prices themselves?
• In this case, getting the price perfectly right is not important at all; your system just needs to get the category right.
• If that’s so, then the problem should have been framed as a classification task, not a regression task. You don’t want to find this out after working on a regression system for months.

Get the Data

• It’s time to get your hands dirty. Don’t hesitate to pick up your laptop and walk through the following code examples in a Jupyter notebook.
• The full Jupyter notebook is available at

https://github.com/ageron/handson-ml2.

Download the Data

• In typical environments your data would be available in a relational database (or some other common datastore) and spread across multiple tables/documents/files.
• To access it, you would first need to get your credentials and access authorizations,10 and familiarize yourself with the data schema.
• In this project, however, things are much simpler: you will just download a single compressed file, housing.tgz, which contains a comma-separated value (CSV) file called housing.csv with all the data.
• You could use your web browser to download it, and run tar xzf housing.tgz to decompress the file and extract the CSV file, but it is preferable to create a small function to do that.

​

Cont…

• It is useful in particular if data changes regularly, as it allows you to write a small script that you can run whenever you need to fetch the latest data (or you can set up a scheduled job to do that automatically at regular intervals).
• Automating the process of fetching the data is also useful if you need to install the dataset on multiple machines.

Cont…

Cont…

• Now when you call fetch_housing_data(), it creates a datasets/housing directory in your workspace, downloads the housing.tgz file, and extracts the housing.csv from it in this directory.
• Now let’s load the data using Pandas. Once again you should write a small function to load the data:

Take a Quick Look at the Data Structure

Cont…

• Each row represents one district. There are 10 attributes (you can see the first 6 in the screenshot): longitude, latitude, housing_median_age, total_rooms, total_bed rooms, population, households, median_income, median_house_value, and ocean_proximity.
• The info() method is useful to get a quick description of the data, in particular the total number of rows, and each attribute’s type and number of non-null values (see Figure 2-6).

Cont…

Cont…

• There are 20,640 instances in the dataset, which means that it is fairly small by Machine Learning standards, but it’s perfect to get started. Notice that the total_bed rooms attribute has only 20,433 non-null values, meaning that 207 districts are missing this feature.
• We will need to take care of this later. All attributes are numerical, except the ocean_proximity field. Its type is object, so it could hold any kind of Python object, but since you loaded this data from a CSV file you know that it must be a text attribute.
• When you looked at the top five rows, you probably noticed that the values in the ocean_proximity column were repetitive, which means that it is probably a categorical attribute. You can find out what categories exist and how many districts belong to each category by using the value_counts() method:

Cont…

Cont…

Cont…

• The count, mean, min, and max rows are self-explanatory. Note that the null values are ignored (so, for example, count of total_bedrooms is 20,433, not 20,640).
• The std row shows the standard deviation, which measures how dispersed the values are the 25%, 50%, and 75% rows show the corresponding percentiles: a percentile indicates the value below which a given percentage of observations in a group of observations falls.
• For example, 25% of the districts have a housing_median_age lower than 18, while 50% are lower than 29 and 75% are lower than 37. These are often called the 25th percentile (or 1st quartile), the median, and the 75th percentile (or 3rd quartile).

Cont….

• Another quick way to get a feel of the type of data you are dealing with is to plot a histogram for each numerical attribute.
• A histogram shows the number of instances (on the vertical axis) that have a given value range (on the horizontal axis).
• You can either plot this one attribute at a time, or you can call the hist() method on the whole dataset, and it will plot a histogram for each numerical attribute (see Figure 2-8). For example, you can see that slightly over 800 districts have a median_house_value equal to about $100,000.

​

%matplotlib inline # only in a Jupyter notebook

import matplotlib.pyplot as plt

housing.hist(bins=50, figsize=(20,15))

plt.show()

Create a Test Set

• Creating a test set is theoretically quite simple: just pick some instances randomly, typically 20% of the dataset (or less if your dataset is very large), and set them aside:
• import numpy as np
• def split_train_test(data, test_ratio):
• shuffled_indices = np.random.permutation(len(data))
• test_set_size = int(len(data) * test_ratio)
• test_indices = shuffled_indices[:test_set_size]
• train_indices = shuffled_indices[test_set_size:]
• return data.iloc[train_indices], data.iloc[test_indices]
• You can then use this function like this:
• >>> train_set, test_set = split_train_test(housing, 0.2)
• >>> len(train_set)
• 16512
• >>> len(test_set)
• 4128

Cont…

• Suppose you chatted with experts who told you that the median income is a very important attribute to predict median housing prices.
• You may want to ensure that the test set is representative of the various categories of incomes in the whole dataset.
• Since the median income is a continuous numerical attribute, you first need to create an income category attribute.
• Let’s look at the median income histogram more closely (back in Figure 2-8): most median income values are clustered around 1.5 to 6 (i.e., $15,000–$60,000), but some median incomes go far beyond 6.
• It is important to have a sufficient number of instances in your dataset for each stratum, or else the estimate of the stratum’s importance may be biased.
• This means that you should not have too many strata, and each stratum should be large enough.

Cont…

• The following code uses the pd.cut() function to create an income category attribute with 5 categories (labeled from 1 to 5): category 1 ranges from 0 to 1.5 (i.e., less than $15,000), category 2 from 1.5 to 3, and so on:

housing["income_cat"] = pd.cut(housing["median_income"],

bins=[0., 1.5, 3.0, 4.5, 6., np.inf],

labels=[1, 2, 3, 4, 5])

• These income categories are represented in Figure 2-9:

housing["income_cat"].hist()

Cont….

Cont…

• Now you are ready to do stratified sampling based on the income category. For this you can use Scikit-Learn’s StratifiedShuffleSplit class:

from sklearn.model_selection import StratifiedShuffleSplit

split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)

for train_index, test_index in split.split(housing, housing["income_cat"]):

strat_train_set = housing.loc[train_index]

strat_test_set = housing.loc[test_index]

• Let’s see if this worked as expected. You can start by looking at the income category proportions in the test set:

>>> strat_test_set["income_cat"].value_counts() / len(strat_test_set)

3 0.350533

2 0.318798

4 0.176357

5 0.114583

1 0.039729

Cont…

​

​

​

​

​

​

​

​

• Now you should remove the income_cat attribute so the data is back to its original state:

for set_ in (strat_train_set, strat_test_set):

set_.drop("income_cat", axis=1, inplace=True)

Discover and Visualize the Data to Gain Insights

• So far you have only taken a quick glance at the data to get a general understanding of the kind of data you are manipulating. Now the goal is to go a little bit more in depth.
• First, make sure you have put the test set aside and you are only exploring the training set.
• Also, if the training set is very large, you may want to sample an exploration set, to make manipulations easy and fast.
• In our case, the set is quite small so you can just work directly on the full set. Let’s create a copy so you can play with it without harming the training set:
• housing = strat_train_set.copy()

Visualizing Geographical Data

• Since there is geographical information (latitude and longitude), it is a good idea to create a scatterplot of all districts to visualize the data (Figure 2-11):

housing.plot(kind="scatter", x="longitude", y="latitude")

Cont…

• This looks like California all right, but other than that it is hard to see any particular pattern. Setting the alpha option to 0.1 makes it much easier to visualize the places where there is a high density of data points.

housing.plot(kind="scatter", x="longitude", y="latitude", alpha=0.1)

Cont…

• Now let’s look at the housing prices (Figure 2-13). The radius of each circle represents the district’s population (option s), and the color represents the price (option c).
• We will use a predefined color map (option cmap) called jet, which ranges from blue (low values) to red (high prices):16

housing.plot(kind="scatter", x="longitude", y="latitude", alpha=0.4,

s=housing["population"]/100, label="population", figsize=(10,7),

c="median_house_value", cmap=plt.get_cmap("jet"), colorbar=True, )

plt.legend()

Cont…

Looking for Correlations

• Since the dataset is not too large, you can easily compute the standard correlation coefficient (also called Pearson’s r) between every pair of attributes using the corr() method:
• corr_matrix = housing.corr()
• Now let’s look at how much each attribute correlates with the median house value:
• >>> corr_matrix["median_house_value"].sort_values(ascending=False)
• median_house_value 1.000000
• median_income 0.687170
• total_rooms 0.135231
• housing_median_age 0.114220
• households 0.064702
• total_bedrooms 0.047865
• population -0.026699
• longitude -0.047279
• latitude -0.142826

​

Cont…

• The correlation coefficient ranges from –1 to 1. When it is close to 1, it means that there is a strong positive correlation;
• for example, the median house value tends to go up when the median income goes up. When the coefficient is close to –1, it means that there is a strong negative correlation;
• you can see a small negative correlation between the latitude and the median house value (i.e., prices have a slight tendency to go down when you go north).
• Finally, coefficients close to zero mean that there is no linear correlation. Figure 2-14 shows various plots along with the correlation coefficient between their horizontal and vertical axes.

​

Cont…

Cont…

• Another way to check for correlation between attributes is to use Pandas’ scatter_matrix function, which plots every numerical attribute against every other numerical attribute.
• Since there are now 11 numerical attributes, you would get 112 = 121 plots, which would not fit on a page, so let’s just focus on a few promising attributes that seem most correlated with the median housing value (Figure 2-15):

from pandas.plotting import scatter_matrix

attributes = ["median_house_value", "median_income", "total_rooms",

"housing_median_age"]

scatter_matrix(housing[attributes], figsize=(12, 8))

Cont…

• The main diagonal (top left to bottom right) would be full of straight lines if Pandas plotted each variable against itself, which would not be very useful.
• So instead Pandas displays a histogram of each attribute (other options are available; see Pandas’ documentation for more details).
• The most promising attribute to predict the median house value is the median ncome, so let’s zoom in on their correlation scatterplot (Figure 2-16):

housing.plot(kind="scatter",x="median_income", y="median_house_value", alpha=0.1)

Cont…

Experimenting with Attribute Combinations

• One last thing you may want to do before actually preparing the data for Machine Learning algorithms is to try out various attribute combinations.
• For example, the total number of rooms in a district is not very useful if you don’t know how many households there are. What you really want is the number of rooms per household.
• Similarly, the total number of bedrooms by itself is not very useful: you probably want to compare it to the number of rooms. And the population per household also seems like an interesting attribute combination to look at. Let’s create these new attributes:

housing["rooms_per_household"]=housing["total_rooms"]/housing["households"]

housing["bedrooms_per_room"]=housing["total_bedrooms"]/housing["total_rooms"]

housing["population_per_household"]=housing["population"]/housing["households"]

• And now let’s look at the correlation matrix again:
• >>> corr_matrix = housing.corr()
• >>> corr_matrix["median_house_value"].sort_values(ascending=False)

median_house_value 1.000000

median_income 0.687160

rooms_per_household 0.146285

total_rooms 0.135097

housing_median_age 0.114110

households 0.064506

total_bedrooms 0.047689

population_per_household -0.021985

population -0.026920

longitude -0.047432

latitude -0.142724

bedrooms_per_room -0.259984

Name: median_house_value, dtype: float64

Prepare the Data for Machine Learning Algorithms

• This will allow you to reproduce these transformations easily on any dataset (e.g., the next time you get a fresh dataset).
• You will gradually build a library of transformation functions that you can reuse in future projects.
• You can use these functions in your live system to transform the new data before feeding it to your algorithms.
• This will make it possible for you to easily try various transformations and see which combination of transformations works best.