2. Mutual Information�

データセットに最初にすべきことは

First encountering a new dataset can sometimes feel overwhelming. You might be presented with hundreds or thousands of features without even a description to go by. Where do you even begin?

​

​

Targetとの関連の重要さを調べる

A great first step is to construct a ranking with a feature utility metric, a function measuring associations between a feature and the target.

Then you can choose a smaller set of the most useful features to develop initially and have more confidence that your time will be well spent.

相互情報量を使う！

The metric we'll use is called "mutual information". Mutual information is a lot like correlation in that it measures a relationship between two quantities.

The advantage of mutual information is that it can detect any kind of relationship, while correlation only detects linear relationships.

Mutual information 　一般的　最初に

Mutual information is a great general-purpose metric and especially useful at the start of feature development when you might not know what model you'd like to use yet. It is:

​

・easy to use and interpret,　　　解釈しやすい

・computationally efficient,

・theoretically well-founded,

・resistant to overfitting, and,　　過適応にならない

・able to detect any kind of relationship

その特徴を知ると、どのくらいtargetがわかるか

Mutual Information and What it Measures¶

Mutual information describes relationships in terms of uncertainty.

The mutual information (MI) between two quantities is a measure of the extent to which knowledge of one quantity reduces uncertainty about the other.

If you knew the value of a feature, how much more confident would you be about the target?

Here's an example from the Ames Housing data.

The figure shows the relationship

between the exterior quality of a house and the price it sold for.

Each point represents a house.

値段

fair

ほどよい

typical

典型的

good

よい

Excellent

すばらしい

(Technical note:

What we're calling uncertainty is measured using a quantity from information theory known as "entropy".

​

The entropy of a variable means roughly:

"how many yes-or-no questions you would need to describe an occurance of that variable, on average."

The more questions you have to ask, the more uncertain you must be about the variable.

Mutual information is

how many questions you expect the feature

to answer about the target.)

Interpreting Mutual Information Scores

The least possible mutual information

between quantities is 0.0.

​

When MI is zero, the quantities are independent:

neither can tell you anything about the other.

​

Conversely, in theory

there's no upper bound to what MI can be.

In practice though values above 2.0 or so are uncommon.

(Mutual information is a logarithmic quantity,

so it increases very slowly.)

０　target予測に

　　役にたたない

2以上　まれ

線形　 スコア2.12

​

ランダム　スコア 0.01

線形でなくても

相互情報量はだいたい１かそれ以上

Here are some things to remember when applying mutual information:

​

・MI can help you to understand the relative potential of a feature as a predictor of the target, considered by itself.

・It's possible for a feature to be very informative when interacting with other features, but not so informative all alone. MI can't detect interactions between features. It is a univariate metric.

・The actual usefulness of a feature depends on the model you use it with. A feature is only useful to the extent that its relationship with the target is one your model can learn. Just because a feature has a high MI score doesn't mean your model will be able to do anything with that information. You may need to transform the feature first to expose the association.

単変量

相関を利用するには変換が必要かも

Example - 1985 Automobiles

The Automobile dataset consists of

193 cars from the 1985 model year.

The goal for this dataset is to predict a car's price (the target) from 23 of the car's features,

such as make, body_style, and horsepower.

In this example, we'll rank the features with mutual information and investigate the results by data visualization.

価格

The scikit-learn algorithm for MI treats

discrete features differently from continuous features.

Consequently, you need to tell it which are which.

As a rule of thumb, anything that must have a float dtype is not discrete.

Categoricals (object or categorial dtype) can be treated as discrete by giving them a label encoding. (ラベルを数値に変換）

(You can review label encodings in our Categorical Variables lesson.)

(解説）

• X[colname].factorize() returns two output values separated by a comma - codes and unique.
• codes: Gives the encoded values of the column
• eg: If the column is ['b', 'b', 'a', 'c', 'b'], "codes" gives the array [0, 0, 1, 2, 0]
• unique: Gives the unique categorical variables of the column
• eg: If the column is ['b', 'b', 'a', 'c', 'b'], "unique" gives the array ['a', 'b', 'c']

Scikit-learn has two mutual information metrics in its feature_selection module:

​

one for real-valued targets (mutual_info_regression) and

one for categorical targets (mutual_info_classif).

​

Our target, price, is real-valued.

The next cell computes the MI scores for our features and

wraps them up in a nice dataframe.

Estimate mutual information

for a continuous target variable.

One-dimensional ndarray

with axis labels

車両重量

価格

​

The fuel_type feature has a fairly low MI score, but as we can see from the figure, it clearly separates two price populations with different trends within the horsepower feature. �This indicates that fuel_type contributes �an interaction effect and might not be unimportant after all. �Before deciding a feature is unimportant from its MI score, it's good to investigate any possible interaction effects -- domain knowledge can offer a lot of guidance here.

Data visualization is a great addition

to your feature-engineering toolbox.

​

Along with utility metrics like mutual information, visualizations like these can help you discover important relationships in your data.

Check out our Data Visualization course to learn more!

演習にgo

Your Turn

​

Rank the features of the Ames Housing dataset

and choose your first set of features to start developing

​