常見資料探勘工作

Business Problems and Data Science Solutions

(Chapter 2 of Data Science for Business)

​

1

Amazon #1 Best Seller in Database Storage and Design

★★★★+half★ (184)

我們前一章討論過的案例

• WallMart: 預測颱風過境時，哪些商品會熱賣? 以便預備充足庫存
• 電信公司預測顧客合約到期時，是否會流失? 我們向顧客建議優惠方案時，他是否會回應?
• Target: 預測有顧客已經懷孕
• Netflix: 預測用戶對一部影片的評等
• Capital One: 建立預測信用卡顧客的獲利率模型 (Fairbanks & Morris)

2

兩種由資料推動的決策

1. 藉由發現資料背後的訊息來做決策
2. 重覆的決策，特別是大規模的決策，藉由資料分析即使提高一點點的決策精確度都能獲得可觀的回報

3

我們討論過的案例

• WallMart: 預測颱風過境時，哪些商品會熱賣? 以便預備充足庫存 (1)
• 電信公司預測顧客合約到期時，是否會流失? 我們向顧客建議優惠方案時，他是否會回應? (2)
• Target: 預測有顧客已經懷孕 (1)
• Netflix: 預測用戶對一部影片的評等 (2)
• Capital One: 建立預測信用卡顧客的獲利率模型 (Fairbanks & Morris) (2)

4

From Business Problems to Data Mining Tasks (將企業問題轉成資料探勘的工作)

• Often we want to find correlations(相關性) between a particular variable describing an individual and other variables. For example, in historical data we may know which customers left the company after their contracts expired. 我們經常想要找出一些變數跟描述某個人狀況之關連性，例如我們想要知道哪些客戶在合約到期後會流失
• We may want to find which other variables correlate with a customer leaving in the future. 我們想要找出哪些變數跟客戶未來流失有關連性
• Finding such correlations are the most basic examples of classification (分類) and regression (迴歸) tasks.

5

6

機器學習

X

y

建立關聯性

Machine Learning ≈ Looking for a Function

7

Framework

8

Framework

9

A set of function

Training Data

Goodness of function f

Better

​

Function input:

​

​

Function output: “monkey” “cat” “dog”

Supervised Learning

Dog or muffin?

10

Dog or mop?

11

All kinds of Cats

12

AI Machine Learning

Traditional Programming

Machine Learning

Human

Programmer

Program

Output

Input

Input

Data

Learning

Algorithm

Program

Output

谷歌雲首席科學家李飛飛：人工智慧 極客公園 2017 年 大會演講

​

Framework

14

A set of function

Training Data

Goodness of function f

“monkey” “cat” “dog”

Using f*

Pick the “Best” Function

f*

Training

Testing

Step 1

Step 2

Step 3

“cat”

Correlation ⇒ Causation (相關未必代表因果關係)

• Harvard University medical researchers have concluded that American men between the ages of 45 to 82 who skip breakfast showed a 27% higher risk of coronary heart disease over a 16-year period. 哈佛大學發現不吃早飯的人有27%較高的風險得到冠狀心臟病
• However, rather than directly affecting health, eating breakfast may more simply be a proxy for lifestyle. 然而不吃早餐並不是直接影響人的心臟健康，而是反應人的生活型態

​

15

Correlation ⇒ Causation (相關未必代表因果關係)

• People who skip breakfast tend to lead more stressful lives. Participants in the study who skipped breakfast “were more likely to be smokers, to work full time, to be unmarried, to be less physically active, and to drink more alcohol,” Harvard researchers report. In other words, the link between breakfast and health may not be causal.不吃早餐的人生活常過於緊張壓力大，參與哈佛大學受測不吃早餐的人，許多人抽菸、全時間工作、未婚、較不活動、喝較多的酒。換句話說，早餐跟健康可能非因果關係。

16

Correlation ⇒ Causation (相關未必代表因果關係)

• Uber offers another useful example. The company discovered that, in San Francisco, more passengers request rides from areas with higher rates of prostitution, alcohol use, theft, and burglary. Uber發現在舊金山有較多的顧客從妓女、使用酒精、偷竊，搶劫比率高的區域搭乘
• However, the company knows that crime itself is not necessarily causing this higher demand, even indirectly. 我們能說犯罪本身導致較高搭乘Uber的需求嗎?

17

Correlation ⇒ Causation (相關並不代表因果關係)

• Rather, their original hypothesis, even before the analysis, was that “crime should be a proxy for nonresidential population.” Higher-crime areas tend to have more people who don’t live in the immediate vicinity, who in turn need rides. Uber的假設是犯罪者常非當地住民，犯罪率較高的區域有許多人並非住在臨近地方，因而需要叫車。

18

太早做結論可能會做出被誤導的決策

• 假設一個線上車商發覺他的網站訪客如果有用網站上的價格計算器最後較會買單
• 有了這個觀察，車商想針對瀏覽網站但沒有使用價格計算器的客戶提供折扣，以便增加他們買車的可能
• 然而上述的觀察並沒有解釋到底是甚麼因素影響顧客的買車的決定，也許有意願的顧客本來就會試試網站上的功能
• 所以努力去開發利用上述的觀察可能不會增加車商的銷售量
• 我們需要有更多的分析去確認一個觀察背後真正的原因

​

19

Classification

• Classification(分類; 例如預測明天是雨天或晴天) and class probability estimation (類別機率的估計;例如預測明天下雨的機率) attempt to predict, for each individual in a population, which of a (small) set of classes this individual belongs to. 每個個體屬於哪一個類別?
• Example question: “Among all customers of MegaTelCo, which are likely to respond to a given offer?” 手機公司的客戶，哪一些會回應公司的行銷案?
• Two classes: will respond and will not respond.
• Scoring or class probability estimation
• Score representing the Probability (quantification of likelihood)

20

Regression (回歸)

• Regression (“value estimation”) attempts to estimate or predict, for each individual, the numerical value of some variable for that individual. 回歸是對每個個體，估計某個變數的值
• 天氣預報最高溫最低溫的預測
• Example question: “ How much will a given customer use the service?” 一位顧客會使用多少金額的服務?
• Classification predicts whether something will happen.
• Regression predicts how much something will happen.

21

Similarity matching (相似度配對)

• Similarity matching attempts to identify similar individuals based on data known about them. 找出相似的個體
• Ex: IBM is interested in finding companies similar to their best business customer, in order to focus their sales force on the best opportunities. IBM找出跟他們最佳顧客相似的公司，以便讓銷售團隊專注在最佳機會
• Recommendations (finding people who are similar to you in terms of the products they have liked or have purchased) 推薦: 根據喜歡或購買的產品，找出跟你相似的人

22

Similarity matching

• Netflix從顧客過去對看過的影片的喜好反應去找出口味相似的顧客以便做影片推薦。
• 舉凡每一次點擊、播放、快轉、暫停，搭配上導演、演員、影片類型等，都會被Netflix列入資料庫，結合大數據及資料探勘（Data Mining）技術，建立個人化推薦引擎，為用戶推薦下一部影片，讓他們有「這個看完，還有下一個影集可以看」之感

23

Clustering (群集)

• Clustering attempts to group individuals in a population together by their similarity, but not driven by any specific purpose. 將母體中相似的個體分群
• Example question: “ Do our customers form natural groups or segments?” 我們的顧客是否有自然的分群 (見下頁)
• Decision-making processes. (What products should we offer or develop? How should our customer care teams (or sales teams) be structured?) 我們該提供或開發那些產品給顧客? 我們該如何組織我們的客服團隊?

24

顧客細分(Customer Segmentation）

• 客戶細分是20世紀50年代中期由美國學者溫德爾·史密斯提出的，其理論依據主要有兩點。
• 顧客需求的異質性: 並不是所有顧客的需求都相同，只要存在兩個以上的顧客，需求就會不同。由於顧客需求、欲望及購買行為是多元的，所以顧客需求滿足呈現差異。
• 企業有限的資源和有效的市場競爭:任何一個企業不能單憑自己的人力、財力和物力來滿足整個市場的所有需求，這不僅緣於企業自身條件的限制，而且從經濟效應方面來看也是不足取的。因為，企業應該分辨出它能有效為之服務的最具有吸引力的細分市場，集中企業資源，制定科學的競爭策略，以取得和增強競爭優勢。
• 客戶細分是指根據客戶屬性劃分的客戶集合。 它既是客戶關係管理（Customer Relationship Management，CRM）的重要理論組成部分，又是其重要管理工具。

​

​

25

Clustering (群集) 應用

• 電話公司要在一個區域架設一些基地台，目標試讓它的用戶有最佳的訊號強度
• 一個醫療連鎖機構要開設緊急醫療病房，它將最容易發生意外的地點納入考量
• 將Wiskey分群，每一群進幾瓶到我店裡來。Wiskey有顏色、嗅覺、酒體、味覺、餘韻等特徵

​

​

26

Co-occurrence grouping (共生分群)

• Co-occurrence grouping attempts to find association between entities based on transactions involving them. 根據交易的項目，找到項目間關聯性
• Example question: “ What items are commonly purchased together?” 是否有哪些物品常被顧客一起購買?
• Ex: analyzing purchase records from a supermarket. 分析超市購買紀錄

27

Co-occurrence grouping (續)

• While clustering looks at similarity between objects based on the objects’ attributes, co-occurrence grouping considers similarity of objects based on their appearing together in transactions. Clustering的近似是根據物件的屬性，co-occurrence grouping的近似則是根據物件是否出現在相同交易中
• For example, analyzing purchase records from a supermarket may uncover that ground meat (絞肉) is purchased together with hot sauce (辣醬) much more frequently than we might expect. (suggest a special promotion, product display, or both) 分析超級市場購買記錄後，發覺絞肉跟辣醬一起被買，頻率超乎預期

28

Co-occurrence grouping (續)

• Recommendation system (affinity grouping finds, for example, pairs of books that are purchased frequently by the same people) 兩本書常被一起購買

29

Profiling (剖析)

• Profiling (also known as behavior description(形為描述)) attempts to characterize the typical behavior of an individual, group, or population. 找出個體、群體或母體常見的行為
• Example question: “ What is the typical cell phone usage of this customer segment?” 這一群顧客使用手機的主要模式
• Profiling is often used to establish behavioral norms for anomaly detection applications. Profiling常用在為了異常偵測建立行為準則
• Fraud detection and monitoring for intrusions to computer systems 例如，舞弊偵測與電腦入侵監控

30

Link prediction (連結預測)

• Link prediction attempts to predict connections between data items, usually by suggesting that a link should exist, and possibly also estimating the strength of the link. 預測資料項之間是否有連結，或是連結的強度
• “Since you and Karen share 10 friends, maybe you’d like to be Karen’s friend?” 例如，因為你跟凱倫有十位共同朋友，也許你也想成為凱倫的朋友

31

Data reduction (資料縮減)

• Data reduction attempts to take a large set of data and replace it with a smaller set of data that contains much of the important information in the larger set. 將大資料集以小資料集取代，小資料集包含大資料集的重要資訊
• Ex: a massive dataset on consumer movie-viewing preferences may be reduced to a much smaller dataset revealing the customer taste preferences. 例如，將顧客看電影喜好巨大的資料集縮減為一個小很多的資料集，但還是反映顧客的偏好

​

32

Data reduction (資料縮減)

• 鸢尾花（Iris）的例子:相對於三種不同的花，花瓣(petal)包含的辨別資訊相對於花萼(sepal)來說要更多一些，因為圖形中花萼的寬度和長度差別更小一些。那麼，該資訊就可以用於特徵選擇，以減少我們的資料集的大小。

​

33

Causal modeling (建立因果關係模型)

• Causal modeling attempts to help us understand what events or actions actually influence others. 要了解哪些事件或動作會真實影響其他的事情
• Ex: Consider that we use predictive modeling to target advertisements to consumers, and we observe that indeed the targeted consumers purchase at a higher rate subsequent to having been targeted. Was this because the advertisements influenced the consumers to purchase? Or did the predictive models simply do a good job of identifying those consumers who would have purchased anyway? 假設我們使用一個預測模型去決定哪些(類)顧客將看到一支新廣告，我們發覺目標客戶事後的確較常購買。但這是因為這支廣告影響了顧客去購買，還是預測模型”準確”的找到本來就會購買的顧客?

34

本書主要討論內容

• Classification
• Regression
• Similarity matching
• Clustering

​

35

Classification

• 合約到期客戶
• 財報待查客戶
• 信用卡客戶獲利

• 預測會流失
• 預測不會流失

​

​

• 預測會舞弊
• 預測不會舞弊

​

​

• 預測會獲利
• 預測不會獲利

​

Class Probability Estimation

• 合約到期客戶
• 財報待查客戶
• 申請信用卡客戶

• 預測會流失的機率

​

​

​

• 預測會舞弊的機率

​

​

​

• 預測會獲利的機率

​

Regression

• 合約到期續約費率
• 信用卡客戶每月可獲利金額

Clustering

• 有關蘋果公司的新聞報導
• 蘋果股價
• 蘋果新產品發布
• 蘋果創辦人

Supervised (監督式) vs. Unsupervised (非監督式) Methods

• Unsupervised: no specific purpose or target specified 非監督式: 沒有特別的目的或指定的目標
• “Do our customers naturally fall into different group?” 我們的顧客是否可分為不同的群組?
• Supervised: specific target defined (有明確的目標)
• “ Can we find groups of customers who have particularly high likelihoods of canceling their service soon after their contracts expires?” 🡪 Target: Will a customer leave when her contract expires? 我們能否找出合約到期後，較有可能取消我們服務的顧客?

​

40

Supervised vs. Unsupervised Methods

• So, if a specific target can be provided, the problem can be phrased as a supervised one. 如果有特定目標，即屬於監督式
• Supervised tasks require different techniques than unsupervised tasks do, and the results often are much more useful. 監督式方法跟非監督式方法使用不同的技術，一般而言，監督式方法的結果較為有用

​

​

41

Supervised vs. Unsupervised Methods

• Supervised data mining: there must be data on the target. 要做監督式資料探勘，我們必須要有目標的資料
• 例如目標是預測哪些顧客會維持2年的顧客關係，那麼我們就要預備過去顧客與我們維持關係的資料:

Customer ID …………………….. Stay for 2 years

1546 …………………….. YES

3245 …………………….. NO

• Acquiring data on the target is a key data science investment. 取得目標變數的資料是資料科學的關鍵投資

42

Supervised vs. Unsupervised Methods

• The value for the target variable for an individual is often called the individual’s label (標籤). So customer 1546’s label is YES. 一個個體的目標變數值我們一般稱做標籤，顧客1546的標籤是YES

43

Supervised vs. Unsupervised Methods

• Two main subclasses of supervised data mining

兩種監督式資料探勘的主要方法:

• Classification 分類
• “Will this customer purchase service S1 if given incentive I?” 如果我們提供誘因I, 這位顧客會買服務S1嗎?
• “Which service package(S1, S2 , or none) will a customer likely purchase if given incentive I?”如果我們提供誘因I, 這位顧客會買哪一個服務方案(S1, S2, 或都不買)?
• “What is the probability that a customer will continue to subscribe to the service?” 🡪 class probability estimation 顧客會持續訂購服務的機率🡪 估計類別機率
• Regression 回歸
• “How much will this customer use the service” 顧客會用這個服務多少金額?

44

Supervised vs. Unsupervised Methods

• A vital part in the early stages of the data mining process 資料探勘起初階段的重要工作
• To decide whether the line of attack will be supervised or unsupervised. 決定以監督式或非監督式方法去解問題
• If supervised, to produce a precise definition of a target variable. This variable must be specific quantity that will be the focus of the data mining. 如果是監督式，便要將目標變數清楚定義。此變數必須成為接下來資料探勘的焦點

45

46

Decision

Tree

機器學習

監督式學習

非監督式學習

強化式學習

迴歸問題

分類問題

分群問題

降維

決策樹

線性迴歸

羅吉斯迴歸

支援向量機

主成分分析

k-平均演算法

教機器 學習

人類的學習方式

大數據(原料) 🡺 機器學習(處理器)

🡺人工智慧(結果)

Reinforcement learning (強化學習) �

47

​

​

​

​

​

​

​

​

​

• NVIDIA Isaac Robot Simulator　黃仁勳 (Jensen Huang)

​

• 人工智慧系統遭遇最強人類檢驗

​

​

Data Mining and Its Results

• Distinction pertaining to mining data: 探勘資料兩大部分
• Mining the data to find patterns and build models. 找出資料背後的模式，並且建立預測模型
• Using the results of data mining. 使用探勘結果在未來的資料上
• See Figure 2-1. Data mining versus the use of data mining results. 資料探勘 vs 使用資料探勘的結果

​

48

Data Mining and Its Result

49

The Data Mining Process

• Cross Industry Standard Process for Data Mining (CRISP-DM) 跨產業資料探勘標準流程

50

Business Understanding 瞭解業務

• It is vital to understand the problem to be solved. 了解要解決的問題是最關鍵的
• A part of the craft where the analysts’ creativity plays a large role. 分析師的創意扮演重要的部分 (如何將企業問題變成資料探勘問題)
• The design team should think carefully about the use scenario. 設計團隊必須仔細思考未來使用資料探勘的情境

51

Data Understanding 瞭解資料

• The data comprise the available raw material from which the solution will be built. 資料是將來解決方案的原料
• Estimating the costs and benefits of each data source and deciding whether further investment is merited. 必須估計每項資料來源的成本與效益，同時決定是否需要更進一步的投資(在資料上)
• Ex: 資料探勘已被廣泛用在舞弊偵測上，但不同的種類的舞弊需要不同的資料探勘方法去偵測
• Credit card fraud
• Medicare fraud

52

Data preparation 準備資料

• Often proceeds along with data understanding. 與對資料的了解同時進行
• Ex.
1. converting data to tabular format. 將資料轉成表格的形式
2. removing or inferring missing values. 捨去有移失的資料項，或做合理推論
3. converting data to different types. 將資料轉換成不同的資料形態

53

Data preparation 準備資料

• Leaks

A variable collected in historical data gives information on the target variable-information that appears in historical data but is not actually available when the decision has to be made. 在歷史資料中有的資訊，但在做決策時卻沒有的資訊。例如，我們想預測網站訪客在特定時間點是否會繼續流覽，「本次瀏覽總網頁數」被發現是有預測能力，然而此變數的值要到訪客結束瀏覽才會知道

Leakage must be considered carefully during data preparation. 在預備資料得時候，需要考量這樣的問題

54

Modeling 建立預測模型

• Output of modeling is some sort of model or pattern capturing regularities in the data. 此階段產生的是抓住資料背後規則的模型或是模式

55

Evaluation

• Assess the data mining results rigorously and to gain confidence that they are valid and reliable before moving on. 此階段要評估資料探勘的結果，並且確認它是有效與可靠的
• Includes both quantitative and qualitative assessments. Stakeholders should check and see whether the model is going to do more good than harm. 這包括量與質的評估。利害關係人必須檢視採用此模型是有利的
• Data science team must consider the comprehensibility of the model to the stakeholders 資料科學團隊必須考量利害關係人可以理解此模型

56

Deployment (佈署)

• Put into real use in order to realize some return on investment. 模型如果通過評估後，將被實際用在相關業務上，期能達到一定的投資報酬率
• The clearest cases of deployment involve implementing a predictive model in some information system or business process. 最顯而易見的佈署方式是將模型納入相關資訊系統或企業流程內
• Churn example: A model for predicting the likelihood of churn could be integrated with the business procedures for churn management --- for example, by sending special offers to customers who are predicted to be particularly at risk. 預測流失機率的模型可以跟管理流失的企業流程整合，例如對於預測極可能流失的客戶，提供特別的優惠

​

57

Deployment (佈署)

• Can also be mush less “technical”; 一個簡單的例子是排除工業印刷機障礙，解決的方法是把資料探勘找到的排除法則寫好貼在所有的印刷機旁
• Can also be more subtle 也可能很複雜，例如改變資料收集的程序、改變策略、行銷等。
• 不論佈署是否成功，我們都需要回到Business Understanding階段 (見CRISP-DM)。資料探勘的過程通常會對企業的問題有進一步深入的洞察。再走一次流程常常能得到更佳的解決方案。

58

機器學習與資料探勘 Machine Learning and Data Mining

• 「從資料萃取出預測模型」在好些領域同時被研究，包括機器學習、應用統計，與模式識別 (pattern recognition)
• 機器學習是「人工智慧」的一支，而人工智慧主要研發改進智慧代理(如機器人)之知識與表現。這種改進通常藉由分析由環境收集來的資料，然後對未知的事物做預測
• 機器學習這種分析資料的能力，經過這些年的研發已經在人工智慧領域扮演非常重要的角色
• 經過多年各自的研發，現今，機器學習、應用統計，與模式識別的區隔已經越來越小
• 機器學習的應用包羅萬象

​

59

Summary

• Data mining is a craft. As with many crafts, there is a well-defined process that can help to increase the likelihood of a successful result. 資料探勘是一項技藝，如同許多其他的技藝，它有一個清楚定義的流程，有助提升達到成功的機率
• We will refer back to the data mining process repeatedly throughout the book, showing how each fundamental concept fits in. 本書後面的章節會不斷提到資料探勘流程，同時深入討論流程中的基本觀念

60