TECHNIQUES UNDER DATA MINING

-Avantika Deb

• Data mining is the process of finding anomalies, patterns and correlations within large data sets to predict outcomes.
• Using a broad range of techniques, we can use this information to increase revenues, cut costs, improve customer relationships, reduce risks and more.
• It involves steps like cleaning raw data, finding patterns, creating models, and testing those models. There is incorporation of statistics, machine learning, and database systems.
• Some of the fundamental techniques of data mining are as follows:
• Classification analysis: used to classify different data in different classes based on the past knowledge of different classes or clusters. An example can be classification of emails as legitimate or spam.

​

2. Clustering analysis: is the process of discovering groups and clusters in the data in such a way that the degree of association between two objects is highest if they belong to the same group and lowest otherwise without previous knowledge of classes/clusters. It can be used to create customer profiling.

​

​

3. Regression Analysis: is the process of identifying and analyzing the relationship among variables. It helps to understand the characteristic value of the dependent variable changes, if any one of the independent variables is varied. It is used for prediction and forecasting.

​

• Outlier or anomaly detection: refers to the observation for data items in a dataset that do not match an expected pattern or an expected behavior. These deviate considerably from the common average within a dataset or a combination of data. These types of items are statistically aloof as compared to the rest of the data and hence, it indicates that something out of the ordinary has happened and requires additional attention.  Outliers are generally removed to improve the accuracy of the results.

​

• Association: refers to the method to identify some relations between different variables in the dataset. This technique can help detect hidden patterns in the data that can be used to identify variables within the data and the concurrence of different variables that appear very frequently in the dataset. Association rules are useful for examining and forecasting customer behavior.

​

​

We will take up a dataset and conduct a case study to understand the importance and application of the various techniques of data mining listed before.

​

BANK MARKETING CAMPAIGN

• The dataset used in the following case study is based on direct marketing campaigns (phone calls) of a banking institution. The marketing campaigns were based on phone calls. Often, more than one contact to the same client was required, in order to access if the product (bank term deposit) would be ('yes') or not ('no') subscribed.

​

• The goal is to study the factors which would lead to subscription of a term deposit (variable y) by the client thus giving an insight to decide appropriate bank campaigns to get more clients to sign up for a term deposit.

​

• We will also look into various classification techniques of ML and see which one is most suitable for the given dataset.

​

​

Some of the important attributes that the dataset has as its columns are as follows:

Age: (numeric)

Job : type of job (categorical)

Marital : marital status (categorical)

Education (categorical)

Default: has credit in default? (categorical)

Housing: has housing loan? (categorical)

Loan: has personal loan? (categorical)

Contact: contact communication type (categorical)

Cons.price.idx: consumer price index - monthly indicator (numeric)

​

y - has the client subscribed a term deposit? (binary: 'yes', 'no')

​

​

​

DATA PREPROCESSING

Loading of the dataset

An overview of the dataset’s attributes

• We have a couple of irrelevant columns in the dataset which aren’t required to determine our target variable “y” i.e subscription of term deposit by the client. Hence, we drop those columns from the dataset.

• Checking for any null values present in the data. This

dataset contains no null values.

DETECTION AND REMOVAL OF OUTLIERS

We plot a boxplot of all the attributes but focus on the age because it is the only relevant numeric variable and observe that the age distribution is positively skewed with a couple of outliers at age>=70

Since our data distribution is skewed, we use IQR based

filtering to remove the outliers. The new boxplot shows the

successful removal of outliers.

EXPLORATORY DATA ANALYSIS

• Age-wise distribution of clients who opted for a term deposit in the bank

The distribution is positively skewed with maximum clients

belonging to the age group of 29-35 years.

• Distribution of clients opting for term deposit with

respect to the number of contacts performed

during this campaign and for this client

We observe that as the frequency of contacts increases, the number of clients converted reduces which makes sense as the clients might tend to get irritated after too many calls leading to non-conversion of calls.

• Monthly trend of clients subscribing to a

term deposit

Maximum clients were converted in the month of

May whereas the the least number of clients were

converted in the month of December.

• Job-wise distribution of clients subscribing to

a term deposit

Maximum clients who opted for a term deposit in the

bank did a job of the administration or technical type whereas the least no. of clients were housemaids which is understandable

given the low amount of savings they tend to have.

• Looking into the distribution of converted clients on

the basis of their education we observe that maximum

of them have a university degree whereas illiterate

clients are rare. This is obvious because the more educated

people are aware of banking habits and its benefits.

• Another observation shows us that maximum clients

who subscribed to a term deposit were married.

RELATIONSHIP B/W THE VARIABLES AND ‘Y’(ASSOCIATION)

• Most of the important variables in the dataset are categorical including the output variable and hence we have to use the chi square test to study the independence of attributes. This test was used to see the relationship b/w y and age, job type, education, having personal loan, having housing loan, mode of contact and outcome of previous campaign.
• We set the level of significance at 5%, if the p-value is greater than 0.05 then the attributes are independent/not correlated.
• It was observed that a client subscribing to a term deposit is independent of them having personal loan or housing loan.

​

A small snippet from the entire code

CLASSIFICATION ANALYSIS

• Our target variable ’y’ i.e whether the client subscribed to the term loan is a categorical variable giving binary results- yes & no. Hence, we have to use classification machine learning techniques for predictive modelling.
• The following models will be implemented on the data set:
• Logistic Regression
• Naïve Bayes
• K Nearest Neighbors
• SVM
• Decision Tree
• Random Forest

We will also evaluate which model is the best fit for this dataset.

​

​

• Before we can implement any ML technique on the dataset, we first preprocess the data like assigning numeric labels to the categorical columns by Label Encoding or One Hot Encoding.
• Also, attributes like having personal or housing loan had no effect on the target variable so we will drop those columns as well.

​

​

​

​

​

​

​

​

• We split this dataset into testing and training datasets in the ratio 3:7.

This is what the dataset looks like after one hot encoding. Everything has been converted

into numeric values and the dataset is now ready for predictive modelling.

1) LOGISTIC REGRESSION

• Logistic regression is a simple and efficient method for binary and linear classification problems.
• Logistic regression essentially uses a logistic function defined below to model a binary output variable. The most common logistic regression models a binary outcome; something that can take two values such as true/false, yes/no, and so on. Multinomial logistic regression can model scenarios where there are more than two possible discrete outcomes.
• After fitting this model to our dataset, we observe an accuracy of roughly 90%.

2) NAÏVE BAYES

• Naive Bayes methods are a set of supervised learning algorithms based on applying Bayes’ theorem with the “naive” assumption of conditional independence between every pair of features given the value of the class variable.
• It is a probabilistic classifier, which means it predicts on the basis of the probability of an object.
• On fitting this model to our dataset, it predicted values with an accuracy of 81% which is pretty low. This is explained due to the fact that our data was positively skewed and our model is gaussian.

​

3) DECISION TREE

• It is a popular tool for classification and prediction. A decision tree is a flowchart like tree structure, where each internal node denotes a test on an attribute, each branch represents an outcome of the test, and each leaf node (terminal node) holds a class label. 
• This model showed an accuracy level of 85% when fitted to our dataset.
• A random forest would perform better in this case as our dataset is pretty large.

​

4) RANDOM FOREST

• Random forest, like its name implies, consists of a large number of individual decision trees that operate as an ensemble. Each individual tree in the random forest spits out a class prediction and the class with the most votes becomes our model’s prediction.
• As expected, the random forest model performed better than the decision tree model, showing an accuracy level of 88%.

​

5) SVM

• It is an algorithm which can be used for both classification and regression problems.
• The goal of the SVM algorithm is to create the best line or decision boundary that can segregate n-dimensional space into classes so that we can easily put the new data point in the correct category in the future. This best decision boundary is called a hyperplane. This hyperplane can be linear, radial, polynomial etc.
• This model showed an accuracy level of 89% when used on our dataset.

​

6) K NEAREST NEIGHBOURS

• K-Nearest Neighbour is one of the simplest Machine Learning algorithms based on Supervised Learning technique. K-NN algorithm assumes the similarity between the new case/data and available cases and put the new case into the category that is most similar to the available categories.
• K-NN algorithm can be used for Regression as well as for Classification but mostly it is used for the Classification problems.
• This model predicts values of the dataset with an accuracy level of 89%.

​

VISUALIZATION OF CLASSIFICATION TECHNIQUES

Logistic Regression

The two classes are neatly demarcated with a straight line

Naïve Bayes

A smooth demarcation is observed between the two classes

Characteristic horizontal and vertical

demarcation lines throughout the graph

Decision Tree

VISUALISATION OF CLASSIFICATION TECHNIQUES

Random Forest

Similar to decision tree, exhibits

vertical and horizontal lines of demarcation b/w the two classes

SVM

KNN

Line of demarcation is jagged

SVM with linear kernel has

a linear demarcation

INFERENCES FROM THE ANALYSIS

• Out of all the classification techniques implemented here, the logistic regression model performs the best on this data set with the highest accuracy level of 90%.

​

• Regression techniques cant be used on this dataset as most of the attributes are categorical in nature including our target variable ‘y’ which is whether the client subscribed to a term deposit or not.

​

• Trivial facts for an ideal client i.e age, job, educational qualifications, etc are backed up by the data analysis conducted. Eg. An administrative worker in his early 30s who is recently married would be more aware of banking habits than a illiterate 55 year old individual or 5^th grade pass out in his early 20s.

​

​

SUGGESTIONS FOR BANK CAMPAIGN

1. To ensure efficiency of the bank campaign, clients from the age group of 29 to 35 years should be given more preference over other age groups.
2. Well-educated individuals completing at least a university level degree should be targeted.
3. Administrative and technical job-holders seem to opt for term deposits more than the others so more importance should be given to them.

​

​

4. Positive conversion rate of calls can be established with minimum number of calls. The bank should avoid spamming(>2 calls) their clients as it might lead to loss of goodwill of their customers and wastage of resources.
5. The campaign should be held during the months of April and May when maximum number of recruits start their jobs, keeping a fresh fiscal year in mind.
6. Married customers should be given more preference.

​

​

DATA MINING TOOLS

• Rapid Miner
• Oracle Data Mining
• Python
• Weka
• IBM SPSS Modeler

• Knime
• Kaggle
• Orange
• Apache Spark
• Teradata

The most popular data mining tools are as follows: