INTRODUCTION TO DATA MINING AND DATA WAREHOUSING

Prof.S.Deepa

Assistant Professor(Sr.G)

Department of Computer Technology – PG

Kongu Engineering College(Autonomous)

INTRODUCTION TO DATA MINING SYSTEMS

Data mining extracts useful patterns and knowledge from large data sets.

It converts raw data into meaningful information for decision making.

Data is collected daily from business, science, healthcare, and social media.

The huge data volume created the need for automatic analysis tools.

EVOLUTION OF DATA MINING

Developed as a natural step in information technology

evolution.

Progressed from data collection → database management

→ data warehousing → data mining.

Helps turn “data tombs” into “golden nuggets” of knowledge.

KNOWLEDGE DISCOVERY PROCESS

A step-by-step process called KDD (Knowledge Discovery in

Databases).

Steps: Data cleaning → integration → selection → transformation

→ mining → evaluation → presentation.

Each step improves the quality and usefulness of the mined knowledge.

KINDS OF DATA IN DATA MINING

Data mining works on many data types: Database data

Data warehouse data Transactional data

Text, multimedia, spatial, web, and time-series data.

Complex data needs special tools for mining.

DATA MINING TECHNIQUES

Characterization – summarize data.

Discrimination – compare different groups.

Frequent Pattern Mining – find associations (e.g., “milk → bread”).

Classification – predict categories. Regression – predict numeric values. Clustering – group similar data.

Outlier Detection – find unusual or rare cases.

APPLICATIONS OF DATA MINING

Business intelligence for marketing, sales, and customer

analysis.

Search engines for ranking and personalization. Healthcare for disease prediction.

Finance for fraud detection.

E-commerce for recommendation systems.

ISSUES IN DATA MINING

Methodology: handle noise, uncertainty, and large datasets.

​

User Interaction: allow dynamic and visual mining.

​

Efficiency: need fast, scalable algorithms.

​

Data Diversity: manage structured and unstructured data.

​

Society: ensure privacy and ethical use.

DATA WAREHOUSING – BASIC CONCEPTS

A data warehouse stores integrated, historical, and non- volatile data.

Defined as: subject-oriented, integrated, time-variant, nonvolatile.

Helps managers analyze trends and make business decisions.

DIFFERENCES BETWEEN OLTP AND OLAP

Users:

OLTP is used by clerks and IT staff for daily operations.

OLAP is used by managers and analysts for business analysis.

Data:

OLTP stores current and detailed data. OLAP stores historical and summarized data.

Data Model:

OLTP uses the Entity-Relationship (ER) model. OLAP uses Star or Snowflake model.

Usage:

OLTP is used for day-to-day transactions. OLAP is used for decision making and analysis.

NEED FOR A SEPARATE DATA WAREHOUSE

OLTP and OLAP have different purposes — one for transactions, the other for

analysis.

Complex analytical (OLAP) queries can slow down transaction systems if both use the same database.

A data warehouse stores clean, integrated, and historical data for better analysis.

It helps in fast query performance without affecting daily operations.

It provides a centralized source for decision-making and business insights.

DATA WAREHOUSE ARCHITECTURE

Three-Tier Architecture:

​

1. Bottom: Database server – extraction, cleaning, loading.

​

• Middle: OLAP server – relational or multidimensional analysis.

​

• Top: Client tools – queries, reports, mining tools.

Metadata repository stores information about warehouse contents

DATA WAREHOUSE MODELS

Enterprise Warehouse:

Covers the entire organization’s data.

Provides a centralized view for all departments.

Data Mart:

A smaller, department-level warehouse (e.g., sales, finance). Easier to build and maintain than a full enterprise warehouse.

Virtual Warehouse:

Provides logical (virtual) views over operational databases.

Does not store data physically, only accesses data when needed.

Helps in faster decision-making as each model serves a specific business need.

ETL PROCESS

Extract: collect data from multiple sources.

Clean: remove errors and duplicates. Transform: convert data into standard format. Load & Refresh: store and update in warehouse. Ensures high-quality data for mining.

METADATA REPOSITORY

1. Metadata means “data about data.”

​

• Describes structure, lineage, and transformation rules of data.

​

• Helps analysts find, understand, and use data warehouse contents.

​

• Acts as a guide or dictionary for the entire data warehouse.

​

• Ensures data consistency and easier maintenance across systems.

DATA CUBE & OLAP

• A Data Cube provides a multidimensional view (e.g., Time × Location × Item).
• OLAP Tools allow users to analyze data interactively.
• Enables analysis at different levels of detail (summary or detail).
• Supports fast and flexible query operations for decision-making.
• Helps users visualize and compare trends easily across dimensions.

DATA WAREHOUSE SCHEMAS

Star Schema: One fact table connected to many dimension tables.

Snowflake Schema: Dimension tables are normalized to reduce redundancy.

Fact Constellation (Galaxy): Multiple fact tables share common dimensions.

Schemas help organize data efficiently for queries and analysis.

The choice of schema depends on data complexity and storage needs.

CONCEPT HIERARCHIES

Define levels of data (e.g., City → State → Country).

Used for roll-up (summarizing) and drill-down (detailing) operations. Help in summarizing or detailing data easily.

Provide different views of data for analysis.

​

Make data navigation and reporting simpler and clearer.

TYPES OF MEASURES

1. Distributive: Can be combined (e.g., SUM, COUNT, MIN, MAX).
2. Algebraic: Computed using distributive results (e.g., AVERAGE, STD DEV).
3. Holistic: Need all data to compute (e.g., MEDIAN, MODE).

​

• Help in measuring business performance and patterns.

​

• Choosing the right measure type improves accuracy of results.

OLAP OPERATIONS

1. Roll-up: Summarizes data to a higher level.

​

• Drill-down: Shows more detailed data.

​

• Slice/Dice: Selects specific parts of data for focused analysis.

​

• Pivot (Rotate): Changes the view to get new perspectives.

​

• OLAP operations provide flexibility for exploring and visualizing data.

SUMMARY

Data Mining discovers patterns and hidden knowledge from data.

Data Warehouses store historical, integrated, and cleaned data. OLAP allows interactive, multidimensional analysis.

Together, they support better and faster decision-making. These systems form the foundation of Business Intelligence (BI).

THANK YOU