Data Warehouse Implementation and Data Mining
:
MODULE 1
1
4/19/2022
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Data Mining
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Needs of Data Mining
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Integrating Heterogeneous Databases
To integrate heterogeneous databases, we have two approaches −
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Query-Driven Approach
This is the traditional approach to integrate heterogeneous databases. This approach was used to build wrappers and integrators on top of multiple heterogeneous databases. These integrators are also known as mediators.
Process of Query-Driven Approach
Disadvantages
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Update Driven Approach
In Update driven approach the information from multiple heterogeneous sources is integrated in advance and stored in a warehouse. This information is available for direct querying and analysis.
ADVANTAGES
This approach has the following advantages: -
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
| ROLAP | MOLAP |
1. | ROLAP stands for Relational Online Analytical Processing. | While MOLAP stands for Multidimensional Online Analytical Processing. |
2. | ROLAP is used for large data volumes. | While it is used for limited data volumes. |
3. | The access of ROLAP is slow. | While the access of MOLAP is fast. |
4. | In ROLAP, Data is stored in relation tables. | While in MOLAP, Data is stored in multidimensional array. |
5. | In ROLAP, Data is fetched from data-warehouse. | While in MOLAP, Data is fetched from MDDBs database. |
6. | In ROLAP, Complicated sql queries are used. | While in MOLAP, Sparse matrix is used. |
7. | In ROLAP, Static multidimensional view of data is created. | While in MOLAP, Dynamic multidimensional view of data is created. |
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
34
4/19/2022
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Data Cube: A multidimensional data model What is
39
Data Warehousing and On-line Analytical Processing
• Facts?
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Example
40
May 18, 2020
Data Mining: Concepts and Techniques
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Data Warehousing and On-line Analytical Processing
41
May 18, 2020
Data Mining: Concepts and Techniques
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Example
42
May 18, 2020
Data Mining: Concepts and Techniques
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
43
May 18, 2020
Data Mining: Concepts and Techniques
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
44
May 18, 2020
Data Mining: Concepts and Techniques
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
45
May 18, 2020
Data Mining: Concepts and Techniques
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
46
May 18, 2020
Data Mining: Concepts and Techniques
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
From Tables and Spreadsheets to Data Cubes
47
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Cube: A Lattice of Cuboids
48
time,item
time,item,location
time, item, location, supplier
all
time
item
location
supplier
time,supplier
time,location item,location
item,supplier
time,item,supplier
time,location,supplier
item,location,supplier
location,supplier
2-D cuboids
0-D (apex) cuboid
1-D cuboids
3-D cuboids
4-D (base) cuboid
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Conceptual Modeling of Data Warehouses
49
May 18, 2020
Data Mining: Concepts and Techniques
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
50
May 18, 2020
Data Mining: Concepts and Techniques
Star schema
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
51
4/19/2022
time
time_key day
day_of_the_week
month quarter year
branch
branch_key branch_name branch_type
time_key |
item_key |
branch_key |
location_key |
|
units_sold |
dollars_sold |
avg_sales |
Measures
location_key street
city state_or_province country
item
item_key item_name brand
type supplier_type
location
Sales Fact Table
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
May 18, 2020
Data Mining: Concepts and Techniques
29
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Example of Snowflake Schema
30
time_key day
day_of_the_week
month quarter year
time
location_key street city_key
location
Sales Fact Table
time_key
item_key
branch_key
location_key
units_sold
dollars_sold
avg_sales
Measures
item_key item_name brand
type
supplier_key
item
branch_key branch_name branch_type
branch
supplier | |
| |
supplier_key supplier_type | |
city_key city
state_or_province country
city
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
54
Data Mining: Concepts and Techniques
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
55
May 18, 2020
Data Mining: Concepts and Techniques
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Example of Fact Constellation
3
time_key day
day_of_the_week month
quarter
year
time
location_key street
city province_or_state country
location
Sales Fact Table
|
time_key |
|
item_key |
branch_key |
|
location_key |
|
|
units_sold |
dollars_sold |
|
avg_sales |
|
Measures
item_key item_name brand
type supplier_type
item
branch_key branch_name branch_type
branch
Shipping Fact Table
time_key
item_key
shipper_key
from_location
to_location
dollars_cost
units_shipped
shipper_key shipper_name location_key shipper_type 3
shipper
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
May 18, 2020
Data Mining: Concepts and Techniques
34
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Dimensions: The role of concept hierarchies
What is Concept Hierarchy? A concept hierarchy defines a sequence of mappings from a set of low-level concepts to higher-level, more general concepts.
Total Order : Street<city<province_or_state<country
Partial Order: day<{month<quarteer;week}<year
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Set Grouping Hierarchy
A concept hierarchy that is a total or partial order among attributes in a database schema is called a schema hierarchy
May 18, 2020
Data Mining: Concepts and Techniques
38
Concept hierarchies may also be defined by discretizing or grouping values for a given dimension or attribute, resulting in a set-grouping hierarchy
There may be more than one concept hierarchy for a given attribute or dimension, based on different user viewpoints. For instance, a user may prefer to organize price by defining ranges for inexpensive, moderately priced, and expensive.
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Multidimensional Data
62
Product
Dimensions: Product, Location, Time
Hierarchical summarization paths
Industry Region Year
Category Country Quarter Product City Month Week
Office Day
Month
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
A Sample Data Cube
63
Total annual sales
of TVs in U.S.A.
Country
sum
TV
PC VCR
sum
1Qtr
2Qtr
Date
3Qtr
4Qtr
U.S.A
Canada
Mexico
sum
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Cuboids Corresponding to the Cube
64
all
product
date
country
product,date
product,country
date, country
product, date, country
0-D (apex) cuboid
1-D cuboids
2-D cuboids
3-D (base) cuboid
Measures: Their Categorization and Computation
What is measure? Categories are:
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
What is Measure?
May 18, 2020
Data Mining: Concepts and Techniques
43
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Data Cube Measures: Three Categories
44
Distributive: An aggregate function is distributive if it can be computed in a distributed manner as follows:
distributive aggregate function.
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
May 18, 2020
Data Mining: Concepts and Techniques
45
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Data Cube Measures: Three Categories
69
Algebraic: An aggregate function is algebraic if it can be computed by an algebraic function with M arguments (where M is a bounded positive integer), each of which is obtained by applying a distributive aggregate function.
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Data Cube Measures: Three Categories
70
Holistic: An aggregate function is holistic if there is no constant bound on the storage size needed to describe a subaggregate.
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Typical OLAP Operations
“How are concept hierarchies useful in OLAP?”
In the multidimensional model, data are organized into multiple dimensions, and each dimension contains multiple levels of abstraction defined by concept hierarchies.
48
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Typical OLAP Operations
72
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Typical OLAP Operations
74
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Typical OLAP Operations
76
= “Toronto” or “Vancouver”) and (time = “Q1” or “Q2”) and (item = “home entertainment” or “computer”).
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
Typical OLAP Operations
78
back-end relational tables (using SQL)
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
/skit.org.in
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in
80
Fig. 3.10 Typical OLAP Operations
Sri Krishna Institute of Technology
(Approved by AICTE, Accredited by NAAC, Affiliated to VTU, Karnataka)
/skit.org.in