UNIT 4 – Data Warehousing, Data Mining & Information Retrieval

Prepared By: Mr. Akshay D. Thorat

Database-System Architectures

• Centralized Systems
• Single server, single database
• Simple management, low cost

​

• Client–Server Systems
• Two-tier & three-tier architectures
• Clients send queries → server processes
• Parallel Systems

​

• Multiple processors for high performance
• Shared memory, shared disk, shared nothing
• Distributed Systems

​

• Data stored across multiple sites
• Supports replication & fragmentation

Centralized vs Distributed Databases

​

• Centralized:
• Easy to maintain
• Single point of failure

​

• Distributed:
• Improved reliability
• Faster local access
• Complex synchronization

​

• Used in large-scale enterprise applications

Introduction to Data Warehousing

​

• Subject-oriented

​

• Integrated

​

• Time-variant

​

• Non-volatile

​

• Stores historical data for analysis

​

• Supports decision-making processes

Data Warehouse Architecture

​

• Data Sources: Operational DBs, external data

​

• ETL Process: Extract → Transform → Load

​

• Data Storage: Warehouse repository

​

• OLAP Engine: Analytical processing

​

• Front-End Tools: Dashboards, reports, BI tools

ETL (Extract, Transform, Load)

​

• Extract: Collect data from multiple sources

​

• Transform: Clean, filter, aggregate

​

• Load: Insert into warehouse

​

• Ensures data consistency and quality

OLAP (Online Analytical Processing)

​

• Supports complex analytical queries

​

• Types:
• ROLAP: Relational OLAP

​

• MOLAP: Multidimensional OLAP

​

• HOLAP: Hybrid OLAP

​

• Operations: Slice, Dice, Roll-up, Drill-down, Pivot

Data Mining – Introduction

​

• Process of discovering patterns from large datasets

​

• Uses statistical, machine learning, and AI techniques

​

• Applications: Fraud detection, recommendation systems, marketing, healthcare

Knowledge Discovery in Databases (KDD)

Steps:

1. Data Cleaning

​

• Data Integration

​

• Data Selection

​

• Data Transformation

​

• Data Mining

​

• Pattern Evaluation

​

• Knowledge Presentation

Classification Techniques

​

• Predict categorical labels

​

• Algorithms:
• Decision Trees
• Naïve Bayes
• K-Nearest Neighbors
• Support Vector Machines
• Neural Networks

​

• Used in spam detection, diagnosis, credit scoring

Clustering Techniques

​

• Group similar data objects

​

• Unsupervised learning

​

• Algorithms:
• K-Means
• Hierarchical Clustering
• DBSCAN

​

• Used in customer segmentation, anomaly detection

Association Rule Mining

​

• Discovers relationships among items

​

• Example: Market Basket Analysis
• Measures:
• Support
• Confidence
• Lift

​

• Algorithms: Apriori, FP-Growth

Other Data Mining Methods

​

• Regression Analysis

​

• Outlier Detection

​

• Text Mining

​

• Web Mining

​

• Time-Series Analysis

​

• Used in forecasting and trend analysis

Information Retrieval (IR) – Introduction

​

• Process of obtaining relevant information from large collections

​

• Used in search engines, document retrieval, digital libraries

​

• Focuses on unstructured data (text)

IR System Components

​

• Document collection

​

• Indexing engine

​

• Query processor

​

• Ranking algorithm

​

• User interface

​

• Uses inverted index for fast search

Indexing & Searching

​

• Inverted Index: Maps terms → documents

​

• Tokenization: Breaking text into words

​

• Stemming: Reducing words to root form

​

• Stop-word Removal: Removing common words
• Improves search efficiency

Ranking & Relevance

​

• Ranking algorithms:
• TF-IDF
• BM25
• PageRank

​

• Relevance feedback improves accuracy

​

• Used in modern search engines

Applications of Data Warehousing & Mining

​

• Business Intelligence

​

• Fraud Detection

​

• Customer Behaviour Analysis

​

• Healthcare Diagnostics

​

• Social Media Analytics

​

• Scientific Research

​

• Recommendation Systems (Netflix, Amazon)