Big Data Analytics

Department: Computer Science & Engineering

Semester: VII

Course Code: 23CSPC402

Teaching Hrs/Week (L:T: P): 3:0:2

Total Hrs: 64(40+24)

CIE Marks: 50

SEE Marks: 50

Exam Hrs: 3

Credits: 4

Course Learning Objectives

This course is designed to:

1. Provide knowledge on characteristics of big data.
2. Impart knowledge on Hadoop framework and MapReduce programming model.
3. Familiarize MongoDB, Cassandra, Hive ,Pig and Spark used in big data analytics.
4. Provide knowledge on big data analytics using machine learning techniques.

Module 1: �Introduction to Big Data Analytics

1. Classification of Digital data.
2. Characteristics of Data.
3. Challenges with big data.
4. Typical data warehouse.
5. Classification of Analytics
6. Importance of Big Data Analytics.
7. Terminologies used in Big data Environments, NoSQL and Hadoop.

Text book 1: 1.1,2.1 to 2.5, 2.10, 3.2, 3.5, 3.8, 3.12, 4.1, 4.2

1.1 Classification of Digital data

Structured Data

Semi-Structured Data

Unstructured Data

​

2.1 Characteristics of Data

​

2.2 Evolution of Big Data

​

2.3 Definition of Big Data

​

2.4 Challenges with Big Data

​

2.5 What is Big Data

2.5.1 Volume

2.5.2 Velocity

2.5.3 Variety

2.10 A Typical Data Warehouse Environment

​

3.2 Big Data Analytics

​

3.5 Classification of Analytics

​

3.8 Why is Big Data Analytics Important

​

3.12 Terminologies used in Big Data Environments

3.12.1 In-Memory Analytics

3.12.2 In-Database Processing

3.12.3 Symmetric Multiprocessor System(SMP)

3.12.4 Massively Parallel Processing

3.12.5 Difference Between Parallel and Distributed Systems

3.12.6 Shared Nothing Architecture

​

4.1 NoSQL (Not Only SQL)

4.1.1 Where is it Used?

4.1.2 What is it?

4.1.3 Types of NoSQL Databases

4.1.4 Why NoSQL?

4.1.5 Advantages of NoSQL

4.1.6 What We Miss With NoSQL?

4.1.7 Use of NoSQL in Industry

4.1.8 NoSQL Vendors

4.1.9 SQL versus NoSQL

4.1.10 NewSQL

4.1.11 Comparison of SQL, NoSQL, and NewSQL

​

​

4.2 Hadoop

4.2.1 Features of Hadoop

4.2.2 Key Advantages of Hadoop

4.2.3 Versions of Hadoop

4.2.4 Overview of Hadoop Ecosystems

4.2.5 Hadoop Distributions

4.2.6 Hadoop versus SQL

4.2.7 Integrated Hadoop Systems Offered by Leading Market Vendors

4.2.8 Cloud-Based Hadoop Solutions

​

​

What is Data

• Data is a plural of Datum, a collection of facts, and numbers,
• quantitative (Numerical, measurable data used for statistical and mathematical analysis (e.g., sales, marks, sensor readings))
• qualitative (Descriptive, non-numerical data representing opinions, experiences, text, images, audio, or video, analyzed using techniques such as text mining and sentiment analysis), digitally stored.
• The messages you send on WhatsApp, tweets, and posts, are all Data.
• The emails you receive, are also data.

​

How Information is Extracted

• Information is extracted from Data. When the data is processed, what we get is called information.
• The processing includes organizing the raw and unorganized data. This makes it meaningful and can be further used.

Introduction��

1.1 Classification of Digital data

1. What comes to mind when you hear the term "digital data"? give examples from your daily life?
2. Do you think all data looks the same, or are there different "types" of data?
3. When you save a photo vs. a Word document vs. a tweet are these stored/structured the same way?
4. Have you heard the terms "structured," "semi-structured," or "unstructured" data before?

​

Introduction

1.1 Classification of Digital data.

Digital data is classified into three categories based on its organization:

1. Structured Data
2. Semi-Structured Data
3. Unstructured Data

​

Structured Data

• Structured data is data that is organized in a standardized and predefined format, making it easy for both computers and humans to access, store, and process.
• It follows a fixed schema and is usually stored in the form of tables consisting of rows and columns, where each column represents an attribute and each row represents a record.
• Because of its organized and quantitative nature, structured data can be efficiently processed using database management systems and SQL to generate meaningful insights.

​

​

Example: Customer table, From this structured data, we can easily determine:

1. Total number of customers.
2. Number of customers in each locality.
3. Customers belonging to a particular city.
4. Customer contact details.
5. Other business insights using SQL queries.

​

​

​

Characteristics of structured data

1. Organised in predefined schemas.
2. Can be stored in rows and columns.
3. Searchable and analysable.
4. Uses relational databases for storage.

​

Examples of structured data

1. Product or customer directories.
2. Inventory control tracking sheets.
3. Web form results.
4. Point-of-sale data.
5. Financial records.

​

​

​

Benefits of structured data

1. Simplifies data querying and reporting(Tool like SQL).
2. Supports robust analytics and insights (business intelligence (BI) tools).
3. Ensures high data accuracy(database rules).
4. Integrates easily with existing systems (Example: CRM).

​

Challenges of Structured data

1. Limited to predefined formats.
2. Lacks flexibility for unstructured inputs.
3. Can't scale well with large datasets.

​

​

​

​

​

Sources of Structured Data

1. Relational Databases (RDBMS).
2. Enterprise Business Applications (ERP, CRM, HRMS).
3. Banking Systems.
4. E-Commerce Systems.
5. Educational Institutions.
6. Hospital Management Systems.
7. Government Databases (Aadhaar, Passport).
8. Inventory Management Systems.

​

Ease of Working with Structured Data

Structured data provides the ease of working with respect to the following:

1. Insert / Update / Delete

The Data Manipulation Language (DML) operations provide the required ease with: Data input, Data storage, Data access, Data processing, Data analysis.

2. Security:

• Ensures the security of information.
• Encryption and tokenization solutions protect information throughout its lifecycle.
• Organizations can: Retain control over data, Maintain compliance, Ensure that only authorized individuals can decrypt and view sensitive information.

3. Indexing

• An index is a data structure that speeds up data retrieval operations.
• Primarily improves the performance of SELECT statements.
• Requires: Additional write operations, Extra storage space.

​

​

4. Scalability

• Traditional RDBMS storage and processing capabilities can be scaled up.
• Scalability is achieved by increasing the database server's resources, such as: Primary storage capacity, Secondary/peripheral storage capacity, Processor capacity.

5. Transaction Processing: RDBMS supports ACID properties of transactions:

• Atomicity: A transaction is atomic.
• Consistency: The database moves from one consistent state to another consistent state.
• Isolation: Resource allocation ensures that each transaction appears to execute independently.
• Durability: All changes made during a transaction are permanent.

​

​

​

​

Un-Structured Data

• Unstructured data is business information that organizations generate and receive every day, such as videos, images, chat messages, social media posts, and other real-world content.
• Unstructured data doesn’t easily fit into predefined tables or standardized formats, and cannot be organized, indexed, or queried effectively in a legacy relational database management system (RDBMS).
• Unstructured data now represents 80-90% of all organizational data.

​

​

Unstructured data is found in:

1. Customer interactions: Emails, chats, reviews, user-generated content, and support tickets.
2. Business documents: Notes, PDFs, and slide decks.
3. Digital forms: Web pages, social posts, mixed multimedia files.
4. Real-time data: Logs, sensor data, and IoT output.
5. Collaboration tools: Messages, recordings, and transcripts.

​

Examples of unstructured data

1. Human-generated: Many of the files people create every day include qualitative data and natural language that don't fit neatly into tables, including:

1. Customer emails, chats, and written feedback.
2. Word processing documents, notes, PDFs, and text files.
3. Audio files and video files from meetings or collaboration.
4. Social media posts and social media comments.
5. Mixed rich media assets that combine text, images, and video.

Example: In healthcare, a single patient visit might generate typed notes, a scanned form, and an audio dictation file and these three are different kinds of unstructured data.

​

2. Machine-generated

Systems and devices produce large volumes of unstructured output, such as:

1. Sensor readings and IoT data.
2. Application and system logs.
3. Surveillance footage and monitoring streams.
4. Scientific outputs such as satellite imagery.
5. Hybrid datasets that mix numeric values with descriptive text.

Example: In manufacturing, one machine might produce short error codes and long diagnostic logs with different formats and too unpredictable for relational tables.

​

Example 1: Steps to Deal with Unstructured Data

1. Data Collection: collect unstructured data from various sources (Emails, Websites).
2. Data Storage: Hadoop Distributed File System (HDFS)
3. Data Preprocessing: Before analysis, it must be cleaned and transformed.

Example: Image Data: Resize images, Remove noise, Image enhancement.

• Data Processing: Big Data frameworks process large volumes of unstructured data and Common tools are Apache Hadoop, Spark, Hive, Pig.

​

​

5. Data Analysis: After preprocessing, different analytical techniques are applied like

• Text Analytics: Analyzes text documents.
• Image Analytics: Analyzes digital images.
• Video Analytics: Processes video streams.
• Audio Analytics: Processes speech and sound.

6. Machine Learning and Artificial Intelligence

Machine Learning algorithms help extract useful patterns from unstructured data.

Examples: Naive Bayes (Text Classification), Logistic Regression, Support Vector Machine (SVM).

​

​

​

​

​

​

​

7. Natural Language Processing (NLP): NLP enables computers to understand human language.

Applications: Chatbots, Language Translation

Example: Analyzing customer reviews to determine whether feedback is positive or negative.

8. Data Visualization: The processed information is presented through dashboards and reports.

Common tools: Power BI, Tableau, Grafana.

​

Example 2: Steps to Deal with Unstructured Data

Techniques Used to Find Patterns in or Interpret Unstructured Data

1. Data Mining: Deals with large data sets.

• Uses methods from: Artificial Intelligence (AI), Machine Learning (ML), Statistics, Database Systems
• Identifies: Consistent patterns in large data sets, Systematic relationships between variables.
• It is the analysis step of the Knowledge Discovery in Databases (KDD) process.

Popular Data Mining Algorithms

a) Association Rule Mining: Also called Market Basket Analysis, Affinity Analysis

Determines "What goes with what?"

Identifies products that are likely to be purchased together.

Example: If a customer buys bread, they may also buy eggs or cheese.

​

​

​

​

​

b) Regression Analysis:

• Predicts the relationship between two variables.
• Dependent Variable: The variable whose value needs to be predicted.
• Independent Variables: Variables used to predict the dependent variable.

c) Collaborative Filtering:

• Predicts a user's preferences based on the preferences of similar users.
• Analyzes user profiles to make recommendations.
• Example: Predicting whether User 4 prefers learning through videos or text by analyzing similar users' preferences.

2. Text Analytics (Text Mining)

• Text data is largely: Unstructured, Amorphous, Difficult to process algorithmically
• Text mining extracts meaningful and high-quality information.
• Uses statistical pattern learning to discover: Patterns, Trends
• Common tasks include: Text categorization, Text clustering, Sentiment analysis, Concept/Entity extraction

​

3. Natural Language Processing (NLP)

• Related to human-computer interaction.
• Enables computers to understand human or natural language.

4. Noisy Text Analytics

• Extracts structured or semi-structured information from noisy unstructured data.
• Sources include: Chats, Blogs, Wikis, Emails, Message boards, Text messages
• Noisy text may contain: Spelling mistakes, Abbreviations, Acronyms, Non-standard words, Missing punctuation, Missing letter case, Filler words such as "uh", "um", etc.

5. Manual Tagging with Metadata

• Involves manually tagging data with appropriate metadata.
• Provides the required semantics to understand unstructured data.

​

6. Part-of-Speech (POS) Tagging

• Also called: POS Tagging, POST, Grammatical Tagging
• Assigns each word in a sentence to its grammatical category.
• Examples of parts of speech: Noun, Verb, Adjective.

7. Unstructured Information Management Architecture (UIMA)

• An open-source platform developed by IBM.
• Used for real-time content analytics.
• Processes text and other unstructured data.
• Finds: Latent meaning, Relevant relationships hidden within the data.

​

​

Semi-structured data

• Is a type of data that lies between structured data and unstructured data.
• It does not follow a fixed schema like structured data, but it contains tags, keys, metadata, that define relationships and hierarchical organization and these elements make the data easier to store, search, and process than unstructured data.
• Semi-structured data is well suited for modern applications such as web services, APIs, Internet of Things (IoT) devices, cloud applications, and social media platforms.

​

Characteristics of semi-structured data

1. Schema: Dynamically changing structure without requiring modification to a rigid schema
2. Format: Often represented in key-value pairs, nested objects or arrays
3. Storage and processing: Can be processed without strict formatting requirements

Examples: JSON, XML, Avro, Parquet, ORC, and data from web applications and IoT sensors

​

Sources of Semi-Structured Data

REST APIs, IoT devices and sensors, Mobile applications, Social media platforms, Cloud services, Web applications, Email systems, Web server logs.

Advantages

1. Flexible schema.
2. Easy to expand by adding new fields.
3. Handles hierarchical and nested data efficiently.
4. Better suited for Big Data than traditional relational databases.
5. Scales well for cloud and distributed environments.

Limitations

1. Complex relationships can be harder to manage.
2. Standard SQL cannot be directly applied in many cases.

​

​

​

​

​

Example 1: JSON

{

"StudentID": 101,

"Name": “Shivu",

"Department": "CSE",

"Skills": ["Java", "Python", "SQL"]

}

Organized using key-value pairs. Different students can have different fields. No fixed schema.

Example 2: XML

<Teacher>

<StudentID>101</StudentID>

<Name>Shivu</Name>

<Department>CSE</Department>

</Teacher>

Data is organized using tags. Easy to exchange data between applications.

​

Questions

1. Explain the differences between structured, semi-structured, and unstructured data with respect to schema, querying techniques, and use cases. Support your answer with suitable examples.
2. Compare structured, semi-structured, and unstructured data by discussing their schema, querying methods, and real-world applications.

2.1 CHARACTERISTICS OF DATA

1. What makes data "good" or "useful" for analysis?
2. If data has errors or is incomplete, what problems could that cause for a business?
3. Can you think of a time when bad or messy data led to a wrong decision?
4. What qualities would you want data to have before trusting it?

​

​

2.1 CHARACTERISTICS OF DATA

​

1. Composition: The composition of data deals with the structure of data, that is, the sources of data, the granularity, the types, and the nature of data as to whether it is static or real-time streaming.
2. Condition: The condition of data deals with the state of data, that is, “Can one use this data as is for analysis?” or “Does it require cleansing for further enhancement and enrichment?”
3. Context: The context of data deals with “Where has this data been generated?” “Why was this data generated?” “How sensitive is this data?” “What are the events associated with this data?”

1. Composition (Structure of Data): Composition tells us how the data is organized or made up.

It answers questions such as:

• What is the source of the data?
• What type of data is it?
• Is it structured, semi-structured, or unstructured?
• Is the data static or real-time?

Example: A student database contains: Student ID, Name, Department, Marks.

Here, the composition describes the structure and organization of these fields.

​

2. Condition (State or Quality of Data): Condition tells us whether the data is ready for analysis or needs improvement.

It answers questions such as:

• Can this data be used as it is?
• Does it contain missing values?
• Are there duplicate records?
• Does it need cleaning?

3. Context (Background of Data): Context explains the meaning and background of the data.

It answers questions such as:

• Where was the data generated?
• Why was it collected?
• Who collected it?
• When was it collected?
• Is the data confidential?

​

2.2 Evolution of Big Data: The major stages are

1. Traditional Data Era (Before 2000): Small, structured data stored in RDBMS.
2. Internet Era (2000–2005): Rapid increase in web and e-commerce data.
3. Social Media Era (2005–2010): Growth of unstructured data such as images, videos, and posts.
4. Big Data Era (2010–Present): Adoption of Hadoop, Spark, NoSQL, and cloud computing for large-scale data processing.
5. AI and Real-Time Analytics Era (Present & Future): Use of AI, IoT, and machine learning for real-time analysis and intelligent decision-making.

​

​

2.3 Definition of Big Data

• Big data refers to extremely large and complex data sets that cannot be easily managed or analyzed with traditional data processing tools, particularly spreadsheets.
• Big data includes structured data, like an inventory database or list of financial transactions; unstructured data, such as social posts or videos; and mixed data sets, like those used to train large language models for AI. 
• "Big data refers to extremely large and complex data sets that cannot be easily managed or analyzed with traditional data processing tools."

​

:Data Evolution:

1. Ancient times: Small amounts of handwritten records.
2. Computer era: Kilobytes (KB) and Megabytes (MB) of structured data.
3. Internet era: Gigabytes (GB) and Terabytes (TB) due to websites, emails, and online transactions.
4. Social media era: Petabytes (PB) from photos, videos, and user-generated content.
5. Modern AI and IoT era: Exabytes (EB) and Zettabytes (ZB) of structured, semi-structured, and unstructured data generated continuously.

As the volume, complexity, and diversity of data grew, traditional tools such as spreadsheets and conventional databases could no longer store or analyze it efficiently.

This need led to the development of Big Data technologies like Hadoop, Spark, NoSQL databases, and cloud computing.

​

Data Measurements From The Smallest Units To The Largest

1. Bit (b) : 1 bit = 1 or 0.
2. Byte (B) : 1 Byte = 8 bits.
3. Kilobyte (KB) : 1 KB = 1,024 Bytes.
4. Megabyte (MB) : 1 MB = 1,024 KB = 1,048,576 Bytes.
5. Gigabyte (GB) : 1 GB = 1,024 MB = 1,073,741,824 Bytes.
6. Terabyte (TB) : 1 TB = 1,024 GB = 1,099,511,627,776 Bytes.
7. Petabyte (PB) : 1 PB = 1,024 TB = 1,125,899,906,842,624 Bytes.
8. Exabyte (EB) : 1 EB = 1,024 PB = 1,152,921,504,606,846,976 Bytes.
9. Zettabyte (ZB) : 1 ZB = 1,024 EB = 1,180,591,620,717,411,303,424 Bytes.
10. Yottabyte (YB) : 1 YB = 1,024 ZB = 1,208,925,819,614,629,174,706,176 Bytes.

​

2.4 Challenges with Big Data

1. Why do you think handling large amounts of data is harder than handling small amounts?
2. What problems might arise when data is coming in from thousands of sources at once?
3. Can traditional databases (like Excel or MySQL) handle terabytes of data efficiently? Why or why not?
4. What do you think happens when data arrives faster than a system can process it?

​

​

2.4 Challenges with Big Data

Challenges with Big Data are the difficulties organizations face while managing large and complex datasets.

The major challenges include:

1. Capture: Collecting data from multiple sources such as sensors, social media, and websites.
2. Storage: Storing massive volumes of data efficiently.
3. Curation: Cleaning, organizing, and maintaining data quality.
4. Search: Retrieving relevant information quickly from large datasets.
5. Analysis: Processing data to discover meaningful patterns and insights.
6. Transfer: Moving large datasets between systems or cloud environments.
7. Visualization: Presenting complex data in graphical and understandable forms.
8. Privacy Violations: Protecting sensitive information from unauthorized access and ensuring data security.

​

​

2.5 What are the Five “Vs” of Big Data?

​

The Five V's of Big Data describe the key characteristics of Big Data.

• Volume: Refers to the large amount of data generated and stored by organizations.
• Velocity: Refers to the speed at which data is generated, transmitted, and processed.
• Variety: Refers to the different forms of data, including structured, semi-structured, and unstructured data.
• Veracity: Refers to the quality, accuracy, and reliability of data used for analysis.
• Value: Refers to the meaningful insights and business benefits obtained by analyzing data.

​

​

1. Volume refers to the amount of data generated, collected, and stored.

• Big Data involves enormous quantities of data.
• Data may be structured, semi-structured, or unstructured.
• Organizations generate data in terabytes (TB), petabytes (PB), and even exabytes (EB).

Examples

• Facebook stores billions of photos.
• YouTube stores millions of videos.
• Banks store millions of daily transactions.
• E-commerce websites store customer purchase records.

​

2. Velocity refers to the speed at which data is generated, collected, processed, and analyzed.

• Data is generated continuously.
• Some applications require real-time or near real-time processing.
• Faster processing enables quick decision-making.

Examples

• Credit card fraud detection.
• Stock market trading.
• GPS navigation.
• Social media feeds.
• IoT sensor data.

​

3. Variety refers to the different types and formats of data.

• Big Data consists of: Structured data, Semi-structured data, Unstructured data.

Examples

• Structured: Customer database, Bank transaction table
• Semi-Structured: XML, JSON, Emails
• Unstructured: Images, Videos, Audio files, Social media posts

​

4. Veracity refers to the accuracy, quality, and reliability of data.

• Data should be correct and trustworthy.
• Poor-quality data can lead to incorrect decisions.
• Organizations must clean and validate data before analysis.

Examples

• Duplicate customer records.
• Missing values.
• Incorrect addresses.
• Fake social media accounts.

​

5. Value refers to the usefulness of data in generating meaningful insights and business benefits.

• Data itself has little value unless it is analyzed.
• Organizations use data to improve operations, understand customers, and support decision-making.

Examples

• Netflix recommends movies based on viewing history.
• Amazon suggests products based on purchase behavior.
• Hospitals analyze patient data to improve healthcare.
• Companies optimize operations using sales data.

​

​

2.10 A Typical Data Warehouse Environment

A classic data warehouse architecture diagram, showing how data flows into and out of a central data warehouse.

​

"Data warehouse" the central repository where data from all these disparate sources is consolidated, cleaned, and stored in a structured format for analysis.

Below are the operational systems where raw business data originates.

• ERP (Enterprise Resource Planning systems)
• CRM (Customer Relationship Management systems)
• Legacy (older/legacy systems)
• Third party apps

Consumers / uses of the Data Warehouse data:

• Reporting / Dashboarding: standard business reports and visual dashboards
• OLAP (Online Analytical Processing): multidimensional analysis
• Ad hoc querying: Exploratory queries by analysts
• Modelling: statistical or predictive modeling /data science work

3.2 Big Data Analytics

Big Data Analytics is the process of examining large, complex datasets often too big or fast-moving for traditional tools to uncover patterns, trends, correlations, and insights that support decision-making.

The 5 V’s (Volume, Velocity, Variety, Veracity, Value ) usually the foundation of Big Data Analytics.

Types of Big Data Analytics

• Descriptive Analytics: What happened? (dashboards, reports)
• Diagnostic Analytics: Why did it happen? (root-cause analysis)
• Predictive Analytics: What will happen? (forecasting, ML models)
• Prescriptive Analytics: What should we do? (optimization, recommendations)

Common Tools & Technologies

• Storage/Processing: Hadoop (HDFS, MapReduce), Apache Spark
• Databases: NoSQL (MongoDB, Cassandra), Hive
• Streaming: Apache Kafka, Spark Streaming, Flink
• Visualization: Tableau, Power BI
• ML/Analytics: Python (Pandas, PySpark), R

​

​

​

3.5 Classification of Analytics

• First school of thought
• Second school of thought

First School of Thought Definition:

The first school (Basic : Operationalized : Advanced : Monetized) classifies analytics by business maturity/integration — how deeply it's used and whether it makes money..

2. Basic Analytics: Slicing/dicing data; historical reporting & visualization; answers "what happened."
3. Operationalized Analytics: Analytics embedded into live business processes/workflows (not a standalone report).
4. Advanced Analytics: Forecasting the future via predictive (what will happen) and prescriptive (what should be done) modeling.
5. Monetized Analytics: Analytics used to directly generate business revenue (sold/licensed as a product).

​

The second school of Thought Definition

(Descriptive : Predictive : Prescriptive) classifies analytics by the nature of the insight / question regardless of how it's deployed in the business.

1. The traditional analytics era 1.0 (mid-1950s – 2009)
• Organizations only had descriptive statistics reporting on what already happened.
• Key question: What happened? Why did it happen?
• Data: Small, structured, from internal legacy systems (ERP, CRM, 3rd-party apps)
• Storage: Data warehouses/marts
• Technology: Relational databases

​

2. Analytics 2.0 (2005 – 2012) The rise of big data.

• This era added predictive statistics on top of descriptive using historical data to forecast the future.
• Key question: What will happen? Why will it happen?
• Data: Big data — mostly unstructured, arriving fast and in huge volumes
• Sourcing: Increasingly external (social media, web logs, sensors) rather than just internal systems
• Technology: This is where Hadoop and massive parallel processing entered needed because traditional relational databases couldn't handle the volume/speed/variety of this new data.

​

3. Analytics 3.0 (2012 – present)

• The convergence era combining everything before it plus prescriptive statistics (recommending actions, not just predicting outcomes).
• Key questions: What will happen? When? Why? What should we do about it?
• Data: A blend of big data + traditional legacy/ERP/CRM data best of both worlds
• Sourcing: Both internal and external
• Technology: More sophisticated in-memory analytics, machine learning, agile analytical methods built for speed and real business impact, not just processing scale.

​

3.8 Why is Big Data Analytics Important?

The four approaches to data analysis and what each leads to:

1. Reactive – Business Intelligence (BI)

2. Reactive – Big Data Analytics

3. Proactive – Analytics

4. Proactive – Big Data Analytics

Reactive (Approaches 1 & 2)

• Reactive means you analyze data after events have already happened.
• Both BI and "Big Data Analytics" in their reactive form work on static/historical data.

Proactive (Approaches 3 & 4)

• Proactive means the analysis is meant to support futuristic decision making.

​

Reactive Business Intelligence:

• Helps businesses make faster and better decisions by providing the right information to the right person at the right time in the right format.
• It focuses on analysis of past/historical data, with findings displayed through enterprise dashboards, alerts, and notifications.
• It supports both pre-specified reports and ad hoc querying.

Reactive Big Data Analytics:

• Analysis is done on huge datasets, but the approach remains reactive since it is still based on static data.

Proactive Analytics:

• Supports futuristic decision making using data mining, predictive modeling, text mining, and statistical analysis.
• However, this is not true big data analysis, as it still relies on traditional database management practices leading to severe limitations in storage capacity and processing capability.

Proactive Big Data Analytics:

• Involves massive data like terabytes, petabytes, and exabytes of information to filter out relevant data for analysis.
• It also includes high-performance analytics to gain rapid insights from big data and the ability to solve complex problems using more data.

3.12 Terminologies used in Big Data Environments

3.12.1 In-Memory Analytics

3.12.2 In-Database Processing

3.12.3 Symmetric Multiprocessor System(SMP)

3.12.4 Massively Parallel Processing

3.12.5 Difference Between Parallel and Distributed Systems

3.12.6 Shared Nothing Architecture

​

3.12.1 In-Memory Analytics

The Problem:

• Accessing data from non-volatile storage (like hard disks) is slow.
• The more data that needs to be fetched from hard disk/secondary storage, the slower the process becomes.

Limitation:

• Pre-process and store data in advance (e.g., cubes, aggregate tables, query sets) so the CPU only needs to fetch a small subset of records.

Drawback: This requires predicting in advance what data will be needed.

Solution: In-Memory Analytics

• All relevant data is stored directly in RAM (Random Access Memory) / primary storage, eliminating the need to access hard disk at all.
• Advantages: Faster access, Rapid deployment, Better insights, Minimal IT involvement

​

3.12.2 In-Database Processing (In-database analytics )

How it works: It combines data warehouses with analytical systems.

The Traditional Process (without in-database processing):

• Data comes from various enterprise OLTP (On Line Transaction Processing) systems.
• This data goes through ETL (Extract, Transform, Load) including cleaning steps like de-duplication and scrubbing.
• The cleaned data is stored in the Enterprise Data Warehouse (EDW).
• The huge datasets are then exported to separate analytical programs for complex, extensive computations.

The Improvement: In-Database Processing

• Instead of exporting data out to a separate analytical program, the database program itself runs the computations. This eliminates the need for exporting data, thereby saving time.

Adoption: Leading database vendors offer this feature to large businesses.

3.12.3 Symmetric Multiprocessor System (SMP)

Definition: SMP has a single common main memory that is shared by two or more identical processors.

Key characteristics:

• All processors have full access to all I/O devices.
• The entire system is controlled by a single operating system instance.

Architecture:

• SMP systems are tightly coupled multiprocessor systems.
• Each processor has its own high-speed cache memory.
• Processors are connected to each other via a system bus.

​

3.12.4 Massively Parallel Processing (MPP)

Definition:

MPP refers to the coordinated processing of programs by multiple processors working in parallel.

Key characteristics:

• Each processor has its own operating system and dedicated (separate) memory.
• Processors work on different parts of the same program simultaneously.
• Processors communicate using a messaging interface.

Challenge:

• MPP systems are more difficult to program, since the application must be divided in a way that allows all executing segments to communicate with each other.

​

3.12.5 Difference Between Parallel and Distributed Systems

Parallel Database System: A tightly coupled system

• The processors co-operate for query processing.
• The user is unaware of the parallelism, since they have no direct access to the individual processors they interact via a front-end computer, which distributes work to the back-end parallel system (P1, P2, P3...).
• The system either uses: A common shared memory Processors connected to shared Memory or Message passing for communication between processors.

Distributed Database System: Known as a loosely coupled system. Composed of individual, separate machines.

• Each machine runs its own individual application and serves its own respective user.
• Data is usually distributed across several machines, meaning multiple machines often need to be accessed to answer a single user query.

​

3.12.6 Shared Nothing Architecture

Three common types of architecture for multiprocessor high transaction rate systems.

1. Shared Memory (SM): A common central memory is shared by multiple processors.

2. Shared Disk (SD): Multiple processors share a common collection of disks, but each has its own private memory.

3. Shared Nothing (SN): Neither memory nor disk is shared among multiple processors each processor is fully independent with its own memory and disk.

​

​

Advantages of a "Shared Nothing Architecture"

1. Fault Isolation

• A fault occurring in a single node is contained and confined to that node exclusively.
• The fault is only exposed to other nodes through messages (or the lack of messages, i.e., silence indicating failure).

2. Scalability

• If disk were a shared resource, the controller and disk bandwidth would also be shared.
• This would require synchronization to maintain a consistent shared state.
• As a result, different nodes would need to take turns accessing critical data creating a bottleneck.

​

4.1 NoSQL (Not Only SQL)

4.1.1 Where is it Used?

4.1.2 What is it?

4.1.3 Types of NoSQL Databases

4.1.4 Why NoSQL?

4.1.5 Advantages of NoSQL

4.1.6 What We Miss With NoSQL?

4.1.7 Use of NoSQL in Industry

4.1.8 NoSQL Vendors

4.1.9 SQL versus NoSQL

4.1.10 NewSQL

4.1.11 Comparison of SQL, NoSQL, and NewSQL

​

Origin & Definition

• Term coined by Carlo Strozzi (1998), reintroduced by Johan Oskarsson (2009) at a Last.fm event; #NoSQL popularized by Eric Evans to describe non-relational databases.

What is NoSQL?

• non-relational, open-source, distributed databases that don't rely on the traditional table-based structure used by relational databases (RDBMS).

Key Features:

• Open source, non-relational, distributed, schema-less, cluster-friendly, born from 21st-century web apps.

Where/Why Used:

• Log data, social media data, real-time web apps, where RDBMS struggles.

​

​

Types of NoSQL Databases:

• Key-Value: big hash table (e.g., Dynamo, Redis, Riak)
• Document: documents in collections (e.g., MongoDB, CouchDB)
• Column: data stored by column (e.g., Cassandra, HBase)
• Graph: data as nodes/relationships (e.g., Neo4j, HyperGraphDB)

Mnemonic: "Kids Drink Cold Ginger"

• Key-Value: Redis
• Document: MongoDB
• Column: Cassandra
• Graph: Neo4j (think "Graph = Network")

​

​

​

Why NoSQL?

1. Scale-out architecture (instead of monolithic architecture)

• NoSQL grows by adding more machines to a cluster, instead of relying on one large, monolithic server like RDBMS.
• Example: Facebook adds more low-cost servers to its Cassandra cluster as user data grows, rather than upgrading a single powerful machine.

2. Houses large volumes of structured, semi-structured, and unstructured data

• NoSQL can store all data types together tables, JSON-like data, and free-form text/media.
• Example: MongoDB storing customer profiles (structured), chat logs (unstructured), and sensor readings (semi-structured) all in one database.

​

​

3. Dynamic schema

• NoSQL lets you insert data without a pre-defined schema, allowing real-time application changes with faster development and less DB administration.
• Example: In MongoDB, one document can have {name, email} while a newer document adds {name, email, address} no need to redesign a table first.

4. Auto-sharding

• NoSQL automatically spreads data across any number of servers, balancing load and replacing failed servers without disrupting the app.
• Example: DynamoDB automatically distributing user data across 10 servers, and instantly rerouting traffic if one server crashes the application never notices.

​

​

5. Replication

• NoSQL supports strong replication, ensuring high availability, fault tolerance, and disaster recovery.
• Example: Cassandra keeping 3 copies of the same data across different data centers, so if one center fails, the data is still fully accessible from another.

​

Advantages of NoSQL

1. Can easily scale up and down: NoSQL database supports scaling rapidly and elastically and even allows to scale to the cloud.
• Cluster scale: It allows distribution of database across 100+ nodes often in multiple data centers.
• Performance scale: It sustains over 100,000+ database reads and writes per second.
• Data scale: It supports housing of 1 billion+ documents in the database.

​

2. Doesn't require a pre-defined schema:

• NoSQL database does not require any adherence to pre-defined schema. It is pretty flexible.
• For example, if we look at MongoDB, the documents in a collection can have different sets of key-value pairs.

{_id: 101, "BookName": "Fundamentals of Business Analytics", "Author_Name": "Seema Acharya", "Publisher": "Wiley India"}

{_id: 102, "Book_Name": "Big Data and Analytics"}

​

3. Cheap, easy to implement:

• Deploying NoSQL properly allows for all of the benefits of scale, high availability, fault tolerance, etc. while also lowering operational costs.

4. Relaxes the data consistency requirement:

• NoSQL databases have adherence to CAP theorem (Consistency, Availability, and Partition tolerance).
• Most of the NoSQL databases compromise on consistency in Favor of availability and partition tolerance. However, they do go for eventual consistency.

​

5. Data can be replicated to multiple nodes and can be partitioned:

1. Sharding:

• Sharding means splitting data into pieces and spreading them across multiple servers.
• NoSQL does this automatically (auto-sharding) the app doesn't need to know which server holds what.
• Servers can be added or removed anytime without stopping the application, and if one server fails, it's replaced smoothly with no disruption.

2. Replication:

• Replication means keeping multiple copies of the same data on different servers (even in different data centers).
• This ensures the data is always available, even if one server or data center goes down.

​

What We Miss With NoSQL?

Joins, Group by, ACID properties, Easy integration with other applications that support SQL.

• NoSQL does not support joins. However, it compensates for it by allowing embedded documents as in MongoDB.
• It does not have provision for ACID properties of transactions. However, it obeys the Eric Brewer's CAP theorem.
• NoSQL does not have a standard SQL interface but NoSQL databases such as MongoDB and Cassandra have their own rich query language [MongoDB and Cassandra query language (CQL)] to compensate for the lack of it.

​

​

Use of NoSQL in Industry

NoSQL is being put to use in varied industries and They are used to support analysis for applications such as

• Web user data analysis
• Log analysis
• Sensor feed analysis
• Making recommendations for upsell and cross-sell, etc.

​

NoSQL Vendors

4.2 Hadoop

• 4.2.1 Features of Hadoop
• 4.2.2 Key Advantages of Hadoop
• 4.2.3 Versions of Hadoop
• 4.2.4 Overview of Hadoop Ecosystems
• 4.2.5 Hadoop Distributions
• 4.2.6 Hadoop versus SQL
• 4.2.7 Integrated Hadoop Systems Offered by Leading Market Vendors
• 4.2.8 Cloud-Based Hadoop Solutions

​

What is Hadoop?

• Apache Hadoop is an open source framework that is used to efficiently store and process large datasets ranging in size from gigabytes to petabytes of data.
• Instead of using one large computer to store and process the data, Hadoop allows clustering multiple computers to analyze massive datasets in parallel more quickly.

​

Features of Hadoop

1. It is optimized to handle massive quantities of structured, semi-structured, and unstructured data, using commodity hardware, that is, relatively inexpensive computers.
2. Hadoop has a shared nothing architecture.
3. It replicates its data across multiple computers so that if one goes down, the data can still be processed from another machine that stores its replica.
4. Hadoop is for high throughput rather than low latency.

Features of Hadoop

5. It is a batch operation handling massive quantities of data; therefore the response time is not immediate.
6. It complements On-Line Transaction Processing (OLTP) and On-Line Analytical Processing (OLAP). However, it is not a replacement for a relational database management system.
7. It is NOT good when work cannot be parallelized or when there are dependencies within the data.
8. It is NOT good for processing small files.
9. It works best with huge data files and datasets.

​

Key Advantages of Hadoop

1. Stores Data in Its Native Format

• Hadoop data storage framework (HDFS — Hadoop Distributed File System) can store data in its native format.
• There is no structure imposed while storing data.
• HDFS is pretty much schema-less. It is only later, when the data needs to be processed, that structure is imposed on the raw data.
• Hadoop can store data in the same format in which it is originally generated or received, without requiring the data to be converted into a predefined format before storing it.
• HDFS does not impose a fixed structure or schema on the data while storing it.
• It can store structured, semi-structured, and unstructured data.
• The structure or schema can be applied later, when the data is processed or analyzed. This is called schema-on-read.

​

2. Scalable: Hadoop can store and distribute very large datasets across hundreds of inexpensive servers that operate in parallel.

3. Cost-Effective: Owing to its scale-out architecture, Hadoop has a much reduced cost/terabyte of storage and processing.

4. Resilient to Failure

• Hadoop is fault-tolerant.
• Whenever data is sent to any node, the same data also gets replicated to other nodes in the cluster. This ensures that, in the event of a node failure, there will always be another copy of data available for use.

​

​

5. Flexibility

• One of the key advantages of Hadoop is its ability to work with all kinds of data: Structured, Semi-structured, Unstructured.
• It can help to derive meaningful business insights from: Email conversations, Social media data, Click-stream data.
• It can be put to several purposes such as: Log analysis, Data mining, Recommendation systems.

6. Fast: Processing is extremely fast in Hadoop as compared to other conventional systems.

​

Versions of Hadoop��Hadoop version 1 (Hadoop 1.x or Hadoop v1)�Hadoop version 2 (Hadoop 2.x or Hadoop v2)��

Hadoop version 1 (Hadoop 1.x or Hadoop v1):

• This was the initial release of Hadoop, also known as Apache Hadoop Core.
• It consisted of two main components: Hadoop Distributed File System (HDFS) and MapReduce.
• Hadoop 1.x had limitations such as a single point of failure (NameNode) and a lack of support for running multiple workloads simultaneously.
• It was suitable for batch processing applications but had scalability and performance limitations for certain use cases.

​

Hadoop version 2 (Hadoop 2.x or Hadoop v2):

• Hadoop 2.x introduced significant architectural changes and improvements over Hadoop 1.x.
• The major enhancement was the introduction of YARN (Yet Another Resource Negotiator), which decoupled the resource management and job scheduling functionalities from MapReduce.
• Hadoop 2.x addressed scalability and multi-tenancy issues (multiple users or applications can use the same Hadoop cluster at the same time.) by providing better resource utilization and support for running multiple applications simultaneously.
• With YARN, Hadoop 2.x became more versatile, supporting various processing models such as batch processing, interactive querying, real-time processing, etc.
• Hadoop 2.x also introduced various other improvements and features, including HDFS High Availability (HA) and federation, performance optimizations, and enhancements to MapReduce.

​

Hadoop version 3 (Hadoop 3.x or Hadoop v3):

• Hadoop 3.x builds upon the improvements made in Hadoop 2.x and introduces several new features and enhancements.
• One significant enhancement in Hadoop 3.x is support for erasure coding in HDFS, which provides more efficient data storage compared to traditional replication methods, reducing storage overhead.
• Hadoop 3.x also includes performance improvements, optimizations, and various updates to its components, including HDFS, YARN, and MapReduce.
• Another important addition in Hadoop 3.x is support for resource types in YARN, which allows users to define custom resources beyond CPU and memory, enabling better resource management for diverse workloads.
• Hadoop 3.x continues to improve scalability, reliability, and security features compared to previous versions.

​

​

Overview of Hadoop Ecosystems

The image shows the Apache Hadoop Ecosystem, which is a collection of tools that work together to store, process, analyze, manage, and monitor Big Data.

​

There are components available in the Hadoop ecosystem for data ingestion, processing, and analysis.

Components chat help With Data Ingestion are:

• Sqoop
• Flume

Components that help with Dara Processing are:

• MapReduce
• Spark

Components that help which Data Analysis are:

• Pig
• Hive
• Impala

Hadoop Ecosystem Components for Data Ingestion

1. Sqoop: Sqoop stands for SQL to Hadoop.

Main Functions of Sqoop

Importing Data:

• Importing data from RDBMS such as: MySQL, Oracle, DB2
• The data can be imported to Hadoop file system: HDFS, Hbase, Hive

Exporting Data

Exporting data from Hadoop File System: HDFS, Hbase, Hive To RDBMS such as: MySQL, Oracle, DB2

​

​

Uses of Sqoop

a) Connector-Based Architecture

It has a connector-based architecture to allow plug-ins to connect to external systems such as: MySQL, Oracle, DB2

b) Provisioning Data

It can provision the data from external system onto HDFS.

It can populate tables in Hive and HBase.

c) Integration with Oozie

It integrates with Oozie, allowing you to schedule and automate import and export tasks.

​

2. Flume

• Flume is an important log aggregator component in the Hadoop ecosystem.
• It aggregates logs from different machines and places them in HDFS.
• Flume has been developed by Cloudera.
• It is designed for high-volume ingestion of event-based data into Hadoop.
• The default destination in Flume, called sink in Flume parlance, is HDFS.
• However, it can also write to: Hbase, Solr.

​

Hadoop Ecosystem Components for Data Processing

1. MapReduce

• It is a programming paradigm that allows distributed and parallel processing of huge datasets.
• It is based on Google MapReduce.
• Google released a paper on the MapReduce programming paradigm in 2004, and that became the genesis of the Hadoop processing model.
• The MapReduce framework gets the input data from HDFS.
• There are two main phases: Map phase and Reduce phase

​

Map Phase

• The map phase converts the input data into another set of data (key-value pairs).
• This new intermediate dataset then serves as the input to the reduce phase.

Reduce Phase

• The reduce phase acts on the datasets to combine (aggregate and consolidate) and reduce them to a smaller set of tuples.
• The result is then stored back in HDFS.

​

2. Spark

• It is both a programming model as well as a computing model.
• It is an open-source big data processing framework.
• It was originally developed in 2009 at UC Berkeley's AmpLab and became an open-source project in 2010.
• It is written in Scala.
• It provides in-memory computing for Hadoop.
• In Spark, workloads execute in memory rather than on disk, owing to which it is much faster (up to 100 times) than when the workload is executed on disk.
• However, if the datasets are too large to fit into the available system memory, it can perform conventional disk-based processing.
• It serves as a potentially faster and more flexible alternative to MapReduce.
• It accesses data from HDFS (Spark does not have its own distributed file system) but bypasses the MapReduce processing.
• Spark can be used with Hadoop, coexisting smoothly with MapReduce (sitting on top of Hadoop YARN) or used independently of Hadoop (standalone).

​

• As a programming model, it works well with: Scala, Python, R programming language
• It has API connectors for using it with Java or Python.

Spark Libraries

a) Spark SQL: Spark also has support for SQL.

Spark SQL uses SQL to help query data stored in disparate applications.

b) Spark Streaming: It helps to analyze and present data in real time.

c) Mllib: It supports machine learning, such as applying advanced statistical operations on data in the Spark Cluster.

d) GraphX: It helps in graph parallel computation.

​

Hadoop Ecosystem Components for Data Analysis

1. Pig

• It is a high-level scripting language used with Hadoop.
• It serves as an alternative to MapReduce.

It has two parts: (a) Pig Latin , (b) Pig Runtime

(a) Pig Latin

• It is a SQL-like scripting language.
• Pig Latin scripts are translated into MapReduce jobs, which can then run on YARN and process data in the HDFS cluster.
• It was initially developed by Yahoo.
• It is immensely popular with developers who are not comfortable with MapReduce.
• However, SQL developers may have a preference for Hive.

​

How Pig Works?

• There is a "Load" command available to load the data from "HDFS" into Pig.
• Then one can perform functions such as: Grouping, Filtering, Sorting, Joining
• The processed or computed data can then be either: Displayed on screen, or Placed back into HDFS.
• It gives you a platform for building data flow for ETL (Extract, Transform and Load), processing and analyzing huge data sets.

​

2. Hive

• Hive is a data warehouse software project built on top of Hadoop.
• Three main tasks performed by Hive are: Summarization, Querying, Analysis
• It supports queries written in a language called HQL or HiveQL.
• HQL/HiveQL is a declarative SQL-like language.
• It converts the SQL-style queries into MapReduce jobs, which are then executed on the Hadoop platform.

​

• Impala: It is a high performance SQL engine that runs on Hadoop cluster. It is ideal for interactive analysis. It has very low latency measured in milliseconds. It supports a dialect of SQL called Impala SQL.
• ZooKeeper: It is a coordination service for distributed applications.
• Oozie: It is a workflow scheduler system to manage Apache Hadoop jobs.
• Mahout: It is a scalable machine learning and data mining library.
• Chukwa: It is a data collection system for managing large distributed systems.
• Ambari: It is a web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters.

​

4.2.5 Hadoop Distributions

Hadoop:

• Hadoop is an Open-source Apache project.
• Anyone can freely download the core aspects of Hadoop.

Core Aspects of Hadoop

1. Hadoop Common
2. Hadoop Distributed File System (HDFS)
3. Hadoop YARN (Yet Another Resource Negotiator)
4. Hadoop MapReduce

Hadoop Distributions or Services

• There are few companies such as: IBM, Amazon Web Services, Microsoft, Teradata, Hortonworks, Cloudera, etc.
• Although each of these companies has a slightly different strategy, the key essence remains its ability to distribute data and workloads across potentially thousands of servers, thus making big data manageable data.

​

​

4.2.7 Integrated Hadoop Systems Offered by Leading Market Vendors

​

4.2.8 Cloud-Based Hadoop Solutions

​

​

Cloud-Based Hadoop Solutions

1. Amazon Web Services

• Amazon Web Services holds out a comprehensive, end-to-end portfolio of cloud computing services to help manage big data.
• The aim is to achieve this and more along with retaining the emphasis on:
• Reducing costs
• Scaling to meet demand
• Accelerating the speed of innovation

​

2. Google Cloud Storage Connector for Hadoop

• The Google Cloud Storage connector for Hadoop empowers one to perform MapReduce jobs directly on data in Google Cloud Storage.
• There is no need to:
• Copy the data to local disk.
• Run it in the Hadoop Distributed File System (HDFS).
• The Connector simplifies Hadoop deployment.
• At the same time, it:
• Reduces cost
• Provides performance comparable to HDFS
• Increases reliability by eliminating the single point of failure of the NameNode.

​