DEVELOP AN OPTIMIZED FRAMEWORK FOR ANALYSIS OF STATIC AND RAPID IOT DATA
RESEARCH & DEVELOPMENT PROJECT
Recap
Introduction
Literature Review
Proposed Methodology
Tech Stack
Workflow
Results And Analysis
Challenges So Far
Future Scope
References
TABLE OF CONTENT
Completed Extract And First Phase of Transform
Analysis On Sample Data
RECAP
ETL Process
INTRODUCTION
INTRODUCTION
2.
CHALLENGE
Traditional data analysis techniques lead to delays and inaccuracies
3.
SOLUTION
Need framework to process massive data
4.
WHY
Analyze both static and real time data. To ensure accuracy
1.
IoT
By analyzing this data, significant insights can be gained.
Developing an optimized framework for analyzing static and rapidly changing IoT And Provide Real-Time Insights Into The Data Generated By IOT Devices
PROBLEM STATEMENT
LITERATURE REVIEW
RELATED
WORKS
03
On-Demand ETL Architecture for BI
GAPS
Papers reviewed recognize the challenges in processing real time data.
01
They rely on the traditional transformation steps and apply these to current ETL/ELT tools to deliver data to the front end, leading to failures.
02
They DO NOT explore the natural ability of stream data to be grouped under “windows” either during
03
PROPOSED METHODOLOGY
OVERVIEW
Loading the final data into Hive Databases
05
Defining the Problem and Objectives
01
Data Transformation
04
Data Collection and Preprocessing
02
Windowing
03
DEFINING THE PROBLEM AND OBJECTIVES
Data Collection & Preprocessing
Data pipeline to collect real-time streaming data from IoT devices
Using Morphline
Flume configuration files for data sources, channels, and sinks
02
-
Helps with ambiguous data
System designed to effectively gather and transfer streaming data.
Defining Problem & Objectives
To retrieve data from IoT devices, we employed Apache Flume
01
-
WINDOWING
Windowing based on local timestamps
Dividing the data into smaller time intervals or windows for better querying and transformation
01
02
Using conf files
Utilizing Morphline properties to pipeline the data.
morphlines.conf
morphex.conf
morph_result.conf
DATA TRANSFORMATION
Partitioning and Clustering
Partitioning - Divides data into directories for easier data management.
Clustering - Organises data based on columns to improve query performance.
Using HIVE
Windowed data to be transformed into a Unified Schema using HIVE’s properties.
01
02
LOADING IN DATABASES
Ease of efficient querying
It enables the integration of data from multiple sources for efficient querying and analysis.
Stores Data on HDFS
Unified Schema
Unified Schema is a standardized way to store and access data in HIVE databases.
01
02
Creation of HIVE Database
Starting by creating a HIVE table using Command Line Interface
03
TECH STACK
TECH STACK
HDFS
Map-Reduce
01
HADOOP
Agents
Data-Flow
02
FLUME
Metastore
Query Engine
03
HIVE
WORKFLOW
INPUT SANITIZATION
The following structure of data gives parsing error due to Grok Expectations
Hence the input was correctly sanitized to be parsed by the upcoming stages
STAGE 1: DATA INPUT AND PARSING
MORPH_RESULT.CONF
STAGE 2: DATA INPUT AND PARSING
MORPHEX.CONF
LOADING OF DATA IN HIVE
timewindow
stage2
stage1
data
LOADING OF DATA IN HIVE
time_05_05_2017/stream.1683606675748
data
Processed Data
s1flumeDir1
EXTRACTION IN HIVE
stage1
data
Split data
TRANSFORMATION OF DATA IN HIVE
Timestamped data
stage2
stage1
PARTITIONING OF DATA IN HIVE
Final data
timewindow
stage2
FINAL HIVE TABLES IN HDFS
user/hive/warehouse/stage1
user/hive/warehouse/stage2
user/hive/warehouse/timewindow
RESULTS AND ANALYSIS
THE CONFIGURATION FILES
Structure for the Stage 1
Morph_result.conf file is used in our Morphline use-case to define a proper structure for the stage 1 of the development
MORPH_RESULT.CONF
Utilizes morphline interceptors
This file processes the input data and puts it into the results folder for the next stage.
02
01
THE CONFIGURATION FILES
Structure for FLUME agent for stage 2
Morphex.conf file is used in our Morphline use-case to define a proper structure for our Flume Agent to work.
MORPHEX.CONF
Defining agent and it’s working
It defines the Source, the Sink, and the channels and their types to facilitate data transfer between both of them
02
01
THE CONFIGURATION FILES
Definition
Morphline.conf is our configuration file used in Flume to define ETL transformations on data streams.
MORPHLINE.CONF
Parsing of Data
It specifies a series of commands to process and modify data coming in from the source.
02
01
HADOOP FILE SYSTEM
All the data is first processed through our 2 phases and then stored in the HDFS directory s1flumeDir1
After the preprocessing is complete the data is arranged in the following tables through HIVE, making the framework more scalable
ANALYSIS
SCALABLE
Partitioning
Clustering
01
FLEXIBLE
Morphex.conf
Morphline.conf
02
Morph_result.conf
HIVE
FLUME
ANALYSIS
EFFICIENT
CUSTOM MONTH INTERCEPTOR
CUSTOM TIME INTERCEPTOR
CHALLENGES SO FAR
ISSUE : PARALLEL LAUNCH OF HIVE METASTORE NOT SUPPORTED
Resolution
Running the shown command separately resolved the issue
03
Parallel launch of HIVE Metastore not supported
01
HADOOP SAFE MODE ON
RESOLUTION
01
02
Hadoop "safemode" is essentially a restricting setting for the HDFS
Led the HIVE to not get started as it requires full data transaction
Manually fix this issue by separately forcing Hadoop as an admin to leave the safemode
ISSUE: UNRESOLVED IMPORTS
We got this issue of unresolved imports in the JAR files
01
02
Didn’t let the codes to execute properly
RESOLUTION: UNRESOLVED IMPORTS
Resolved by importing the required libraries
01
02
Libraries correctly added to the classpath of the newly exported JAR files
ISSUE: RECURRING WARNINGS
Infinite loop of the above WARNING messages.
01
02
Warn messages were occuring due to expectations of grok patterns.
RESOLUTION: INPUT SANITIZATION
01
02
We manually fixed the csv file by sanitising the input files as per the Grok.
By changing the structure of the data.
FUTURE PROSPECTS
SHORTCOMINGS
SCOPE OF IMPROVEMENT
Seamless Flume-To-Hive Integration For Automation
Data Visualization And Dashboards
Scaling Data Analysis For Variety of Datasets
Continuous Optimisation And Automation
Integration Of Machine Learning Algorithms
REFERENCES
REFERENCES
3. Adilah Sabtu, Nurulhuda Firdaus Mohd Azmi , Nilam Nur Amir Sjarif , et al. (2017, November 30). THE CHALLENGES OF EXTRACT, TRANSFORM AND LOAD (ETL) FOR DATA INTEGRATION IN NEAR REAL TIME ENVIRONMENT. Journal of Theoretical and Applied Information Technology, 95(22), 6314–6322.
4. Pravin Gokhe (2016, March 26). ENTERPRISE REAL-TIME INTEGRATION (1st ed., Vol. 1) [English].(pp 51-57)
REFERENCES
5. Tönjes, R., Barnaghi, P., Ali, M., Mileo, A., Hauswirth, M., Ganz, F., & Puiu, D. (2014, June). Real time IoT stream processing and large-scale data analytics for smart city applications. (pp. 24-28, 39-44)
6. Hoffman, S. (2015). Apache flume: Distributed log collection for hadoop. Birmingham Packt Publishing.(pp. 62-84)
8. Apache (2014) Apache Hadoop. http://hadoop.apache.org/.
9. Capriolo E, Wampler D, Rutherglen J (2012 December). Programming Hive. O’Reilly Media, Inc. (pp. 49-97)
7. Waas, F., Wrembel, R., Freudenreich, T., Thiele, M., Koncilia, C., & Furtado, P. (2013, April 1). On-Demand ELT Architecture for Right-Time BI. International Journal of Data Warehousing and Mining, 9(2), 21–38.
THANK YOU
Now we will move to the Demonstration