1 of 24

Introduction to Data Science

By

​

S.V.V.D.Jagadeesh

Sr. Assistant Professor

Dept of Artificial Intelligence & Data Science

​

​

LAKIREDDY BALI REDDY COLLEGE OF ENGINEERING

2 of 24

  • Session Outcomes
  • Types of Machine Learning
  • Supervised Learning
  • Un-Supervised Learning
  • Semi-Supervised Learning

​

​

​

​

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Previously Discussed Topics

LBRCE

IDS

3 of 24

At the end of this session, Student will be able to:

  • Understand the problems and solutions to handle large amounts of data(Understand- L2)

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Session Outcomes

LBRCE

IDS

4 of 24

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Problems with handling large data

LBRCE

IDS

5 of 24

  • A computer only has a limited amount of RAM.
  • When you try to squeeze more data into this memory than actually fits, the OS will start swapping out memory blocks to disks, which is far less efficient than having it all in memory.
  • But only a few algorithms are designed to handle large data sets; most of them load the whole data set into memory at once, which causes the out-of-memory error.
  • Other algorithms need to hold multiple copies of the data in memory or store intermediate results.

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Lesser Memory

LBRCE

IDS

6 of 24

  • Even when you cure the memory issues, you may need to deal with another limited resource: time.
  • Although a computer may think you live for millions of years, in reality you won’t.
  • Certain algorithms don’t take time into account; they’ll keep running forever.
  • Other algorithms can’t end in a reasonable amount of time when they need to process only a few megabytes of data.

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Computational Speed and time

LBRCE

IDS

7 of 24

  • A third thing you’ll observe when dealing with large data sets is that components of your computer can start to form a bottleneck while leaving other systems idle.
  • Although this isn’t as severe as a never-ending algorithm or out-of-memory errors, it still incurs a serious cost.
  • Think of the cost savings in terms of person days and computing infrastructure for CPU starvation.

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Components used for Computing

LBRCE

IDS

8 of 24

  • Certain programs don’t feed data fast enough to the processor because they have to read data from the hard drive, which is one of the slowest components on a computer.
  • This has been addressed with the introduction of solid state drives (SSD), but SSDs are still much more expensive than the slower and more widespread hard disk drive (HDD) technology.

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Components used for Computing

LBRCE

IDS

9 of 24

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Solutions to handle large data

LBRCE

IDS

10 of 24

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Choose the right Algorithm

LBRCE

IDS

11 of 24

  • Several, but not all, machine learning algorithms can be trained using one observation at a time instead of taking all the data into memory.
  • Upon the arrival of a new data point, the model is trained and the observation can be forgotten; its effect is now incorporated into the model’s parameters.
  • For example, a model used to predict the weather can use different parameters (like atmospheric pressure or temperature) in different regions. When the data from one region is loaded into the algorithm, it forgets about this raw data and moves on to the next region.

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Online Learning Algorithms

LBRCE

IDS

12 of 24

  • By cutting a large data table into small matrices, for instance, we can still do a linear regression.
  • The logic behind this matrix splitting and how a linear regression can be calculated with matrices can be found in the sidebar

​

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Dividing Bigger Matrix in Small Chunks

LBRCE

IDS

13 of 24

  • bcolz is a Python library that can store data arrays compactly and uses the hard drive when the array no longer fits into the main memory.
  • Dask is a library that enables you to optimize the flow of calculations and makes performing calculations in parallel easier.

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Libraries for Dividing Bigger Matrix in Small Chunks

LBRCE

IDS

14 of 24

  • MapReduce algorithms are easy to understand with an analogy: Imagine that you were asked to count all the votes for the national elections.
  • Your country has 25 parties, 1,500 voting offices, and 2 million people. You could choose to gather all the voting tickets from every office individually and count them centrally, or you could ask the local offices to count the votes for the 25 parties and hand over the results to you, and you could then aggregate them by party.
  • Map reducers follow a similar process to the second way of working.
  • They first map values to a key and then do an aggregation on that key during the reduce phase.

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Map Reduce Algorithm

LBRCE

IDS

15 of 24

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Choose A Right Data Structure

LBRCE

IDS

16 of 24

  • A sparse data set contains relatively little information compared to its entries (observations).
  • Data like this might look ridiculous, but this is often what you get when converting textual data to binary data.

​

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Sparse Data

LBRCE

IDS

17 of 24

  • Trees are a class of data structure that allows you to retrieve information much faster than scanning through a table.
  • A tree always has a root value and subtrees of children, each with its children, and so on.
  • Simple examples would be your own family tree or a biological tree and the way it splits into branches, twigs, and leaves.
  • Simple decision rules make it easy to find the child tree in which your data resides

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Tree Data Structures

LBRCE

IDS

18 of 24

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Tree Data Structure

LBRCE

IDS

19 of 24

  • Hash tables are data structures that calculate a key for every value in your data and put the keys in a bucket.
  • This way you can quickly retrieve the information by looking in the right bucket when you encounter the data.
  • Dictionaries in Python are a hash table implementation, and they’re a close relative of key-value stores.

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Hash Tables

LBRCE

IDS

20 of 24

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Selecting the right Tool

LBRCE

IDS

21 of 24

  • Cython-For a computer, adding 1 + 1 is different from adding 1.00 + 1.00. The first example consists of integers and the second consists of floats, and these calculations are performed by different parts of the CPU. In Python you don’t have to specify what data types you’re using, so the Python compiler has to infer them. But inferring data types is a slow operation and is partially why Python isn’t one of the fastest languages available. Cython, a superset of Python, solves this problem by forcing the programmer to specify the data type while developing the program. Once the compiler has this information, it runs programs much faster.

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Python Tools

LBRCE

IDS

22 of 24

  • Numexpr—Numexpr is at the core of many of the big data packages, as is NumPy for in-memory packages. Numexpr is a numerical expression evaluator for NumPy but can be many times faster than the original NumPy.
  • Numba—Numba helps you to achieve greater speed by compiling your code right before you execute it, also known as just-in-time compiling. This gives you the advantage of writing high-level code but achieving speeds similar to those of C code.
  • Bcolz—Bcolz helps you overcome the out-of-memory problem that can occur when using NumPy. It can store and work with arrays in an optimal compressed form. It not only slims down your data need but also uses Numexpr in the background to reduce the calculations needed when performing calculations with bcolz arrays

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Python Tools

LBRCE

IDS

23 of 24

  • Blaze—Blaze is ideal if you want to use the power of a database backend but like the “Pythonic way” of working with data. Blaze will translate your Python code into SQL but can handle many more data stores than relational databases such as CSV, Spark, and others
  • Theano—Theano enables you to work directly with the graphical processing unit (GPU) and do symbolical simplifications whenever possible, and it comes with an excellent just-in-time compiler. Theano is a great library for working with tensors
  • Dask—Dask enables you to optimize your flow of calculations and execute them efficiently. It also enables you to distribute calculations.

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Python Tools

LBRCE

IDS

24 of 24

  • Session Outcomes
  • Problems with Handling Large Data- Lesser Memory, Computational Speed and Time, Components used for Computing
  • Solutions to Handle Large Data- Choose the Right Algorithm, Choose the Right Data Structures, Choose the Right Tools

​

​

​

​

​

S.V.V.D.Jagadeesh

Tuesday, January 21, 2025

Summary

LBRCE

IDS