Introduction to Data Science

By

​

S.V.V.D.Jagadeesh

Sr. Assistant Professor

Dept of Artificial Intelligence & Data Science

​

​

LAKIREDDY BALI REDDY COLLEGE OF ENGINEERING

• Session Outcomes
• Problems with Handling Large Data- Lesser Memory, Computational Speed and Time, Components used for Computing
• Solutions to Handle Large Data- Choose the Right Algorithm, Choose the Right Data Structures, Choose the Right Tools

​

​

​

​

​

S.V.V.D.Jagadeesh

Tuesday, January 28, 2025

Previously Discussed Topics

LBRCE

IDS

At the end of this session, Student will be able to:

• Understand the general tips to handle large amounts of data(Understand- L2)

S.V.V.D.Jagadeesh

Tuesday, January 28, 2025

Session Outcomes

LBRCE

IDS

S.V.V.D.Jagadeesh

Tuesday, January 28, 2025

General tips for handling large data

LBRCE

IDS

• Exploit the power of databases. The first reaction most data scientists have when working with large data sets is to prepare their analytical base tables inside a database. This method works well when the features you want to prepare are fairly simple. When this preparation involves advanced modeling, find out if it’s possible to employ user-defined functions and procedures.
• Use optimized libraries. Creating libraries like Mahout, Weka, and other machine-learning algorithms requires time and knowledge. They are highly optimized and incorporate best practices and state-of-the art technologies. Spend yourtime on getting things done, not on reinventing and repeating others people’s efforts, unless it’s for the sake of understanding how things work.

S.V.V.D.Jagadeesh

Tuesday, January 28, 2025

Don’t Reinvent the wheel

LBRCE

IDS

• Feed the CPU compressed data. A simple trick to avoid CPU starvation is to feed the CPU compressed data instead of the inflated (raw) data. This will shift more work from the hard disk to the CPU, which is exactly what you want to do, because a hard disk can’t follow the CPU in most modern computer architectures.
• Make use of the GPU . Sometimes your CPU and not your memory is the bottle-neck. If your computations are parallelizable, you can benefit from switchinto the GPU . This has a much higher throughput for computations than a CPU . The GPU is enormously efficient in parallelizable jobs but has less cache than the CPU . But it’s pointless to switch to the GPU when your hard disk is the problem.

S.V.V.D.Jagadeesh

Tuesday, January 28, 2025

Get the most out of your hardware

LBRCE

IDS

• Several Python packages, such as Theano and NumbaPro, will use the GPU without much programming effort. If this doesn’t suffice, you can use a CUDA (Compute Unified Device Architecture) package such as PyCUDA . It’s also a well-known trick in bitcoin mining, if you’re interested in creating your own money.
• Use multiple threads. It’s still possible to parallelize computations on your CPU .You can achieve this with normal Python threads.

S.V.V.D.Jagadeesh

Tuesday, January 28, 2025

Get the most out of your hardware

LBRCE

IDS

• Profile your code and remediate slow pieces of code. Not every piece of your code needs to be optimized; use a profiler to detect slow parts inside your program and remediate these parts.
• Use compiled code whenever possible, certainly when loops are involved. Whenever possible use functions from packages that are optimized for numerical computations instead of implementing everything yourself. The code in these packages is often highly optimized and compiled.
• Otherwise, compile the code yourself. If you can’t use an existing package, use either a just-in-time compiler or implement the slowest parts of your code in a lower-level language such as C or Fortran and integrate this with your codebase.

S.V.V.D.Jagadeesh

Tuesday, January 28, 2025

Reduce Your Computing Needs

LBRCE

IDS

• If you make the step to lower-level languages (languages that are closer to the universal computer bytecode), learn to work with computational libraries such as LAPACK, BLAST, Intel MKL, and ATLAS. These are highly optimized, and it’s difficult to achieve similar performance to them.
• Avoid pulling data into memory. When you work with data that doesn’t fit in your memory, avoid pulling everything into memory. A simple way of doing this is by reading data in chunks and parsing the data on the fly. This won’t work on every algorithm but enables calculations on extremely large data sets.

S.V.V.D.Jagadeesh

Tuesday, January 28, 2025

Reduce Your Computing Needs

LBRCE

IDS

• Use generators to avoid intermediate data storage. Generators help you return data per observation instead of in batches. This way you avoid storing intermediate results.
• Use as little data as possible. If no large-scale algorithm is available and you aren’t willing to implement such a technique yourself, then you can still train your data on only a sample of the original data.
• Use your math skills to simplify calculations as much as possible. Take the following equation, for example: (a + b)2 = a2 + 2ab + b2 . The left side will be computed much faster than the right side of the equation; even for this trivial example, it could make a difference when talking about big chunks of data.

S.V.V.D.Jagadeesh

Tuesday, January 28, 2025

Reduce Your Computing Needs

LBRCE

IDS

• Session Outcomes
• General Tips to Handle Large Data
• Don’t Reinvent the Wheel
• Get the most out of your Hardware
• Reduce Your Computer Needs

​

​

​

​

​

​

S.V.V.D.Jagadeesh

Tuesday, January 28, 2025

Summary

LBRCE

IDS