Building ML Models for Large Datasets

​

Big Data Analytics with Python, AIMS 2023

Dunstan Matekenya, PhD

Contents

1. The ML process/pipeline
2. How data can be big for ML
3. Dealing with big data in ML
• Data reduction, preliminary optimization
• Distributed ML algorithms, model training
• Model compression techniques
• Scalable deep learning/machine learning
4. Tooling/resources for big data in ML
5. ML with spark

Machine Learning Pipeline

Data processing/preprocessing

Model building

Model serving/inference

Feature engineering

Hyparameter tuning

Model evaluation

“ML model building, evaluation and inference with large datasets is a much bigger problem than any other task in data science when large datasets are involved”

1. Do you agree or disagree? Why
2. Outline the ML process and use it to answer question below
3. Which part of the ML pipeline do you think requires more compute resources and why?
4. Big computer vision models: Training the AlexNet computer vision model. How many parameters does the model have? How long did it take to train according to this post?
5. Big language models-GPT-3 and BERT. How many parameters does the model have? How long did it take to train? How much data was it trained on?
6. For the models you have trained in your ML course, what was the largest number of parameters? What’s the relationship between number of parameters and model training time?

Group (pairs) Activity (10 minutes)

How Data can be Big for ML

Both large number of features (p) and number of observations (n) can present problems in ML

features (p)

features (n)

How Data can be Big for ML cont’d

• In computer vision, for an input image, each pixel is considered as a feature, so, for a color images (with 3 channels), even a small image: 284 by 284 by 3 ~ 281, 968 features. Most images are huge
• In other fields such as NLP, in order to achieve better accuracy, you need to train your model on massive text corpus. For example, the popular GPT-3 trained on over 40TB of text data
• For speech recognition, audio data also takes up alot of space

Audio Data

Just 2 hours of speech data for Kinyarwanda ~ 57GB. How do you train a model on all that data?

What are the main challenges of big data in ML?

• Curse of dimensionality

​

• Can lead to overfitting

​

• Computational issues

Strategies for Dealing with Big Data in ML

Data reduction

Speed up processing

Exploit parallelization/distributed processing

Exploit specialized hardware

• Sample subset, instead of using whole dataset
• Dimensionality reduction
• Feature selection

• By using faster algorithms such as stochastic gradient descent instead of gradient descent
• Use efficient data structures
• Creating parallelizable algorithms

• Parallelize model training on a single computer
• Scale up with multiple computers (distributed) training

• Use GPU instead of CPU based processor
• Google has TPU’s

Model reduction

• Compress model during serving

Data Reduction to Deal with Big Data in ML

Downsample

Feature Engineering and/or feature selection

Often the first thing you wanna try before you invest in expensive and/or sophisticated resources/approaches

Dimensionality Reduction, lower dimension embedding

• Instead of using all n-observations, use a subset

• Select a subset of features
• Use domain knowledge to select impactful features
• Use model performance to do feature selection

• Reduce dimensionality of feature space from p to anything smaller than p
• Traditionally use PCA but t-SNE has better performance in some use cases
• Can use other embedding techniques based on data to move from higher dimension to lower dimension.
• For example, in NLP vector representation of words would enable us translate high dimension space to low dimension if that's what we want

The obvious disadvantage to these approaches is that they lead to loss of data which can lead to overfitting and poor model performance.

Speeding up Model Training with Faster Algorithms

• Stochastic gradient descent instead of batch gradient descent

​

• In some cases, usually computer vision, you have no choice but to use mini-batches because the full data cannot even fit in memory

​

• However, SGD is used eleven in cases where data can fit in memory

​

• SGD isn't perfect but it still works, see this post for some mathematical explanation

In Deep Learning Models, ‘batch_size’ is one of the key hyperparameters

In addition to affecting how fast your training will go (or not go at all), batch_size also affect model performance

Speeding Up Model Training with Better Data Structures

• As we know, data structures are very important in computer science. For instance, in distributed processing we mentioned that immutable data structures are prefered over mutable ones.
• As Data Scientist, you need to be able to pick appropriate data structures for the problem at hand. For example, in some cases, a Python list though slower will work better than a dictionary.
• In ML, distributed frameworks such as spark have different data structures for encoding variables to speed up processing
• For the mathematically inclined and curious, see this master thesis from CMU

Speeding up Model Training Specialized Hardware: CPU vs. GPU

• For traditional ML models, training on CPU perform is okay. However, for deep learning models, training on CPU is almost a non-starter if you have a couple of GB of data
• GPU’s can speed up performance by over 70% when compared to CPUs
• Tech giants such as Google are working on even more specialised processors such as Tensor Processing Units (TPU) which are available in Colab to accelerate machine learning tasks
• Given above, although GPU’s are expensive, simple choice of GPU over CPU makes dramatic difference in model training.

• What are GPUs?
• Why do GPUs work better than CPU’s for ML tasks?
• What are price comparisons between GPU and CPU processors?
• Use the following articles: 1, 2 and 3 to explore performance differences in ML model deployment training and inference when using GPU vs. CPU
• What’s the most astounding performance difference you have seen?
• Does the performance difference apply to both model training and inference?
• What about the effect of model architectures on the performance difference?

Group (pairs) Activity: Explore performance benchmarks between GPU vs. CPU

Distributed ML model training

• Which parallelization: Data vs. model parallelism

​

• Communication strategy across worker nodes: centralized vs. decentralized

​

• Parameter update strategy: synchronous vs. asynchronous

In distributed training the workload to train a model is split up and shared among worker nodes. There are several design decisions to be tackled

Data vs. Model Parallelism

Data Parallelism

Model Parallelism

• Model need to have independent parts of computations which can run in parallel
• For a deep learning, a layer (or group of layers) of the model is deployed on one node of a cluster,
• For traditional models, similar approach can be used on different parts of the model

• It’s the easiest
• The entire model is deployed to multiple nodes of a cluster and the data is shared (horizontally split). Each instance of the model works on a part of the data

In both data and model parallelism, communication across nodes during training can be centralized or decentralized. Secondly, parameter update strategy can be synchronous or asynchronous

You can also have a combination of both implemented

Data Parallelism: View with Multiple Workers

Data Parallelism View with Linear Regression Computation

Model Parallelism View with Linear Regression Computation

Model and Data Parallelism View with Linear Regression Computation

More on Model Parallelism

• Leads to development of distributed version of algorithms. For example, distributed linear regression, distributed random forest

​

• For optimization, it also leads to distributed version of algorithms such as distributed gradient descent or distributed stochastic gradient descent. Same for back propagation in neural networks

​

• In different optimization algorithms, distribution brings about several mathematical and computational challenges such as stale gradients. So, it's still active areas of research for those who want to stay Mathematical but still do machine learning

Decentralized vs. Centralized Communication in Distributed Training

• How the worker nodes regarding parameter/weight updates is crucial. For instance, parameters are initialized, how the weights/biases are updated etc.
• In a centralized communication, there exists a node or group of nodes responsible for synchronizing the model parameters, this node is called a parameter server.
• The advantage centralized training is that it’s easy to synchronize the model parameters. However, the parameter server can itself become a bottleneck for a huge cluster (due to communication) and also a single point of failure
• In the decentralized communication, each node communicates with every other node to update the parameters. The advantage with this approach is that peer-peer updates are faster, sparse updates can be made by exchanging only what has changed and there is no single point of failure

Decentralized vs. Centralized Training

Centralized training with a parameter server

Decentralized training

Synchronous and Asynchronous Parameter Updates

• This applies to both model and data parallelism setup but our explanation will make use of the data parallelism scenario
• Can be also viewed as asynchronous vs. synchronous gradient updates
• Recall that during model training, once the gradients of loss function are available, the updated weights are computed.
• In data distributed training, since the gradients are being computed on different slices of data,they need to be combined somehow so that at the next step, the workers use updated weights.
• In synchronously parameter update strategy: all worker nodes need to receive updated weights/parameters (this is done through averaging of gradients) before proceeding to next iteration. In other words, parameter updates only happen when all worker nodes have sent their parameters which are then combined and distributed to all learners, and the next batch of training begins.

Synchronous Parameter Updates in a Data Parallelism Setup

​

Compute gradients on worker nodes, once finished, average them and then update model params

Synchronous Parameter Updates in a Data Parallelism Setup

​

The process of combing (adding gradients) before averaging (during averaging phase) is a non-trivial reduce operation, like the one we looked at in the map, reduce computation strategies with distributed processing.

Check here and here to see differences between reduce, ring all reduce and all reduce. Ring-all-reduce is often preferred, see details here

Synchronous Parameter Updates Cont’d

• This synchronously parameter update strategy is good for model accuracy but reduces cluster utilization.
• However, there are strategies to reduce the disadvantages of this strategy by implementing hybrid type of strategies
• For instance, instead of waiting for all worker nodes, only wait for k
• Alternatively, update at the end of K-batches of data, instead of k workers.

Asynchronous vs. Synchronous Parameter Updates Cont’d

• In asynchronously parameter update strategy: instead of waiting for all the nodes to send the weight updates, they are sent as soon as they are available. So, only combine parameters which are available.
• This increases the cluster utilization and improves the training speed but of course leads to stale gradients problem.
• Most of the frameworks which implement asynchronous updates apply some strategies to reduce the impact in favor of higher cluster utilization
• In asynchronously parameter update strategy, there is often a parameter server which manages the parameters and communication with worker nodes. Worker nodes sends updates to this server but also fetches updates.

Asynchronous Parameter Updates in a Data Parallelism Setup

​

What about model inference? Any challenges with large datasets/models?

Data processing/preprocessing

Model building

Model serving/inference

Feature engineering

Hyperparameter tuning

Model evaluation

When models can reach 800GB in size, we need solutions for model inference too

This mostly an issue for deep learning/NN models and where you are doing inference matters

Model compression techniques

• Quantization (low-precision inference)

• Pruning

• Knowledge distillation

• Efficient architectures (+ architecture search)

Main solution for this is Model compression whose goal is to find a smaller, cheaper model with the same (or similar) accuracy

​

Model Compression: Quantization (Low precision training/inference)

• Mostly used for NN because they get very large
• Simple technique: use low-precision arithmetic in inference
• Can make any signals in the model low precision
• How to determine the ‘right’ amount of precision?
• One heuristic: keep lowering the precision of signals until the accuracy decreases
• Extreme example: binarization (1-bit {-1,1} or {0,1})

​

Model Compression: Quantization (Low precision training/inference)

Again, for the mathematically inclined, this is another good area of research: if you can find low precision training/inference and still maintain good accuracy, you are the king

In deep learning software such as Pytorch, this often termed mixed precision training. It's one of the parameters you have to deal with when training with GPUs and/or CPUs. See this blog post and this one

Other Model Compression Techniques

• Pruning. Remove ‘low impact’ weights or activations
• Reduces model size
• Technical issues include when and how often to prune, picking correct weight threshold to remove and more
• Knowledge distillation. Take a large/complex model and train a smaller network to match its output with the hope that it’s easier to learn from the model’s output [the teacher] than the true labels.
• Efficient architectures. Some neural network architectures are designed to be efficient at inference time

​

Here is a paper on knowledge distillation which I find amusing

Contents

• The ML process/pipeline
• How data can be big for ML
• Dealing with big data in ML
• Data reduction, preliminary optimization
• Distributed ML algorithms, model training
• Model compression techniques
• Scalable deep learning/machine learning
• Tooling/resources for big data in ML
• ML with spark

Tooling/resources for big data in ML

1. Standard ML, statistics and data science tools
1. How to effectively select features, reduce dimensionality, utilize better data structures and more.
2. Frameworks for implementing distributed machine learning
• Hadoop MapReduce (still in the running), Apache Spark, Baidu All Reduce, Horovod, Caffe2, Microsoft Cognitive Tool Kit (CNTK), DistBelief, Tensorflow, Pytorch, DIANNE, MXNet, Petumm
• See this blog for details

Tooling/resources for big data in ML cont’d

• Parallelized ML training on a single machine. Most ML packages support some form of parallelization
• In most cases, this parallelization doesn't run/come by default
• In Python sklearn, you can set number of cores as a parameters
• For Pytorch, Tensoflow, you can install your package with GPU support
• During training with GPU, for Tensorflow and Pyttorch you need to put extra steps to enable parallel training
• Cloud compute resources for distributed ML
• AWS, Microsoft Azure, Google Cloud Platform (GCP) all have numerous offerings from specialized hardware (e.g., TPU for Google) and different versions of GPU’s for AWS to specialised software framework (e.g., AWS Sagemaker

Overview of ML in Spark

Introducing Spark MLlib

• Its the de facto machine learning library in Apache Spark
• It solves some of the challenges associated with building ML models with large datasets by providing a framework for distributed ML and feature engineering.
• It provides several functionalities through out the ML model building life cycle
• Feature preparation (Featurization). For instance, converting from categorical to numerical using One-hot encoding
• Transforming the data into a format that Spark MLlib consumes
• Fit models, build estimators and evaluate models
• Save, load and serve models

You can use DataFrame API or RDD

Available Functions and Utilities in Spark MLlib

• ML Algorithms: common learning algorithms such as classification, regression, clustering, and collaborative filtering
• Featurization: feature extraction, transformation, dimensionality reduction, and selection
• Pipelines: tools for constructing, evaluating, and tuning ML Pipelines
• Persistence: saving and load algorithms, models, and Pipelines
• Utilities: linear algebra, statistics, data handling, etc.

​

From Spark MLlib documentation page

Main Components of Spark MLlib API

• Transformer-For data preparation and also inference
• Takes a DataFrame as input, and returns a new DataFrame with one or more columns appended to it.
• Transformers do not learn any parameters from your data and simply apply rule-based transformations to either prepare data for model training or generate predictions using a trained MLlib model.
• They have a .transform() method.
• Estimator-you can think of this as the model object as it earns (or “fits”) parameters from your DataFrame via a .fit() method and returns a model, which is a transformer.
• Pipeline-Organizes a series of transformers and estimators into a single model.

Further Resources

1. Machine Learning System Design- A must read for any student who wants to pass DS interviews
2. Spark MLlib documentation for a hello world example
3. A Guide to Scaling ML Systems in Production
4. Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
5. Explore ML research areas from the world’s best deep learning conference: NIPS