Defensive Programming for Deep Learning with PyTorch

By:

Akash Swamy

Sr.ML Engineer

Topics

1. Building a High-Level Abstraction

Build modules and using Python module to build high-level abstractions and API.

• Defining Annotations for Deep Learning/Machine Learning

Effective use of Python Typing Module for PyTorch Model Training

• Error Handling

Techniques to Handle Errors

• Unit Testing

Types of Unit Testing

​

Building a High-Level Abstraction

Building Abstractions is one of the most important aspect of any software building process. An effective High-Level Abstraction will help define modules, how modules will interact with each other and how new updates will be handled.

1. Defining PyTorch modules
2. Interaction between each PyTorch modules
3. Ingest research and updates quickly

High-Level Abstraction Cycle

Step-1 Research:

Reading research papers on the subject is the great way to understand how to write modules.

Reading several papers on Computer Vision ( CV ) will help us create a general picture of most commonly used components in building a CV model.

For Example:

“Backbone” is commonly used to define a Feature Extraction model like VGG16, ResNet etc.

​

Step-2 Define Interactions:

Defining Interactions b/w modules means how they can be connected to each other to generate a prediction model.

For Example:

Combining a “Backbone” with “Detection Head” will convert it into a Detection Model.

Combining a “Backbone” with “Classification Head” will convert it into a Classification Model.

​

Step-3 Create Modules:

Once Step-1 and Step-2 are complete then comes the implementations part where python’s standard abstraction library can be used to create PyTorch modules for each components.

​

​

PyTorch/Python Practical Example of Modularity

Defining Annotations for Deep Learning using PyTorch

Type Annotations were introduced in Python 3.0 for type hinting in python, type annotations makes it easier for developers to define the constraints and ability of the class or function to handle input.

Type Annotations have become quite popular among authors of various libraries and should be employed by anyone intending to author a library, SDK or a framework.

Type Annotations != Type Checking

Example of Type Annotation

Most common type annotations are : [str, int, float, List, Dict, Union, Tuple, Optional, Classes]

They can be mixed and matched to represent any data structure.

For example:

x : List[str] = [‘a’, ‘b’, ‘c’]

Probabilities : List[float] = [0.2, 0.2, 0.4, 0.2]

Tensors: List[torch.Tensor] = [...]

Real World Example in Object Detection using Type Annotations

Error Handling

Error Handling will not only eliminate unwanted system breakdown, it will also make the software easier to debug.