How to use TensorRT C++ API for High Performance GPU Inference

A VCV Presentation by Cyrus Behroozi

Code: https://github.com/cyrusbehr/tensorrt-cpp-api

Hi, I’m Cyrus!

• Senior software developer at Trueface
• SDK Team Lead
• Skilled in packaging & deploying machine learning inference solutions

​

What we will go over today

• Installing TensorRT on Ubuntu 20.04
• Generating a TRT engine file optimized for your GPU
• Specifying a simple optimization profile
• Reading / writing data from / into GPU memory
• Synchronous inference
• Models with dynamic batch sizes

​

​

• Switching between multiple optimization profiles at runtime
• Asynchronous inference
• Cuda streams

​

What we will not go over today

Why use TensorRT?

• Best inference framework for NVIDIA GPUs
• Speed
• Memory

Motivation for making this presentation

TensorRT docs are not user friendly

18 lines of code

5 compiler errors

Installing TensorRT

• Download TensorRT 8.x tarball
• https://developer.nvidia.com/nvidia-tensorrt-8x-download

​

Set up the CMakeLists.txt to link TensorRT

Note: the above is not a complete CMakeLists.txt file

​

TensorRT workflow

Implementation overview

Avoid regenerating the engine file when not necessary

Build phase

​

Build phase 2

​

Loading the engine from disk

Running inference

NHWC to NCHW conversion

• NHWC: For each pixel, the three values are stored together in RGB order.
• NCHW: All the R channel values are stored first, then the G, then B.
• OpenCV is NHWC, TensorRT expects NCHW

NHWC to NCHW conversion cont.

​

Running inference cont.

Fin.

Any questions?

​

​

​

​

​

Code: https://github.com/cyrusbehr/tensorrt-cpp-api