1 of 47

Supporting GPUs in

Docker Containers on Apache Mesos

MesosCon Europe - 2016

​

Kevin Klues

Senior Software Engineer

Mesosphere

Yubo Li

Staff Researcher

IBM Research China

2 of 47

Kevin Klues

Yubo Li

Kevin Klues is a Senior Software Engineer at Mesosphere working on the Mesos core team. Since joining Mesosphere, Kevin has been involved in the design and implementation of a number of Mesos’s core subsystems, including GPU isolation and Pods. When not working, you can usually find Kevin on a snowboard or up in the mountains in some capacity or another.

Dr. Yubo Li is a Staff Researcher at IBM Research, China. He is the architect of the GPU acceleration and deep-learning as a service (Dlaas) components of SuperVessel, an open-access cloud running OpenStack on OpenPOWER machines. He is currently working on GPU support for several cloud container technologies, including Mesos, Kubernetes, Marathon and Open Stack.

Email: liyubobj@cn.ibm.com

Slack: @liyubobj

Email: klueska@mesosphere.com

Slack: @klueska

3 of 47

Why GPUs?

  • GPUs are the tool of choice for many big-data cloud applications
    • Deep Learning
    • Natural Language Processing
    • Genome Sequencing

3

4 of 47

Why GPUs?

4

Credit: www.slideshare.net/DataScienceMD/deep-learning-with-gpus

5 of 47

Why GPUs?

  • Mesos users have been asking for GPU support for years
    • First email asking for it can be found in the dev-list archives from 2011
    • The request rate has increased dramatically in the last 9-12 months

5

6 of 47

Why GPUs?

  • Without direct support for GPUs, there is no isolation guarantee
    • No built-in coordination to restrict access to GPUs
    • Possible for multiple frameworks / tasks to access GPUs at the same time

​

​

​

    • Ad-hoc solutions have emerged to give access to GPUs, but they all rely on co-operative scheduling rather than true isolation.

6

7 of 47

Why GPUs?

  • Enterprise users currently partition clusters to make use of GPUs
    • They’d like to consolidate them
    • Not willing to do so without isolation guarantees

7

Unified Job

Scheduler

8 of 47

Why GPUs?

8

METRONOME

(Batch)

MARATHON

(Long Running)

ON-PREMISE

GPU-Accelerated

Node

CLOUD

GPU-Accelerated

Node

Source: www.nvidia.com/object/apache-mesos.html

9 of 47

Why Docker?

  • Extremely popular image format for containers
    • Build once → run everywhere
    • Configure once → run anything

9

Source: DockerCon 2016 Keynote by Docker’s CEO Ben Golub

Image Pulls

Dockerized Applications

10 of 47

Why Docker?

Nvidia-docker

Wrapper around docker to allow GPUs to be used inside docker containers

10

Source: https://devblogs.nvidia.com/parallelforall/nvidia-docker-gpu-server-application-deployment-made-easy/

11 of 47

Why Docker?

Machine Learning Frameworks

Support exists for many popular machine learning frameworks with nvidia-docker (including TensorFlow, Caffe, CNTK, etc.)

11

Source: https://data-shaker.com/docker-tensorflow-with-jupyter-notebook-on-windows/

12 of 47

Overall Goal

Test locally with nvidia-docker

​

Deploy to production with Mesos

​

12

13 of 47

Overview of Talk

  • Challenges of supporting Nvidia GPUs in docker containers
  • How nvidia-docker addresses these challenges
  • How Apache Mesos addresses these challenges
  • Isolation Demo
  • Demo from IBM

13

14 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • Before containers it was easy

​

14

15 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • Before containers it was easy
    • Buy some GPUs

15

16 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • Before containers it was easy
    • Buy some GPUs
    • Install them on your box

16

Linux Kernel

17 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • Before containers it was easy
    • Buy some GPUs
    • Install them on your box
    • Install the base nvidia drivers

17

nvidia base libraries

Linux Kernel

nvidia-kernel-module

18 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • Before containers it was easy
    • Buy some GPUs
    • Install them on your box
    • Install the base nvidia drivers
    • Install some advanced�toolkit libraries

18

nvidia base libraries

CUDA / TensorFlow libraries

Linux Kernel

nvidia-kernel-module

19 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • Before containers it was easy
    • Buy some GPUs
    • Install them on your box
    • Install the base nvidia drivers
    • Install some advanced toolkit libraries
    • Link a GPU accelerated application�against these libraries

19

nvidia base libraries

CUDA / TensorFlow libraries

Application

Linux Kernel

nvidia-kernel-module

20 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • Before containers it was easy
    • Buy some GPUs
    • Install them on your box
    • Install the base nvidia drivers
    • Install some advanced toolkit libraries
    • Link a GPU accelerated application�against these libraries
    • Run your application

20

nvidia base libraries

CUDA / TensorFlow libraries

Application

Linux Kernel

nvidia-kernel-module

21 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • So what about containers?
    • Buy some GPUs
    • Install them on your box
    • Install the nvidia-kernel-modules

21

Linux Kernel

nvidia-kernel-module

22 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • So what about containers?
    • Buy some GPUs
    • Install them on your box
    • Install the nvidia-kernel-module
    • Build a docker image
      • Bundle the base nvidia libraries
      • Bundle some advanced toolkit libraries
      • Bundle a GPU accelerated�application to use these libraries

22

Container

Linux Kernel

nvidia-kernel-module

nvidia base libraries

CUDA / TensorFlow libraries

Application

23 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • So what about containers?
    • Buy some GPUs
    • Install them on your box
    • Install the nvidia-kernel-module
    • Build a docker image
      • Bundle the base nvidia libraries
      • Bundle some advanced toolkit libraries
      • Bundle a GPU accelerated�application to use these libraries
    • Run your docker container

23

Container

Linux Kernel

nvidia-kernel-module

nvidia base libraries

CUDA / TensorFlow libraries

Application

24 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • Straightforward, right?

24

Container

Linux Kernel

nvidia-kernel-module

nvidia base libraries

CUDA / TensorFlow libraries

Application

25 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • Will only work if the kernel / user driver versions match

25

Linux Kernel

nvidia-kernel-module (v1)

Linux Kernel

nvidia-kernel-module (v2)

Container

nvidia base libraries (v1)

CUDA / TensorFlow libraries

Application

26 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • Won’t work if they don’t

26

Linux Kernel

nvidia-kernel-module (v1)

Linux Kernel

nvidia-kernel-module (v2)

Container

nvidia base libraries (v1)

CUDA / TensorFlow libraries

Application

27 of 47

Challenges of Supporting Nvidia GPUs in Docker containers

  • Either way, you have to map in the GPU devices somehow

27

Linux Kernel

nvidia-kernel-module (v1)

Linux Kernel

nvidia-kernel-module (v2)

Container

nvidia base libraries (v1)

CUDA / TensorFlow libraries

Application

28 of 47

nvidia-docker and GPUs

  • Components of nvidia-docker
    • Set of docker images that set custom labels / environment variables
    • nvidia-docker-plugin ( standard docker volume plugin)
    • nvidia-docker (wrapper script around docker itself)

28

docker run ...

​

nvidia-docker run ...

29 of 47

nvidia-docker and GPUs

  • nvidia-docker-plugin

​

Finds all standard nvidia libraries / binaries on the host and consolidates them into a single place as a docker volume

​

/var/lib/docker/volumes

└── nvidia_XXX.XX (version number)

├── bin

├── lib

└── lib64

​

​

29

30 of 47

nvidia-docker and GPUs

  • nvidia-docker wrapper script

Looks for the label:

com.nvidia.volumes.needed = nvidia_driver

​

When found, it maps the nvidia_XXX.XX volume into the container at:

/usr/local/nvidia

​

Enumerates all GPUs on the machine and maps them into the container as available devices

​

Passes all other docker options straight through to docker

30

31 of 47

nvidia-docker and GPUs

31

Linux Kernel

nvidia-kernel-module (v1)

Linux Kernel

nvidia-kernel-module (v2)

Container

nvidia base libraries (v1)

CUDA / TensorFlow libraries

Application

nvidia base libraries (v2)

Container

CUDA / TensorFlow libraries

Application

32 of 47

nvidia-docker and GPUs

32

Linux Kernel

nvidia-kernel-module (v1)

Linux Kernel

nvidia-kernel-module (v2)

Container

nvidia base libraries (v1)

CUDA / TensorFlow libraries

Application

nvidia base libraries (v2)

Container

CUDA / TensorFlow libraries

Application

33 of 47

Apache Mesos and GPUs

  • Mimics functionality of nvidia-docker
    • Supports nvidia docker images with custom labels
    • Maps consolidated volume of binaries / libraries into /usr/local/nvidia
    • Enumerates GPUs and injects them into containers

​

  • Additionally isolates access to GPUs between tasks
    • Not possible with nvidia-docker alone

33

34 of 47

Apache Mesos and GPUs

  • Multiple Containerizer support
    • Mesos (aka unified) containerizer (fully supported)
    • Docker containerizer (currently being worked on)

​

  • Why support both?
    • Many people are asking for Docker containerizer support�to bridge the feature gap
    • People are already familiar with existing docker tools
    • Unified containerizer needs time to mature

34

35 of 47

Apache Mesos and GPUs

  • GPU_RESOURCES framework capability
    • Frameworks must opt-in to receive offers with GPU resources
    • Prevents legacy frameworks from consuming non-GPU resources�and starving out GPU jobs

​

  • Use agent attributes to select specific type of GPU resources
    • Agents advertise the type of GPUs they have installed via attributes
    • Only accept an offer if the attributes match the GPU type you want

35

36 of 47

Apache Mesos and GPUs

36

(Unified) Mesos Containerizer

Containerizer API

Mesos Agent

Isolator API

CPU

Memory

37 of 47

Apache Mesos and GPUs

​

37

(Unified) Mesos Containerizer

Containerizer API

Mesos Agent

Isolator API

CPU

Memory

GPU

38 of 47

Apache Mesos and GPUs

​

38

(Unified) Mesos Containerizer

Containerizer API

Mesos Agent

Isolator API

CPU

Memory

GPU

Nvidia GPU Isolator

Linux devices cgroup

Nvidia GPU

Allocator

39 of 47

Apache Mesos and GPUs

​

39

(Unified) Mesos Containerizer

Containerizer API

Mesos Agent

Isolator API

CPU

Memory

GPU

Nvidia GPU Isolator

Linux devices cgroup

Nvidia GPU

Allocator

Nvidia Volume

Manager

Mimics functionality of nvidia-docker-plugin

40 of 47

Apache Mesos and GPUs

​

40

(Unified) Mesos Containerizer

Containerizer API

Mesos Agent

Isolator API

CPU

Memory

GPU

Linux devices cgroup

Nvidia GPU

Allocator

Nvidia Volume

Manager

Nvidia GPU Isolator

41 of 47

Apache Mesos and GPUs

​

41

Docker Containerizer

Containerizer API

Mesos Agent

Isolator API

CPU

Memory

(Unified) Mesos Containerizer

GPU

Composing Containerizer

Nvidia GPU

Allocator

Nvidia Volume

Manager

GPU

42 of 47

Implementation Timeline

  • Support for the Unified containerizer
    • Supports both image-less and docker-image based containers�(fully compatible with nvidia-docker)
    • Supported in the recent Mesos 1.0.0 release (with patch fixes in 1.0.1)
    • Supported in upcoming Marathon 1.3 release (coming in next few weeks)
    • Targeted for DC/OS 1.9 release ( mid October)

​

  • Support for the Docker Containerizer
    • Designed and partially implemented
    • Target Mesos release: 1.1.0
    • Target Marathon release: 1.4
    • Target DC/OS release: 1.10

42

43 of 47

Looking forward

  • Expose Underlying GPU Topology
    • Include in offers sent to schedulers
    • Help make more informed decision about which GPUs to choose

​

  • Add support for other GPU / accelerator types
    • AMD, Intel, Custom FPGAs, etc.

​

43

44 of 47

Special Thanks to All Collaborators

44

​

​

Vikram Ditya

Andrew Iles

Jonathan Calmels

Felix Abecassis

Rob Todd

Rajat Phull

Shivi Fotedar

Seetharami Seelam

Yong Feng

Guangya Liu

Ian Downes

Niklas Nielson

Connor Doyle

Benjamin Mahler

Tim Chen

45 of 47

ISOLATION

DEMO

45

https://github.com/klueska-mesosphere/mesos-gpu-docker

46 of 47

46

47 of 47

DEMO

FROM IBM

47