1 of 37

How do I Troubleshooting on Container, more than Docker?

Phil Huang <phil.huang@redhat.com>

Solution Architect, Red Hat

2020/06/12

Special Edition

2 of 37

# whois Phil Huang 黃秉鈞

Red Hat HK/TW Solution Architect

Ansible IT Automation
OpenShift Container Platform
Software-Defined Networking (SDN)
Network Function Virtualization (NFV)

OpenSource Community Member

Cloud Native Taiwan User Group (CNTUG)
SDNDS-TW

Personal Blog

https://blog.pichuang.com.tw

Ref: https://www.linkedin.com/in/phil-huang-09b09895/

3 of 37

Agenda

Prologue

Container is Linux
Why Podman
Why Network Namespace is Important?

Container? Docker?

Understanding of Container Networking Model/Interface
3 Useful container network mode for troubleshooting

Pod? Kubernetes? OpenShift?

Understanding of 5 OpenShift Network Traffic
How to do OpenShift network troubleshooting?

How to obtain or get a debug container?

4 of 37

Prologue

5 of 37

Containers are Linux

A linux container is nothing more than a process that runs on Linux

Linux namespaces
cgroups
SELinux
seccomp

It also mean you can use Linux performance tools to do some troubleshooting works

Ref: http://www.brendangregg.com/linuxperf.html�https://www.redhat.com/en/blog/containers-are-linux

6 of 37

Why Podman?

Support multiple image formats including the �OCI and Docker image formats
3 Benefits

Daemonless container engine
Provides a familiar command experience compatible with the Docker CLI
Build and run rootless containers as non-root

How to start?

dnf install -y podman
alias docker=podman

Ref: https://speakerdeck.com/pichuang/the-first-journey-from-docker-to-podman

7 of 37

Why Network Namespace is Important?

Container uses many Linux namespace technologies for isolation resource, such as user namspace / process / mnt / net ...
For network isolation, container uses Linux network namespace technology
Each network namespace can have its own:

Network interface
Routing tables
Firewall rules
DNS lookup
IP address
Subnets
...

Ref: http://redhatgov.io/workshops/containers_the_hard_way/

8 of 37

Docker ? Container ?

Docker Registry => Container Registry
Docker Images => Container Images
Docker Containers => OCI Containers
Dockerfile => Containerfile

Ref: https://dwalsh.fedorapeople.org/ReplacingDockerWithPodman.pdf

OCI: Open Container Initiative

9 of 37

Container Networking Interface

Why need container networking?

Allow containers to communicate to host machine
Containers need to talk to Internet
Containers can attach to multi networks

Explore the nature of communication between container resource, instead of focusing on the implementation details for specific container networking standards

Docker use Container Network Model (CNM)
Podman use Container Network Interface (CNI)

3 Useful container network mode for troubleshooting

Bridge mode
Container mode
Host mode

Ref: https://www.nuagenetworks.net/blog/container-networking-standards/

10 of 37

3 Useful Container Network Interface for Troubleshooting

11 of 37

$ man podman run

Implement 7 network modes
By default, use bridge mode

12 of 37

Bridge Mode

Create a network namespace on the default bridge

# podman run -it quay.io/pichuang/debug-container

13 of 37

Container Mode

Reuse another container’s network namespace

# podman run -it --net container:<container_name> quay.io/pichuang/debug-container

14 of 37

Host Mode

Use host network namespace

# podman run -it --net host quay.io/pichuang/debug-container

15 of 37

Kubernetes Networking Model

Kubernetes project DOES NOT HAVE a network model default implementation
There are multiple implementations in the world, all of which must follow CNI spec

The most popular CNI plugins in community: Flannel and Calico�

Explore the nature of communication between Kubernetes resource, instead of focusing on the implementation details for each of CNI plugins
5 Kubernetes network traffic
4 level debugging methods

CNI: Container Network Interface

16 of 37

Enterprise Grade Kubernetes Platform

17 of 37

Undestanding of 5 Kubernetes �Network Traffic

18 of 37

Basic Concept: 1 ~ n Containers per Pod

Pods are the smallest deployable units in Kubernetes
A Pod is a group of one or more containers with shared storage&network

19 of 37

Traffic Model: Container to Container

A network namespace provides a new network stack for all the containers per Pod
Containers within a Pod share an IP Address and port space

20 of 37

Traffic Model: Pod-to-Pod in the same node

Connecting namespaces using a linux bridge
Every Pod gets its own IP address in a flat shared networking space

21 of 37

Traffic Model: Pod-to-Pod across different nodes

Kubernetes uses iptables to handle many networking and port forwarding rules

iptables
routing tables

22 of 37

Traffic Model: Pod-to-Service

Kubernetes Service

Internal load balancer that routes the traffic to Pods

When load balancing for multiple backend pods, it uses unweighted round-robin scheduling

23 of 37

Traffic Model: Service-to-Internet

Ingress

Routing Internet traffic to Kubernetes

Egress

Routing traffic to the Internet

24 of 37

How to do OpenShift Network Troubleshooting?

25 of 37

Running Container Level Debugging

# Get a Shell to a Running Container

oc rsh welcome-1-xqbm9 -- /bin/bash

Ref: https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/

26 of 37

Running Container Level Debugging

27 of 37

Namespace Level Debugging

# Running one Pod in namespace and specific node

oc run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}}}'

Debug Pod

Namespace

28 of 37

Namespace Level Debugging

29 of 37

Node Level Debugging

# Running one Pod on specific Node

oc run ocp-debug-container --image quay.io/pichuang/debug-container --restart=Never -it --attach --rm --overrides='{ "apiVersion": "v1", "spec": { "nodeSelector":{"kubernetes.io/hostname":"compute-1"}, "hostNetwork": true}}'

Debug Pod

Container OS

30 of 37