Why you’re going to FAIL running Java on docker!

​

bit.ly/javadockerfail

​

@burrsutter @rafabene

1

Change History

0.1 - JBCNConf.com 2017 - Barcelona

0.2 - JavaOne 2017 - San Francisco

0.3 - JavaDay UA 2017 - Kiev

@burrsutter

Big Wins for Developers

• Highly Portable Packaging solution - for microservices, web apps
• Lightweight, Encapsulated OS abstraction - carry your OS with you
• Getting Started (docker run -it centos/wildfly) Instantly
• Dev Environments that more closely match Prod Environments
• Dev Environments that match OTHER Dev Environments (no more...but it works on my machine)
• No more waiting 3+ weeks for a VM to be provisioned by Ops just so you can run a series of tests

Linux Container

@burrsutter

@burrsutter

Virtualization vs Containers

Server Machine

Host Operating System

Hypervisor

Guest

OS

Guest

OS

Guest

OS

Bins

Libs

Bins

Libs

Bins

Libs

App 1

App 2

App 3

Server Machine

Host Operating System

Bins

Libs

Bins

Libs

App 1

App 2

App 3

Docker

• Shared bins/libs and Host Operating System make containers lighter
• You can run multiple versions of app dependencies (Java 6 and 7)
• “Golden Image” deployment model
• Process isolation

Pros vs Cons

Pros:

• Docker containers start very fast, take less memory
• Super easy to try out a new technology (e.g docker pull centos)
• Supports the “immutable image” deployment model
• It caches layers to make builds faster

Cons:

• It is not a VM - it is a process running within the Host OS, do not assume you can run any flavor of container OS on any host OS
• It is not perfectly portable - you will eventually wish to get out of boot2docker-vm into a real Linux VM - one that more closely matches your production environment
• Lineage of docker images at Docker Hub is unknown

​

History of Containers

2000

2010

2005

2015

2000: �JAILS ADDED �TO FREEBSD

2006: �GENERIC PROCESS CONTAINERS

2008: �KERNEL AND USER NAMESPACES

2014: �GOOGLE KUBERNETES

2008: �LINUX CONTAINER PROJECT (LXC)

2015: �STANDARDS VIA OCI AND CNCF

2013: �RED HAT ENTERPRISE LINUX

2013: �DOTCLOUD BECOMES DOCKER

2007: �GPC RENAMED CONTROL GROUPS

2003: �SELINUX ADDED TO LINUX MAINLINE

2015: �RHT CONTAINER PLATFORM

2001: �LINUX -VSERVER PROJECT

2013: �DOTCLOUD PYCON LIGHTNING TALK

2005: �FULL RELEASE OF SOLARIS ZONES

History of Java

Java 1.0.2 - 30 May 1996

J2EE 1.2 - 12 Dec 1999

https://en.wikipedia.org/wiki/Java_EE_version_history#J2EE_1.2_.28December_12.2C_1999.29

https://en.wikipedia.org/wiki/Java_version_history

​

A Challenge

Server Hardware

Operating System

Application Server

.war or .ear

Java Virtual Machine

Custom Configuration

Linux Kernel Version & Distribution

Java 1.6.6_45 or Java 1.7.0_67

Weblogic 10.x.y, Tomcat 6.x.y, JBoss EAP 6.x.y

JDBC driver, datasource, JMS queue, users

Have you ever had “/” vs “\” break your app? Or perhaps needed a unique version of a JDBC driver? Or had a datasource with a slightly misspelled JNDI name? Or received a patch for the JVM or app server that broke your code?

Containerize

Your

App

Email

MyApp.war has been tested with the following

On my Windows 7 desktop

JDK 1.8.43

Wildfly 9

Configuration:

Datasource: MySQLDS

Tested with: mysql-connector-java-5.1.31-bin.jar

Production Environment

​

Red Hat Enterprise Linux 6.2

​

JRE 1.7.3

​

WebSphere 8.5.5

​

Oracle 9

​

Dockerfile

Container Guest OS

Java App Server

custom configuration

FROM centos/wildfly�

​

�COPY standalone.xml /opt/wildfly/standalone/configuration/�

COPY mysql-connector-java-5.1.31-bin.jar /opt/wildfly/standalone/deployments/�COPY mysql-sample-ds.xml /opt/wildfly/standalone/deployments/�

�COPY myapp/target/your.war /opt/wildfly/standalone/deployments/

your.war

dependencies

Note: There are better ways to handle Java apps, this is for illustration purposes

Behind the Scenes

@burrsutter

Cgroups and Namespaces

Cgroups

• cpu (cpu shares)
• cpuacct
• cpuset (limit processes to a CPU)
• memory (swap, dirty pages)
• blkio (throttle reads/writes)
• devices
• net_prio (packet class and priority)
• freezer

​

Namespaces

• pid (processes)
• net (network interfaces, routing)
• ipc (system V ipc)
• mnt (mount points, filesystems)
• uts (hostname)
• user (UIDs)

​

/sys/fs/cgroup/memory

​

cgroup.clone_children memory.kmem.tcp.limit_in_bytes memory.move_charge_at_immigrate

cgroup.event_control memory.kmem.tcp.max_usage_in_bytes memory.oom_control

cgroup.procs memory.kmem.tcp.usage_in_bytes memory.pressure_level

memory.failcnt memory.kmem.usage_in_bytes memory.soft_limit_in_bytes

memory.force_empty memory.limit_in_bytes memory.stat

memory.kmem.failcnt memory.max_usage_in_bytes memory.swappiness

memory.kmem.limit_in_bytes memory.memsw.failcnt memory.usage_in_bytes

memory.kmem.max_usage_in_bytes memory.memsw.limit_in_bytes memory.use_hierarchy

memory.kmem.slabinfo memory.memsw.max_usage_in_bytes notify_on_release

memory.kmem.tcp.failcnt memory.memsw.usage_in_bytes tasks

​

@burrsutter

DevOps Challenges for Multiple Containers

• How to scale?
• How to avoid port conflicts?
• How to manage them on multiple hosts?
• What happens if a host has trouble?
• How to keep them running?
• How to update them?
• Where are my containers?

@burrsutter

• Greek for “Helmsman,” also the root of the word “Governor” (from latin: gubernator)
• Container orchestrator
• Supports multiple cloud and bare-metal environments
• Inspired by Google’s experience with containers
• Open source, written in Go
• Manage applications, not machines

Meet Kubernetes

SCM�(Git/Svn)

Master

API Server

Service Layer

Node

Node

Logger

CI/CD

Automation

Registry

SDN Overlay Network

Controllers

- Scheduler

- Deployments

- Services

- Builds

- Routes

- DeploymentConfigs

Kubernetes

OpenShift

- Builds

- ImageStreams

SCM�(Git/Svn)

Master

API Server

Service Layer

Node

Node

Logger

CI/CD

Automation

Registry

SDN Overlay Network

Controllers

- Scheduler

- Deployments

- Services

- Builds

- Routes

- DeploymentConfigs

Kubernetes

OpenShift

- Builds

- ImageStreams

SCM�(Git/Svn)

Master

API Server

Service Layer

Node

Node

Logger

CI/CD

Automation

Registry

SDN Overlay Network

Controllers

- Scheduler

- Deployments

- Services

- Builds

- Routes

- DeploymentConfigs

Kubernetes

OpenShift

- Builds

- ImageStreams

Java

@burrsutter

Why Fail?

8GB

4 cores

64GB

16 cores

Developer Workstation

Production Server

Why does Java Fail?

It sees ALL the HOST resources, it is blind to cgroups by default

System.out.println("Memory: " + Runtime.getRuntime().maxMemory()/1024/1024);

​

System.out.println("CPUs: " + Runtime.getRuntime().availableProcessors());

​

​

​

Note: free also fails�https://fabiokung.com/2014/03/13/memory-inside-linux-containers/

@burrsutter

@burrsutter

Who uses those APIs?

Tomcat, Wildfly, Vert.x

Tomcat

https://github.com/apache/tomcat/blob/trunk/java/org/apache/catalina/core/ContainerBase.java#L304

https://github.com/apache/tomcat/blob/trunk/java/org/apache/tomcat/util/net/NioEndpoint.java#L138

https://github.com/apache/tomcat/blob/trunk/java/org/apache/tomcat/websocket/AsyncChannelGroupUtil.java#L85

https://github.com/apache/tomcat/blob/trunk/java/org/apache/catalina/manager/StatusTransformer.java#L224

Wildfly

https://github.com/wildfly/wildfly-common/blob/master/src/main/java/org/wildfly/common/cpu/ProcessorInfo.java#L61

Vert.x

https://github.com/eclipse/vert.x/blob/master/src/main/java/io/vertx/core/VertxOptions.java#L41

​

​

@burrsutter

JVM Ergonomics

https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/ergonomics.html

Memory (max heap defaults to ¼ of available memory)

JIT Optimizations

Mode: Client or Server

Thread Management

Garbage Collector

​

Do NOT just “java -jar myVertxApp.jar” nor “java -jar mySpringBootApp.jar”

Do try “java -XX:+PrintFlagsFinal -XX:+PrintGCDetails $JAVA_OPTIONS -jar myApp.jar”

@burrsutter

Memory

@burrsutter

JVM Memory: Heap is about 50%

Heap (-Xms -Xmx)�------------------------------

Native JRE

Perm (-XX:MaxPermSize), Java8 Meta (-XX:SurvivorRatio, -XX:MaxNewSize, -XX:NewRatio)

JIT Bytecode

JNI

NIO

Threads (defaults to 512K to 1024K depending on platform)�

Tip: Either -Xmx at 50% of container constrained memory or�-XX:MaxRam=500m - your heap will be approximately 250mb *

@burrsutter

@burrsutter

Workaround

docker pull fabric8/java-jboss-openjdk8-jdk�

https://hub.docker.com/r/fabric8/java-jboss-openjdk8-jdk/

​

https://github.com/fabric8io-images/java/blob/master/images/jboss/openjdk8/jdk/container-limits

max_memory() {� # High number which is the max limit until which memory is supposed to be� # unbounded. � local max_mem_unbounded="$(cat /sys/fs/cgroup/memory/memory.memsw.limit_in_bytes)"� local mem_file="/sys/fs/cgroup/memory/memory.limit_in_bytes"� if [ -r "${mem_file}" ]; then� local max_mem="$(cat ${mem_file})"� if [ ${max_mem} -lt ${max_mem_unbounded} ]; then� echo "${max_mem}"� fi� fi�}

​

@burrsutter

Heap Patched: Java 8u131, included in Java 9

https://bugs.openjdk.java.net/browse/JDK-8170888

​

@burrsutter

Try it yourself

Just need Docker for Mac or Windows

docker run -m 100MB openjdk:8u121 java -XshowSettings:vm -version

​

vs

​

docker run -m 100MB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm -version

​

UseCGroupMemoryLimitForHeap tells the JVM to look to the cgroup’s value for memory in /sys/fs/cgroup/memory/memory.limit_in_bytes

@burrsutter

Cores

@burrsutter

Docker CPU Settings

docker run --cpus="1.5" -m 400M openjdk:8u131

*docker run --cpu-quota=150000 --cpu-period=100000 -m 400M openjdk:8u131

docker run --cpu-shares=1536 -m 400M openjdk:8u131

**docker run --cpuset-cpus=0,2 -m 400M openjdk:8u131

​

https://docs.docker.com/engine/admin/resource_constraints/#configure-the-default-cfs-scheduler

​

* Kubernetes uses this model

** JVM properly calculates availableProcessors()

​

@burrsutter

Docker CPU Mapped to Java

Some “rules of thumb”

docker run -it --cpus="2" -m 200M java

java -XX:ParallelGCThreads=2

-XX:ConcGCThreads=2

-Djava.util.concurrent.ForkJoinPool.common.parallelism=2

-XX:MaxRam=200m

​

// Half a core cpu-quota / cpu-period

docker run -it --cpu-quota=50000 --cpu-period=100000 -m 200M openjdk:8u131�java -XX:+UseSerialGC -Xmx=100m

​

Note: Set Max heap (-Xmx) to about 50% of total container memory

Note: UseSerialGC when core count is below 2

@burrsutter

CPU Set Patched

https://bugs.openjdk.java.net/browse/JDK-6515172

@burrsutter

Demo Boom

https://github.com/burrsutter/javadockerfail

Demo Script

https://docs.google.com/document/d/1Td_nnjk12Miq6SpRs3RysDZbpQUjpxXk_p0J526VhcE/edit?usp=sharing

@burrsutter

Hot Tips

https://hub.docker.com/r/fabric8/java-jboss-openjdk8-jdk/

Base Docker Image

FROM fabric8/java-jboss-openjdk8-jdk:1.2.7

​

Maven Plugin:

• Docker builds leveraging base image
• Deploys to Kubernetes and OpenShift easily

https://fabric8.io/gitbook/mavenPlugin.html

mvn io.fabric8:fabric8-maven-plugin:3.3.5:setup

mvn fabric8:deploy

@burrsutter

More and More Resources

https://developers.redhat.com/products/cdk/download/

https://github.com/burrsutter/javadockerfail

https://developers.redhat.com/blog/2017/03/14/java-inside-docker/

https://developers.redhat.com/blog/2017/04/04/openjdk-and-containers/

https://dzone.com/articles/running-a-jvm-in-a-container-without-getting-kille

https://github.com/redhat-developer-demos/java-container

https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-limits-are-run

https://github.com/fabric8io-images/java/tree/master/images/jboss/openjdk8/jdk

https://github.com/fabric8io-images/java/blob/master/images/jboss/openjdk8/jdk/container-limits

http://cs.oswego.edu/pipermail/concurrency-interest/2016-March/015014.html

bit.ly/javadockerfail

@burrsutter

Why you’re going to FAIL running Java on docker!

​

bit.ly/javadockerfail

​

@burrsutter @rafabene

52

Previously Used Slides

@burrsutter

Creative Commons/Flikr, SF History Center, SF Public Library, GG Bridge 1935-1936 at http://www.onlyinyourstate.com/northern-california/san-francisco/life-san-francisco-1935/

Creative Commons/Flikr, SF History Center, SF Public Library, GG & Bay Bridges 1935 at http://www.onlyinyourstate.com/northern-california/san-francisco/life-san-francisco-1935/