JVM in Container Die Hard

Antonin Stefanutti

@astefanut

JVM in Container Die Hard / @astefanut / #JBCNConf

Why Run a JVM into a Container?

JVM in Container Die Hard / @astefanut / #JBCNConf

For development...

• Highly portable packaging solution - for microservices, Web apps, ...
• Lightweight, encapsulated OS abstraction - carry your OS with you
• Getting started instantly:�$ docker run -it openjdk:8u141
• No more waiting N weeks for a VM to be provisioned by ops just so you can run a series of tests
• Dev environments that more closely match prod environments (no more… “but it works on my machine”)

3

JVM in Container Die Hard / @astefanut / #JBCNConf

… And Operations

• Process isolation
• Immutable images
• Image layers
• Faster deployments
• You can run multiple versions of application dependencies
• Shared bins/libs and host operating system make containers lighter
• Higher utilisation

4

JVM in Container Die Hard / @astefanut / #JBCNConf

5

Density FTW!

memory

CPU

JVM in Container Die Hard / @astefanut / #JBCNConf

Action!

JVM in Container Die Hard / @astefanut / #JBCNConf

Action!

$ docker run -it \� -v `pwd`/target:/target -p 8080:8080� -m=XXXM --memory-swap=XXXM \� openjdk:8u171 \� java -jar /target/spring-boot-web.jar

​

$ curl -w '\n' http://localhost:8080/

7

JVM in Container Die Hard / @astefanut / #JBCNConf

BOOM!

8

8GB

4 cores

64GB

16 cores

Developer Workstation

Production Server

But it works on my machine!#*?

JVM in Container Die Hard / @astefanut / #JBCNConf

Behind the Scene

JVM in Container Die Hard / @astefanut / #JBCNConf

Cgroups and Namespaces

Cgroups

• cpu (cpu shares)
• cpuacct
• cpuset (limit processes to a CPU)
• memory (swap, dirty pages)
• blkio (throttle reads/writes)
• devices
• net_prio (packet class and priority)

Namespaces

• pid (processes)
• net (network interfaces, routing)
• ipc (system V ipc)
• mnt (mount points, filesystems)
• uts (hostname)
• user (UIDs)

10

JVM in Container Die Hard / @astefanut / #JBCNConf

OOM Killer

11

--oom-kill-disable

--oom-score-adj

badness_for_task = total_vm_for_task / (sqrt(cpu_time_in_seconds) *�sqrt(sqrt(cpu_time_in_minutes)))

out_of_memory

oom_kill

select_bad_process

badness

int_sqrt

oom_kill_task

force_sig

force_sig_info

https://www.kernel.org/doc/gorman/html/understand/understand016.html

CAP_SYS_RAWIO?

SIGTERM

SIGKILL

JVM in Container Die Hard / @astefanut / #JBCNConf

The JVM

JVM in Container Die Hard / @astefanut / #JBCNConf

Why does the JVM die hard?

It relies on the host resources, not the container’s!

​

System.out.println("Memory: " + Runtime.getRuntime().maxMemory());

​

System.out.println("CPUs: " + Runtime.getRuntime().availableProcessors());

13

JVM in Container Die Hard / @astefanut / #JBCNConf

JVM Ergonomics

14

• Heap
• JIT (compiler, code)
• Garbage collector

​

-XX:+PrintFlagsFinal

-XX:NativeMemoryTracking=summary

-XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics

• Threads (stack trace)
• Metaspace (class metadata)
• Symbols

JVM in Container Die Hard / @astefanut / #JBCNConf

Memory

JVM in Container Die Hard / @astefanut / #JBCNConf

Memory

16

# Run without resource restrictions�$ docker run --rm -it openjdk:9 jshell

# Run with resource restrictions

$ docker run --rm -it -m=256M openjdk:9 jshell

​

JVM in Container Die Hard / @astefanut / #JBCNConf

17

# Run with JVM configuration

$ docker run --rm -it -m=256M openjdk:9 jshell \

-J-XX:+UnlockExperimentalVMOptions \

-J-XX:+UseCGroupMemoryLimitForHeap \

-R-XX:+UnlockExperimentalVMOptions \

-R-XX:+UseCGroupMemoryLimitForHeap

​

Available in JDK 8u131+ and JDK 9

https://bugs.openjdk.java.net/browse/JDK-8170888

​

JVM in Container Die Hard / @astefanut / #JBCNConf

Even free fails!

18

$ docker run --rm -it \

-m=256M

openjdk:9 \

free -h

JVM in Container Die Hard / @astefanut / #JBCNConf

Yeah?

19

$ docker run --rm -it \

-v `pwd`/target:/target -p 8080:8080 \

-m=190M --memory-swap=190M \

openjdk:8u171 java \

-XX:+UnlockExperimentalVMOptions \

-XX:+UseCGroupMemoryLimitForHeap \

-jar /target/spring-boot-web.jar

​

$ curl -w '\n' http://localhost:8080/

JVM in Container Die Hard / @astefanut / #JBCNConf

OOMKilled!

20

$ ab -c 25 -n 10000 http://localhost:8080/

JVM in Container Die Hard / @astefanut / #JBCNConf

Cores

JVM in Container Die Hard / @astefanut / #JBCNConf

22

# Run without resource restrictions�$ docker run --rm -it openjdk:9 jshell

# Run with resource restrictions

$ docker run --rm -it --cpuset-cpus=0 openjdk:9 jshell

​

JVM in Container Die Hard / @astefanut / #JBCNConf

Yeah?

23

$ docker run --rm -it \

-v `pwd`/target:/target -p 8080:8080 \

-m=190M --memory-swap=190M --cpuset-cpus=0 \

openjdk:8u171 java \

-XX:+UnlockExperimentalVMOptions \

-XX:+UseCGroupMemoryLimitForHeap \

-jar /target/spring-boot-web.jar

​

$ ab -c 25 -n 10000 http://localhost:8080/

JVM in Container Die Hard / @astefanut / #JBCNConf

Kubernetes :(

24

$ docker run --rm -it \� --cpu-quota=50000 --cpu-period=100000 \� openjdk:9 jshell

​

$ docker run --rm -it --cpus=1.0 openjdk:9 jshell

$ docker run --rm -it \� --cpu-shares=<weight> \� openjdk:9 jshell

JVM in Container Die Hard / @astefanut / #JBCNConf

Java 10

25

• Supports cpu-shares and cpu-quota / cpu-period
• Supports limits for physical memory (not only heap)
• Deprecates -XX:UseCGroupMemoryLimitForHeap experimental flag
• Adds -XX:ActiveProcessorCount VM flag
• Adds -XX:UseContainerSupport option
• Adds -XX:InitialRAMPercentage, -XX:MinRAMPercentage, -XX:MaxRAMPercentage options

JVM in Container Die Hard / @astefanut / #JBCNConf

Yeah!

26

$ docker run --rm -it \

-v `pwd`/target:/target -p 8080:8080 \

-m=190M --memory-swap=190M --cpus=1 \

openjdk:10 java \

-jar /target/spring-boot-web.jar

​

$ ab -c 25 -n 10000 http://localhost:8080/

JVM in Container Die Hard / @astefanut / #JBCNConf

Workaround

27

• -XX:ParallelGCThreads
• -XX:ConcGCThreads
• -XX:CICompilerCount
• -Djava.util.concurrent.ForkJoinPool.common.parallelism

​

https://github.com/fabric8io-images/java

https://github.com/fabric8io-images/run-java-sh

JVM in Container Die Hard / @astefanut / #JBCNConf

Optimisation

JVM in Container Die Hard / @astefanut / #JBCNConf

Optimisation

29

• Adjust the heap size with -Xmx or -XX:MaxRAMPercentage to preserve enough available non-heap memory
• Either:
• Set -Xms or -XX:MinRAMPercentage for fail-fast behaviour
• Or set -XX:MinHeapFreeRatio=20 and -XX:MaxHeapFreeRatio=40 options to instruct the heap to shrink aggressively and to grow conservatively
• Disable C2 JIT compiler: -XX:TieredStopAtLevel=1
• Application tuning

JVM in Container Die Hard / @astefanut / #JBCNConf

Optimisation

30

$ docker run --rm -it \

-v `pwd`/target:/target -p 8080:8080 \

-m=100M --memory-swap=100M --cpus=1 \

openjdk:10 java \

-Xms15m -Xmx15m -XX:TieredStopAtLevel=1 \

-Dspring.application.json='{"server": {"tomcat":{"max-threads": 1, "min-spare-threads": 1}}}' \

-jar /target/spring-boot-web.jar

​

$ ab -c 25 -n 10000 http://localhost:8080/

JVM in Container Die Hard / @astefanut / #JBCNConf

Going Further

JVM in Container Die Hard / @astefanut / #JBCNConf

Going Further

32

• Smaller images:
• openjdk:10-slim (581MB)
• openjdk:10-jre-slim (286MB)
• OpenJDK Portola project: JDK port to Alpine Linux
• openjdk:8-jre-alpine (82MB), updated with JDK11 EA
• Custom Java runtimes with jlink
• Application Class Data Sharing (AppCDS)
• Container metrics API (JDK-8203359, JMX, -XshowSettings:system)

JVM in Container Die Hard / @astefanut / #JBCNConf

Thank you!

Antonin Stefanutti

@astefanut

JVM in Container Die Hard / @astefanut / #JBCNConf

/sys/fs/cgroup/memory

34

cgroup.clone_children memory.kmem.tcp.limit_in_bytes memory.move_charge_at_immigrate

cgroup.event_control memory.kmem.tcp.max_usage_in_bytes memory.oom_control

cgroup.procs memory.kmem.tcp.usage_in_bytes memory.pressure_level

memory.failcnt memory.kmem.usage_in_bytes memory.soft_limit_in_bytes

memory.force_empty memory.limit_in_bytes memory.stat

memory.kmem.failcnt memory.max_usage_in_bytes memory.swappiness

memory.kmem.limit_in_bytes memory.memsw.failcnt memory.usage_in_bytes

memory.kmem.max_usage_in_bytes memory.memsw.limit_in_bytes memory.use_hierarchy

memory.kmem.slabinfo memory.memsw.max_usage_in_bytes notify_on_release

memory.kmem.tcp.failcnt memory.memsw.usage_in_bytes tasks

JVM in Container Die Hard / @astefanut / #JBCNConf