Android CI with

Anton Malinskiy

Development Manager & Principal Engineer, Agoda

Why CI/CD?

Phase 1: project init

Phase 2: more people

Phase 3: more teams

Phase 4+: more people

The Problem

A

B

C

develop/master

Git log of a large team

CI/CD

A

B

C

1. Prerequisites
2. Deployment of CI
3. Unit testing
4. Instrumentation testing
5. Self-healing & monitoring

Prerequisites

Prerequisites: software

• Docker image with build, test & deploy dependencies
• JDK
• gradle
• android-sdk
• etc (static analysis, custom scripts...)

Prerequisites: hardware

• Bare-bone kubernetes cluster
• usb devices
• full control over the hardware is needed
• KVM

Prerequisites: knowledge

• Familiar with kubernetes entities like
• Pod/Deployment
• Service
• Ingress
• etc
• Package manager, e.g. helm

Deployment of CI/CD

CI/CD installation on kubernetes

• TeamCity/Jenkins
• Server
• Agents
• Docker registry
• Nexus
• Artifactory
• etc

CI/CD pipeline for infrastructure

• Same pipeline for infrastructure as for the app

CI/CD pipeline for infrastructure

Deployment of CI image

• Scale via helm upgrade
• Scale dynamically e.g. Horizontal Pod Autoscaler
• Plugins for CI systems, e.g. teamcity-kubernetes-plugin

Unit testing

Unit testing

• jUnit
• Parallelizing via maxParallelForks
• forkEvery as a stability fix

Unit testing

• Robolectric
• Slow ClassLoader initialization (e.g. @Config’s minsdk or maxsdk)

Unit testing

• Mockito
• Mocks cache not reused between JVM forks
• Reported test time for Robolectric is broken

Unit testing: JVM considerations

​

​

• Xmx/Xms

Unit testing: JVM considerations

​

​

• Xmx/Xms
• Forking the JVM

Unit testing: JVM considerations

​

​

• Xmx/Xms
• Forking the JVM
• jUnit’s maxParallelForks, forkEvery
• kotlin.compiler.execution.strategy="out-of-process"
• gradle worker processes

Unit testing: JVM considerations

​

​

• UnlockExperimentalVMOptions
• UseCGroupMemoryLimitForHeap
• MinRAMFraction
• MaxRAMFraction
• Test different garbage collectors
• SerialGC
• ConcMarkSweepGC

Unit testing: monitoring

​

​

What can you do with monitoring?

Unit testing: G1GC, OOM crash

​

​

Unit testing: ConcMarkSweep, maxParallelForks=1

​

​

Unit testing: ConcMarkSweep, maxParallelForks=4

​

​

Unit testing: SerialGC, maxParallelForks=4

​

​

Application deployment

• Publishing API
• gradle-play-publisher as a wrapper
• no support for client side validation
• Lot’s of retries
• Kubernetes secrets
• signing config
• credentials
• etc

Instrumented tests

The problem

Agent

Agent

Agent

Agent

Device

?

Device

?

Emulator?

Emulator?

?

Android connections

console 5554

adb 5555

Android

telnet ip console

adb connect ip:adb

adbd

adbd 5037

Real devices

• Dedicated nodes with USB devices
• DaemonSet with privileged container with access to:
• /dev/bus/usb
• /sys/bus/usb
• /sys/devices

kubectl label nodes node-x.cluster.local openstf.io/provider=true

​

kubectl taint nodes node-x.cluster.local openstf.io/provider=true:NoSchedule

​

Emulators

Android requires custom kernel => virtualization

• Linux
• KVM
• Privileged container

kubectl label nodes node-x.cluster.local openstf.io/emulator=kvm

​

Emulators: qemu

Android options:

• adb+console ports
• memory
• screen multi-touch
• camera-back/front none
• skip-adb-auth
• ….

qemu-system-i386 $ANDROID_OPTS -qemu $QEMU_OPTS

​

​

Emulators: config.ini

• hw.cpu.arch
• hw.cpu.ncore
• hw.gpu.mode
• hw.lcd.density
• hw.lcd.height
• hw.lcd.width
• ...

ANDROID_CONFIG=” \

skin.name=600x1024;\

hw.lcd.density=160; \

hw.lcd.height=600; \

hw.lcd.width=1024; \

hw.device.name=7in WSVGA (Tablet); \

avd.ini.displayname=7 WSVGA (Tablet) API 23;”

Emulators: GPU rendering

• auto
• host
• angle_indirect (windows)
• swiftshader_indirect
• off

Emulators: image size

Main contributors:

• userdata.img (~1Gi)
• system.img (~3Gi)

tar.gz

Emulators: unique id trick

Default mode

TCP/IP mode

$ adb devices

List of devices attached

​

emulator-5554 device

$ adb devices

List of devices attached

​

10.2.5.177:5555 device

emulator-5554 device

​

Emulators: additional tips

• x86 is faster than x86_64
• VNC is available for the whole container X system
• Multi-touch support is enabled by default

The problem

Agent

Agent

Agent

Agent

Device

Device

Emulator

Emulator

?

Instrumented tests: OpenSTF

• Unified scalable architecture for providing Android devices
• WebUI
• Backed by RethinkDB
• Remote adb connections
• VNC-like connections
• + many more

Instrumented tests: OpenSTF

• Deployment
• Scaling
• Monitoring
• Self-healing
• Upgrades
• Rollbacks

​

​

Connecting to remote devices

Agent

Agent

Agent

Agent

Device

Device

Emulator

Emulator

?

Connecting to remote devices

• Connects via TCP/IP
• Not stable

• Wraps adb connect
• Multiple devices support
• Daemon mode
• External metrics
• Custom notes support

adb connect 10.0.0.1:5555

stf-client connect -n 30 -f sdk:25

stf-client connect -n 4 -f notes:tablet

Connecting to remote devices

Connecting to remote devices

Agent

?

Agent

?

Agent�?

Agent�?

Device

Device

Emulator

Emulator

Test runner

Test runner

• Sharding doesn’t really work in current runners
• Assumption: test duration is equal for all tests

Test runner

• Sharding doesn’t really work in current runners
• Assumption: test duration is equal for all tests
• Reconnect to device in real-time during test run

Test runner

• Sharding doesn’t really work in current runners
• Assumption: test duration is equal for all tests
• Reconnect to device in real-time during test run
• Rerun test on DIFFERENT device if test fails
• Prefer running flaky tests first
• Prefer running long tests first
• Batch tests in groups

Test runner

Test runner

Test runner

Test runner

Everything eventually fails

Stability: usb devices, software

• /dev/bus/usb driver needs rebind
• adb devices doesn’t return device
• adb devices returns the device in Unauthorized/Offline state
• stf-client can’t connect to device (shell health-check fails)
• ….

Stability: usb devices, hardware

• USB cable issues
• Battery discharges
• Battery fails

Stability: emulators

• adb devices returns the device in Unauthorized/Offline state
• Emulator gets too hungry for resources
• Emulator’s adb server takes over

Stability: remote connection

• stf-client can’t connect to device (shell health-check fails)
• network issues
• ...

Monitoring

Metrics are collected from

• adb-butler (side-container)
• Devices in adb devices output
• stf-client
• Failed connections
• openstf
• Battery temperature
• Present devices

Monitoring

Monitoring

Monitoring

Performance statistics: unit tests

Performance statistics: instrumented tests

Licensing issue

• Can’t redistribute Android SDK
• Not feasible to compile from AOSP

​

make PROXY=registry-url/ build tag push

Links

• https://github.com/agoda-com/android-farm
• https://github.com/agoda-com/adb-butler
• https://github.com/agoda-com/docker-emulator-android
• https://github.com/agoda-com/fork
• https://github.com/Malinskiy/stf-client
• https://github.com/Malinskiy/docker-android

​

Q&A

anton@malinskiy.com

slides: https://goo.gl/6bLWT3

​

​

Cost, 60 devices, 24/7, 1 year

Firebase test lab

​

$5 device/hour

365x24x60=525600 hours

​

​

​

Total: $2_628_000

​

​

In-house solution

​

$700 x 60 devices

$80k support engineer

$20k servers + $5k etc

​

​

Total: $147_000

​

¢28 device/hour

Android CI

with