1 of 40

Why and how we moved from Jenkins to Gitlab CI

Romain Chivot & Alain Vagner

2 of 40

Intro

  • In 2018, our small team moved from Jenkins to Gitlab CI
  • Disclaimer
    • This talk is opinionated, take it with a grain of salt!
    • No majordomo was harmed during the migration ;)

3 of 40

CI? CD? CDE?

4 of 40

Sources: https://martinfowler.com/bliki/ContinuousDelivery.html

https://semaphoreci.com/blog/2017/07/27/what-is-the-difference-between-continuous-integration-continuous-deployment-and-continuous-delivery.html

​

5 of 40

  • Integrating
  • Building
  • Testing

code within the development environment.

Sources: https://martinfowler.com/bliki/ContinuousDelivery.html

https://semaphoreci.com/blog/2017/07/27/what-is-the-difference-between-continuous-integration-continuous-deployment-and-continuous-delivery.html

​

6 of 40

  • Software deployable throughout its lifecycle
  • Priority on keeping the software deployable over working on new features
  • Fast, automated feedback on the production readiness
  • Push-button deployments of any version of the software to any environment

Sources: https://martinfowler.com/bliki/ContinuousDelivery.html

https://semaphoreci.com/blog/2017/07/27/what-is-the-difference-between-continuous-integration-continuous-deployment-and-continuous-delivery.html

​

7 of 40

Benefits:

  • Reduced Deployment Risk
  • Believable Progress
  • User Feedback

Sources: https://martinfowler.com/bliki/ContinuousDelivery.html

https://semaphoreci.com/blog/2017/07/27/what-is-the-difference-between-continuous-integration-continuous-deployment-and-continuous-delivery.html

​

8 of 40

Possibility of many production deployments every day

Sources: https://martinfowler.com/bliki/ContinuousDelivery.html

https://semaphoreci.com/blog/2017/07/27/what-is-the-difference-between-continuous-integration-continuous-deployment-and-continuous-delivery.html

​

9 of 40

Sources: https://martinfowler.com/bliki/ContinuousDelivery.html

https://semaphoreci.com/blog/2017/07/27/what-is-the-difference-between-continuous-integration-continuous-deployment-and-continuous-delivery.html

​

10 of 40

11 of 40

Jenkins, what's the hell?

12 of 40

Jenkins: what it is?

  • Started in 2004 as Hudson at Sun
  • 2011 Jenkins as fork of Hudson (thanks Oracle, as usual…)
  • Done in Java / Java-oriented

13 of 40

Jenkins: what we did with it?

  • Build, test, deploy
  • Triggers based on Github
  • Notify about the status in Slack
  • Create and share release notes
  • ⚠︎ We didn’t use Jenkins Pipeline
    • Released ~2016, experimental
    • Hard to debug
    • Still needs a lot of plugins

​

source: https://jaxenter.com/jenkins-pipelines-continuous-delivery-156656.html

14 of 40

Jenkins: what were the pain points?

​

From Kohsuke Kawaguchi, creator of Jenkins in 2018:

  • Service Instability
  • Brittle Configuration
  • Assembly Required
  • Reduced Development Velocity

So he launched 2 initiatives to revamp completely Jenkins (“Jenkins X”)

Source: https://jenkins.io/blog/2018/08/31/shifting-gears/

​

15 of 40

Jenkins: what were the pain points?

​

From James Rawlings, contributor of Jenkins X:

  • Single Point of Failure
  • servers frequently run out of disk space
  • Plugin version mismatches during upgrades
  • GitHub rate limiting, caused by multibranch plugin scans
  • Large JVM requiring high memory even when no builds are running

Source: https://medium.com/@jdrawlings/serverless-jenkins-with-jenkins-x-9134cbfe6870

​

16 of 40

Jenkins: what were the pain points?

From our experience:

  • One "devops" responsible of maintaining it
  • Slow, slow and slow
  • Instability & impossible to update (plugins conflicts)
  • Lots of documentation that is not up to date
  • Configuration not versioned by default
  • Need to have dirty hacks to make some plugins working
  • Strange behaviors

​

​

​

17 of 40

Jenkins: what were the pain points?

From our experience:

  • Community seems to be shrinking
  • Runners often out of disk space
  • Jenkins = a general purpose automation tool
    • Need for plugins for every basic tasks
    • Not adapted to current workflows & development practices

​

18 of 40

Different generations

  • Remember RCS, CVS, Subversion?
  • Are you still using them?
  • Why?

19 of 40

Sounds familiar...

"Why Abstruse?

We saw many projects relying on outdated open source continuous integration (CI) solutions that were widely adopted in the past, but unfortunately cannot answer new requirements from the industry. On the other hand, commercial CI solutions have all these great features (i.e. Travis CI), but they cost money. Hence, numerous organizations decide to cut costs and go with legacy open source CI solutions or not use CI solutions at all. This kills the code quality and increases software maintenance costs."

From: https://github.com/bleenco/abstruse

​

20 of 40

21 of 40

Let's choose another tool!

22 of 40

Various CI solutions

  • Jenkins
  • TeamCity
  • Travis CI
  • Go CD
  • GitLab CI
  • CircleCI
  • Codeship
  • AWS Codebuild
  • Drone
  • Github Actions
  • ContinuousPHP
  • ...

​

​

https://en.wikipedia.org/wiki/Comparison_of_continuous_integration_software

23 of 40

Some selection criterias

  • Hosting: SaaS / Self hosted
  • Open Source? Community with critical mass?
  • NR simultaneous builds
  • Configuration management
  • Integration with your dev environment
  • Sustainability of the solution
  • Stability of the solution
  • Quality of the documentation
  • Costs

24 of 40

The "developer acceptance factor"

  • Integrate easily in the development workflow
  • Require no special skill
  • Improve collaboration among devs
  • Feeling: devs should like working with this tool
  • Collective responsibility of the build / deploy process, no ops required

25 of 40

26 of 40

Say Hello to Gitlab CI!

27 of 40

Gitlab: what it is?

  • Gitlab started in 2011: Open source clone of Github?
  • Since 2013: Gitlab CE vs Gitlab EE -> Open Core Model
  • Acquired: Gitorious, Gitter, Gemnasium
  • Stack: Ruby, Go, Vue.js

​

28 of 40

Gitlab CI

  • Pros
    • Strong integration with Github
    • Good Developer eXperience + nice documentation
    • Versioning of the conf / scripts
    • Separation of concerns: Tools in dockers, not in the instance running the CI
    • Parallel builds & caching of assets
    • Integration with Docker and AWS, autoscaling�
  • Cons
    • Need to git commit + push to test your changes on gitlab-ci file
    • Cloud version : runner availability is random
    • EE version can be expensive (19$/month/user)

​

29 of 40

Gitlab CI architecture (AWS)

30 of 40

docker-machine

  • Auto scaling is done via docker-machine available on gitlab-runner instance
  • Goal: sharing the load among a maximum number of runners
  • Docker-machine create some Docker Engine on virtual hosts
  • Docker-machine exposes some commands to manage these hosts :
    • docker-machine ls
    • docker-machine ssh
    • docker-machine stop
    • ...

​

31 of 40

gitlab-runner configuration (1)

concurrent = 12

check_interval = 0

​

[[runners]]

name = "ip-xxx"

limit = 6

url = "https://your-gitlab-instance"

token = "your_token"

executor = "docker+machine"

[runners.docker]

tls_verify = false

image = "dockerhub_account/gitlab"

privileged = true

disable_cache = false

volumes = [

"/cache",

"/var/run/docker.sock:/var/run/docker.sock"

]

​

[runners.cache]

Type = "s3"

ServerAddress = "s3.amazonaws.com"

AccessKey = "your_accesskey"

SecretKey = "your_secret_key"

BucketName = "s3-gitlab-cache"

BucketLocation = "eu-west-1"

Shared = true

32 of 40

Docker-runner configuration (2)

[runners.machine]

IdleCount = 0

IdleTime = 1800

MachineDriver = "amazonec2"

MachineName = "gitlab-%s"

MachineOptions = [

"amazonec2-access-key=your_access_key", � "amazonec2-secret-key=your_secret_key",

"amazonec2-ssh-user=ubuntu",

"amazonec2-region=eu-west-1",

"amazonec2-instance-type=c4.2xlarge",

"amazonec2-vpc-id=your_vpc_id",

"amazonec2-subnet-id=your_subnet",

"amazonec2-use-private-address=true",� "amazonec2-security-group=gitlab-runner-docker-machine",

"amazonec2-zone=c",

"amazonec2-root-size=32",

"amazonec2-request-spot-instance=true",

"amazonec2-spot-price=0.3"

]

OffPeakTimezone = ""

OffPeakIdleCount = 0

OffPeakIdleTime = 0

​

33 of 40

Gitlab pipelines

STAGES

JOBS

PIPELINE

34 of 40

gitlab-ci.yml

stages:

- build-frontends

- build-backend

- deploy

​

dev-build-frontend-1:

stage: build-frontends

environment:

name: dev

only:

- dev

cache:

key: "some_unique_key"

paths:

- $WORKSPACE/node_modules/

script: # build frontend script

artifacts:

paths:

- $WORKSPACE/build

​

dev-build-backend:

stage: build-backend

environment:

name: dev

only:

- dev

cache: *build-backend_cache # here we use a reference

script: *build-backend_script # here as well

​

​

.unit-tests # this is a “hidden key”

services:

# use a mongodb image from dockerhub

- name: mongo:3.2

command: ["mongod"]

# pipeline will not stop if unit tests are failing

allow_failure: true

script: # unit test script

​

dev-unit-tests:

<<: *unit-tests # here we inject a definition (see above)

stage: build-backend

environment:

name: dev

only:

- dev

# we don't need to download any artifact for this task

dependencies: []

​

​

dev-deploy:

stage: deploy

environment:

name: dev

only:

- dev

script: # deploy script

dependencies: []

​

35 of 40

Gitlab-ci.yml (suite)

​

  • There are a lot more features like :
    • Include other gitlab-ci.yml (include)
    • Run a script before/after all jobs (before_script, after_script)
    • Retry a job if it failed (retry)
    • Run several instances of a job in parallel (parallel)
    • ...

36 of 40

Migration

  • Around ~ 3 months done by 4 people
  • Background task, progressive migration
  • 10 projects, 4 environments
  • Migrate CI only (sync with Github repositories)
  • Migrate first non-prod env

37 of 40

Feedback

  • Faster to build/deploy
    • Jenkins (prod) > 20min
    • Gitlab (prod) ~ 14min�
  • More consistent with our stack (docker)
  • Easier to maintain and to create new jobs (copy/paste instead of interface)
  • More reliable (no disk usage issues)
  • Easier to collaborate on build scripts (git)
  • More developers are involved in the CI management

38 of 40

39 of 40

Use the right tool...

… for the job to be done!

  • Gitlab CI is no silver bullet
  • Should be possible to change of CI software when needed
  • Don't risk your velocity or code quality for the sake of some habits
  • "Developer acceptance factor" is important, especially for non-core developer skills

40 of 40

Let's discuss!