CI/CD with docker on Gitlab

Israël

CTO @ Flare Systems

Find these slides at http://confoo.isra17.xyz

2

Why?

CI & CD with Docker on Gitlab

3

Why CI?

  • Enforce automation of integration process
    • Code health
    • Green test
    • Packaging
  • Hard truth
  • Fearless development

4

Why CD?

  • Enforce automation of deployment process
  • Fearless deployment

5

Why Docker?

  • Package your application into a controlled environment
  • Versionize those package
  • Link between Dev and Ops

6

Why Gitlab?

  • I’m not a Gitlab salesman, but;
  • Free Software
  • Free Hosting
  • Free CI
  • Quite complete overall

7

FooBar Web App

Let’s build its pipeline

8

FooBar

  • Revolutionary web app empowering the world to echo foo back at you
  • Python + Flask
  • Lives at http://gitlab.com/isra17/foobar

9

How it works

from flask import Flask
app = Flask(__name__)

@app.route(
"/")
def foobar():
return "Foobar."

import foobar

def test_foobar(client):
rv = client.get(
'/')
assert b'Foobar.' in rv.data

10

How to run and test

Run

FLASK_APP=foobar.py flask run

Test

py.test

11

The Plan

  • Install test environment
  • Run test suite
  • Build docker image
  • Deploy

12

The Plan

  • Install test environment
  • Run test suite
  • Build docker image
  • Deploy

13

The Plan

  • Build docker image
  • Run test suite
  • Deploy

14

Build the Image

15

Build Image

Dockerfile

FROM python:3.7
COPY requirements.txt foobar.py /app/
WORKDIR /app
RUN pip install -r requirements.txt
ENV FLASK_APP=/app/foobar.py
ENTRYPOINT ["flask", "run"]

16

Build Image

Dockerfile

FROM python:3.7
COPY requirements.txt foobar.py /app/
WORKDIR /app
RUN pip install -r requirements.txt
ENV FLASK_APP=/app/foobar.py
ENTRYPOINT ["flask", "run"]

17

Build Image

Dockerfile

FROM python:3.7
COPY requirements.txt foobar.py /app/
WORKDIR /app
RUN pip install -r requirements.txt
ENV FLASK_APP=/app/foobar.py
ENTRYPOINT ["flask", "run"]

18

Build Image

Dockerfile

FROM python:3.7
COPY requirements.txt foobar.py /app/
WORKDIR /app
RUN pip install -r requirements.txt
ENV FLASK_APP=/app/foobar.py
ENTRYPOINT ["flask", "run"]

19

Build Image

Dockerfile

FROM python:3.7
COPY requirements.txt foobar.py /app/
WORKDIR /app
RUN pip install -r requirements.txt
ENV FLASK_APP=/app/foobar.py
ENTRYPOINT ["flask", "run"]

20

Build Image

Dockerfile

FROM python:3.7
COPY requirements.txt foobar.py /app/
WORKDIR /app
RUN pip install -r requirements.txt
ENV FLASK_APP=/app/foobar.py
ENTRYPOINT ["flask", "run"]

21

Build Image Fast

Dockerfile

FROM python:3.7
COPY requirements.txt /app/
WORKDIR /app
RUN pip install -r requirements.txt

COPY foobar.py /app/
ENV FLASK_APP=/app/foobar.py
ENTRYPOINT ["flask", "run"]

22

Build in Gitlab

.gitlab-ci

script:

  • docker build \
    -t foobar:latest \
    -t foobar:$CI_COMMIT_REF_SLUG \
    .

23

Docker in Docker

24

Docker in Docker

.gitlab-ci

variables:

  • DOCKER_DRIVER: overlay2
  • DOCKER_HOST: tcp://docker:2375/

25

Docker in Docker

.gitlab-ci

Build Image:
...
services:

- docker:dind

26

Build in Gitlab

.gitlab-ci

script:

  • docker build \
    -t foobar:latest \
    -t foobar:$CI_COMMIT_REF_SLUG \
    .
  • ...

27

Where’s our image

28

Where’s our image

29

Where’s our image

30

It’s Gone

31

Push it Out

32

Push to Gitlab

.gitlab-ci

before_script:

- docker login -u gitlab-ci-token \
-p $CI_BUILD_TOKEN $CI_REGISTRY

33

Push to Gitlab

.gitlab-ci

variables:
CONTAINER_IMAGE: $CI_REGISTRY_IMAGE/foobar:$CI_COMMIT_SHA
script:

- docker build -t $CONTAINER_IMAGE …
- docker push $CONTAINER_IMAGE

34

It’s Out!

35

Testing

As simple as “py.test”

36

Testing

37

As simple as py.test

.gitlab-ci

Test:
stage: test
script:

- py.test

38

As simple as py.test

.gitlab-ci

Test:
stage: test
script:

- py.test

39

The image attribute

.gitlab-ci

Test:
image: foobar:$CI_COMMIT_SHA
stage: test
script:

- py.test

40

DinD Again!

.gitlab-ci
image: docker:latest

script:

- docker pull $CONTAINER_IMAGE
- docker run --rm --init $CONTAINER_IMAGE py.test
services:
- docker:dind

41

Where’s py.test?

[FATAL tini (6)] exec py.test failed: No such file or directory

42

Docker Multi-Stage

.Dockerfile
FROM python:3.7 as app

FROM app as test
COPY requirements.test.txt ./
COPY test_foobar.py ./
RUN pip install -r requirements.test.txt

43

Docker Multi-Stage

.gitlab-ci.yml
variables:
CONTAINER_TEST_IMAGE: $CI_REGISTRY/foobar-test:$CI_COMMIT

script:
- docker build -t $CONTAINER_IMAGE --target app .
- docker build -t $CONTAINER_TEST_IMAGE --target test .
- docker push $CONTAINER_IMAGE
- docker push $CONTAINER_TEST_IMAGE

44

Docker Multi-Stage

.gitlab-ci.yml

Test:
script:
- docker pull $CONTAINER_TEST_IMAGE

- docker run --rm --init $CONTAINER_TEST_IMAGE py.test

45

Tada

46

Deploy

Depend on your stack.

47

Tag docker

.gitlab-ci.yml

Deploy:
image: docker:latest
script:
- docker pull $CONTAINER_IMAGE

- docker tag $CONTAINER_IMAGE foobar:latest
- docker push foobar:latest
only: [“master”]

48

Going Further

49

Cache mean fast

  • Reuse layers as much as possible
  • Keep rare changes in first layers, frequent changes in latest
  • Dependencies should not have to be built or installed at each CI job

50

Current CI

  • New layer hash at each commit
  • No caching in DinD

51

Git mtime

  • New layer hash at each commit
    • Git clone set a new “mtime” (Modification time) on each clone
    • Docker rely on file mtime to invalidate cache
    • End up with new layer hash on each build

52

Git mtime

Set fix mtime

find ./ | grep -v .git/ | xargs touch -t 200001010000.00

53

Docker cache

  • Empty cache in DinD
    • Fill cache by pulling existing images
    • Use --cache-from option

54

Docker cache

.gitlab-ci.yml

- docker pull foobar:latest
- docker pull foobar:$CI_COMMIT_REF_SLUG
- docker build --pull
--cache-from foobar:latest
--cache-from foobar:$CI_COMMIT_REF_SLUG
...

55

Docker cache

  • Still have to pull each time
  • Use non-shared runner to extend configuration
    • Mount host’s docker.sock to executor
    • No need for Docker in Docker anymore

56

Expose host docker

Start your own runner

sudo gitlab-runner register \

--docker-volumes /var/run/docker.sock:/var/run/docker.sock
...

57

Exposed host docker

  • Unsafe for shared instance!
  • But, keep image locally between jobs
  • Modify .gitlab-ci.yml
    • Remove: services: [“docker:dind”]
    • Remove: DOCKER_HOST: tcp://docker:2375/

58

Run tests services

  • Gitlab exposts “services” key in .gitlab-ci
  • Not directly available to our DinD container
  • Use “services” property and bind port to test container

59

Run test services

.gitlab-ci.yml

services:
- postgres:9
script:
- >
docker run
--add-host postgres:$POSTGRES_PORT_5432_TCP_ADDR

60

In two or three columns

Yellow

Is the color of gold, butter and ripe lemons. In the spectrum of visible light, yellow is found between green and orange.

Blue

Is the colour of the clear sky and the deep sea. It is located between violet and green on the optical spectrum.

Red

Is the color of blood, and because of this it has historically been associated with sacrifice, danger and courage.

61

A picture is worth a thousand words

A complex idea can be conveyed with just a single still image, namely making it possible to absorb large amounts of data quickly.

62

Use diagrams to explain your ideas

63

Lorem ipsum congue tempus

Lorem ipsum tempus

Lorem ipsum congue tempus

Lorem ipsum tempus

And tables to compare data

64

A

B

C

Yellow

10

20

7

Blue

30

15

10

Orange

5

24

16

Maps

65

our office

89,526,124

Whoa! That’s a big number, aren’t you proud?

66

89,526,124$

100%

185,244 users

That’s a lot of money

Total success!

And a lot of users

67

Our process is easy

68

first

Lorem Ipsum

Lorem ipsum dolor sit amet, consectetur adipiscing.

Sit Amet

Lorem ipsum dolor sit amet, consectetur adipiscing.

second

Donec Ultrices

Lorem ipsum dolor sit amet, consectetur adipiscing.

third

Litora

Lorem ipsum dolor sit amet, consectetur adipiscing.

last

Let’s review some concepts

Yellow

Is the color of gold, butter and ripe lemons. In the spectrum of visible light, yellow is found between green and orange.

Blue

Is the colour of the clear sky and the deep sea. It is located between violet and green on the optical spectrum.

Red

Is the color of blood, and because of this it has historically been associated with sacrifice, danger and courage.

Yellow

Is the color of gold, butter and ripe lemons. In the spectrum of visible light, yellow is found between green and orange.

Blue

Is the colour of the clear sky and the deep sea. It is located between violet and green on the optical spectrum.

Red

Is the color of blood, and because of this it has historically been associated with sacrifice, danger and courage.

69

You can insert graphs from Google Sheets

70

Android project

Show and explain your web, app or software projects using these gadget templates.

71

Place your screenshot here

iPhone project

Show and explain your web, app or software projects using these gadget templates.

72

Place your screenshot here

Tablet project

Show and explain your web, app or software projects using these gadget templates.

73

Place your screenshot here

Desktop project

Show and explain your web, app or software projects using these gadget templates.

74

Place your screenshot here

Thanks!

Any questions?

You can find me at @username & user@mail.me

75

Credits

Special thanks to all the people who made and released these awesome resources for free:

  • Presentation template by SlidesCarnival
  • Photographs by Unsplash

76

Presentation design

This presentation uses the following typographies:

  • Titles: Raleway ExtraBold
  • Body copy: Raleway Light

You can download the fonts at https://www.fontsquirrel.com/fonts/raleway

77

You don’t need to keep this slide in your presentation. It’s only here to serve you as a design guide if you need to create new slides or download the fonts to edit the presentation in PowerPoint®

78

SlidesCarnival icons are editable shapes.

This means that you can:

  • Resize them without losing quality.
  • Change fill color and opacity.
  • Change line color, width and style.

Isn’t that nice? :)

Examples:

79

Now you can use any emoji as an icon!

And of course it resizes without losing quality and you can change the color.

How? Follow Google instructions https://twitter.com/googledocs/status/730087240156643328

✋👆👉👍👤👦👧👨👩👪💃🏃💑❤😂😉😋😒😭👶😸🐟🍒🍔💣📌📖🔨🎃🎈🎨🏈🏰🌏🔌🔑 and many more...

😉

Confoo 2019 - Google Slides