Production Ready
Cloud-Native Services
with Go and Kubernetes

Elena Grahovac & Daniel Mahlow

GopherCon Europe 2019

What should be prepared before the workshop begins

Agenda, part 1

09:00 - Warm Up
09:15 - Introduction. What we are going to achieve today
09:30 - Theory. How to be Cloud Native?
10:00 - Practice. Write a Go service from scratch
11:00 - Break & catch up (15 min)
11:20 - Practice. Continue preparing service
12:45 - Prepare your env for Kubernetes
13:00 - Lunch

Agenda, part 2

14:00 - Hello, Kubernetes!
30 - How we are going to deploy the services
14:45 - Dockerization
15:15 - Helm configuration
15:30 - Break & catch up (15 min)
15:45 - Configure GitHub integration to run CI/CD
16:00 - Deploy services
16:30 - Discuss questions and additional topics

Cleanup Notes

  • The cluster will be available for a couple of weeks
  • On June, 14th the cluster and all the data will be deleted
  • If you want us to delete your profiles and services earlier, please let us know

Lead TechOps Automation Engineer @N26

DevOps Enthusiast



Co-Founder & TechOps lead @ Contiamo



“Hello, World”

“Production ready”

How to be Cloud-Native?

  • Operate managed infrastructure
  • Develop isolated containerized services
  • Communicate through network
  • Follow microservice design principles
  • Use DevOps techniques



People and culture

Design and development practices

Cloud-Native Requirements

  • 12 factor
  • Fault tolerance
  • Reliability
  • Maintainability
  • Operability
  • Observability
  • Security
  • Automation

Production Readiness

These and other wonderful gophers:

Four-layer model of the microservice ecosystem

by Susan J. Fowler, Production-Ready Microservices

Production Readiness and Kubernetes

Structuring in Practice

Structuring in Practice

  • Flat
  • Layer-based
  • Module-based
  • Context-based
  • “Hexagonal”

Kat Zień - How do you structure your Go apps:

Structuring in Practice

  • cmd
  • internal
  • vendor

Structuring in Practice

cmd for small and simple entry points

If I have cmd/myctl and cmd/myserver
I can simply do go get cmd/… to have myctl and myserver binaries inside $GOPATH/bin

Structuring in Practice

internal for packages that should be available for current project only

If I don’t want a package being available for import by other projects, I’ll store it under internal

See also: ‘Go 1.4 Internal Packages’ -

Structuring in Practice

vendor to store dependencies :)

Other possible ideas (for inspiration):

Let’s start

Dependency Management

Dependencies: problem definition

  • Design & quality
  • Testing
  • Activity & maintenance
  • Licenses
  • Integrity & dependencies
  • Immutability & updates

Dependencies checklist

  • Description
  • Documentation
  • Go Report Card
  • Issues & pull requests
  • Code coverage
  • Other reports
  • Repeat the same for dependencies of this dependency

Dependencies: go mod cheatsheet

  • GO111MODULE=on go mod init
  • GO111MODULE=on go mod tidy
  • GO111MODULE=on go mod download
  • GO111MODULE=on go mod vendor


  • Availability
  • Independency
  • Immutability
  • Archives are faster than git repos
  • Additional opportunities


  • Workflows around module management
  • Current implementations

GOPROXY - Additional task *

  • Try to use go mod with GOPROXY, e.g.
    GOPROXY= go mod tidy

Code Quality

This and other wonderful gophers:

Code Quality


  • Testing
  • Static code analyzers
  • Profiling

Code Quality: Testing

If you love the standard library and hate external dependencies:

func TestDoSomething(t *testing.T) {

// "want" is your expected result

have, err := mypkg.DoSomething()

if err != nil {
t.Errorf("%v", err)
} else if !reflect.DeepEqual(have, want) {
t.Errorf("have %+v, want %+v", have, want)


Code Quality: Testing

If you love to “assert”:

func TestDoSomething(t *testing.T) {

// "want" is your expected result

have, err := mypkg.DoSomething()

require.NoError(t, err)
assert.Equal(t, want, have)

Code Quality: Testing *

Additional topics:

Fuzzy testing *

  • My application works with a complex input
  • I validate it
  • I want to make sure that my validation is good enough

Fuzzy testing can help you to check if your validation is good enough

Code Quality: Analysis

Static code analyzers:

  • Check code style
  • Detect potential bugs
  • Scan for security problems

Code Quality: Analysis


Code Quality: Analysis *

When Develop/CI/CD-ing:

  • Local hooks (pre-commit)
  • Server hooks (push, pull request)
  • Separated Docker image
  • Continuous Integration stage

When one should run linters?

Code Quality: Runtime *


  • Simply import net/http/pprof to enable profiling handlers
  • Human-friendly interface: http://localhost:8080/debug/pprof
  • Tip: try to check out new features

Code Quality: Runtime *

package main

import (



_ "net/http/pprof"


func main() {

log.Println(http.ListenAndServe(":8080", nil))


Hello, pprof!

Observability & Operability

This and other wonderful gophers:



  • Log errors or strange situations!
  • Log “stages” of your application
  • Try multiple log levels
  • Try “tags” to classify your logs
  • Tip: if something goes wrong, check performance of your logger


Hello, logs!

package main

import (
log ""

func main() {
"animal": "walrus",
}).Info("A walrus appears")

Observability *


  • Define, measure and report business and SLA metrics
  • Business metrics might be useful as well!


  • Tracing is a technique that shows how a single request goes through your service or services

How to start? Check out OpenCensus:


Health checks:

  • “healthy/unhealthy”: application state, business and performance metrics
  • “readiness”: report if application is ready to handle requests

Operability: Diagnostics Server *

Prepare a dedicated http.Server to handle health checks and pprof endpoints, it will listen to some other port (e.g. we can set it via DIAG_PORT).

Graceful Shutdown

  • Recover panics
  • Don’t panic, return errors
  • Handle errors
  • Listen to OS signals
  • Provide shutdown for your functional units



Use the linker to provide a version, commit hash or any other data to identify the version:
go build -ldflags "-X main.version=put_version_here"

package main

import (

var (
version = "unset"

func main() {
fmt.Printf("The version is: %s\n", version)


  • Set configuration via env
  • Try as a systems solution
  • Keep secrets safe
  • Prepare a systems solution if you need to deal with secrets on the application level


Repeatable Actions

  • Define repeatable actions and prepare a tool to call them fast:
    • The actions you call when CI/CD-ing
    • Checkers and tests
    • Application building
    • Dealing with container images
  • GNU Make as a classic approach
  • A fancy alternative:

Automate it *

  • Prepare a typical “Cloud-Native Hello World” service
  • Define templates based on it
  • Use code generation to produce new services from the templates
  • Consider the details specific for your infrastructure or project



  • Compile a binary on the CI/CD node & copy it
  • Multi-staging builds

Simplest “mono-staging” image

FROM scratch


COPY bin/myapp /myapp


CMD ["myapp"]

Multi-staging builds: GOPROXY

# Initial stage: download modules

FROM artifactory/golang:1.12

ADD go.mod go.sum /m/

RUN cd /m && go mod download

While go.mod and go.sum are not changed, the stage will be cached

Multi-staging builds: Toolchain

# Intermediate stage: Build the binary

FROM artifactory/golang:1.12 as builder

# add a non-privileged user

RUN useradd -u 10001 myapp

RUN mkdir -p /go/src/

ADD . /go/src/

WORKDIR /go/src/

# Build the binary with go build

RUN CGO_ENABLED=0 go build \

-o bin/myapp

# Stage 1. Run the binary

Multi-staging builds: The binary

# Final stage: Run the binary

FROM scratch


# certificates to interact with other services

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

# don't forget /etc/passwd from previous stage

COPY --from=0 /etc/passwd /etc/passwd

USER myapp

# and finally the binary

COPY --from=0 /go/src/ /myapp


CMD ["myapp"]

Let’s deploy!

This and other wonderful gophers:

Hello, Kubernetes!

Prepare kubectl

Deploy “Hello, World!”

  • Let’s try a “Hello Application” example:
    kubectl run hello-app \
    -n ${NAMESPACE}
  • To check Deployment and Pod:
    kubectl get deployments -n ${NAMESPACE}
    kubectl get pods -n ${NAMESPACE}
  • Create a Service resource:
    kubectl expose deployment hello-app -n ${NAMESPACE}
  • Check it:
    kubectl get services -n ${NAMESPACE}

Make “Hello, World!” accessible from Internet

  • Create a manifest to define Ingress resource (store it as a file):
    - see the next slide -

apiVersion: extensions/v1beta1

kind: Ingress


name: hello-app-ingress

namespace: rumyantseva

annotations: nginx /$2



- host:



- path: /rumyantseva/hello(/|$)(.*)


serviceName: hello-app

servicePort: 8080


- hosts:


secretName: tls-secret

Make “Hello, World!” accessible from Internet

  • Create an Ingress resource (from manifest):
    kubectl apply -f ./hello-app-ingress.yaml
  • Check how it works:
    curl -i${namespace}/hello

If you prefer UI

  • You can login with the token you got at
  • Please note that your permissions allows you to work with resources within your namespace
  • Set your namespace - github username (lowercased) - in the field on the left to navigate to your namespace

Let’s prepare charts

Ready for integration

  • Install GitHub App which will listen to the events in your repo:
  • Push something to master
  • Check the results
  • If everything is ok, your app should be available within your namespace
  • If something is wrong, check the logs, fix the problem and try again


These and other wonderful gophers:

Additional resources to try and play



  • Think about structure before you start writing the application
  • Write unit tests
  • Try TDD to make your structure better and define how separated modules should interact to each other
  • Check your code not only for its style but also for potential bugs and security problems


  • Add a profiler before you released your first version to have the profile prepared as soon as you need it
  • Log errors, stages and events produced by your application
  • Define, measure and report metrics
  • Try tracing
  • Shutdown gracefully
  • Design configuration in advance


  • Keep version info inside the build
  • Manage dependencies
  • Think about security when containerize
  • Separate a container where you check the code and a container where you run the application
  • Define and describe repeatable actions
  • Automate

Bonus Topic: a bit of Security

These and other wonderful gophers:

The role of Security in DevOps


Examples of Security Input

Define Way of Working

General Requirements, Policies


Threat Modelling, Design Guidelines


Coding Rules, Codereview, Tools


Testing Plan, Toolkits


Configuration, Checklists


Incident Management, Scanning, Pen Test


Our applications interact with a lot of sensitive data internally: a database password, or an API key are as much important as user data. As we can see from this example, some people don’t really care about their secrets.


...Other people forget to clean their git history

Secrets: Kubernetes case study

  • How the data of secrets are stored?
  • Who can access and modify secrets?
  • Do you use secrets as ENV parameters?

So, I mustn’t store my secrets in source code.

Secrets: Kubernetes case study

  • How the data of secrets are stored?
  • Who can access and modify secrets?
  • Do you use secrets as ENV parameters?

So, I mustn’t store my secrets in source code...

Secrets: Solutions

Security: References and additional reading

Useful tools to check out