Production Ready
Cloud-Native Services
with Go and Kubernetes

Elena Grahovac & Daniel Mahlow

GopherCon Europe 2019

What should be prepared before the workshop begins

Agenda, part 1

09:00 - Warm Up
09:15 - Introduction. What we are going to achieve today
09:30 - Theory. How to be Cloud Native?
10:00 - Practice. Write a Go service from scratch
11:00 - Break & catch up (15 min)
11:20 - Practice. Continue preparing service
12:45 - Prepare your env for Kubernetes
13:00 - Lunch

Agenda, part 2

14:00 - Hello, Kubernetes!
14:30 - How we are going to deploy the services
14:45 - Dockerization
15:15 - Helm configuration
15:30 - Break & catch up (15 min)
15:45 - Configure GitHub integration to run CI/CD
16:00 - Deploy services
16:30 - Discuss questions and additional topics

Cleanup Notes

  • The cluster will be available for a couple of weeks
  • On June, 14th the cluster and all the data will be deleted
  • If you want us to delete your profiles and services earlier, please let us know

Lead TechOps Automation Engineer @N26

DevOps Enthusiast

​

​

hello@grahovac.pro

webdeva

rumyantseva

Co-Founder & TechOps lead @ Contiamo

​

​

dmahlow@gmail.com

danielbln

dmahlow

“Hello, World”

“Production ready”

How to be Cloud-Native?

  • Operate managed infrastructure
  • Develop isolated containerized services
  • Communicate through network
  • Follow microservice design principles
  • Use DevOps techniques

DevOps

Cloud-Native

People and culture

Design and development practices

Cloud-Native Requirements

  • 12 factor
  • Fault tolerance
  • Reliability
  • Maintainability
  • Operability
  • Observability
  • Security
  • Automation

Production Readiness

These and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Four-layer model of the microservice ecosystem

​

by Susan J. Fowler, Production-Ready Microservices

Production Readiness and Kubernetes

Structuring in Practice

Structuring in Practice

  • Flat
  • Layer-based
  • Module-based
  • Context-based
  • “Hexagonal”

​

Kat Zień - How do you structure your Go apps: https://youtu.be/VQym87o91f8

Structuring in Practice

  • cmd
  • internal
  • vendor

Structuring in Practice

cmd for small and simple entry points

If I have cmd/myctl and cmd/myserver
I can simply do go get cmd/… to have myctl and myserver binaries inside $GOPATH/bin

Structuring in Practice

internal for packages that should be available for current project only

If I don’t want a package being available for import by other projects, I’ll store it under internal

See also: ‘Go 1.4 Internal Packages’ - https://docs.google.com/document/d/1e8kOo3r51b2BWtTs_1uADIA5djfXhPT36s6eHVRIvaU/edit

Structuring in Practice

vendor to store dependencies :)

Other possible ideas (for inspiration): https://github.com/golang-standards/project-layout

Let’s start

https://github.com/rumyantseva/tenerife

Dependency Management

Dependencies: problem definition

  • Design & quality
  • Testing
  • Activity & maintenance
  • Licenses
  • Integrity & dependencies
  • Immutability & updates

Dependencies checklist

  • Description
  • Documentation
  • Go Report Card
  • Issues & pull requests
  • Code coverage
  • Other reports
  • Repeat the same for dependencies of this dependency

Dependencies: go mod cheatsheet

  • GO111MODULE=on go mod init
  • GO111MODULE=on go mod tidy
  • GO111MODULE=on go mod download
  • GO111MODULE=on go mod vendor

Wiki: https://github.com/golang/go/wiki/Modules

GOPROXY: pros

  • Availability
  • Independency
  • Immutability
  • Archives are faster than git repos
  • Additional opportunities

GOPROXY: cons

  • Workflows around module management
  • Current implementations

GOPROXY - Additional task *

  • Try to use go mod with GOPROXY, e.g.
    GOPROXY=https://gocenter.io go mod tidy

Code Quality

This and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Code Quality

Practices:

  • Testing
  • Static code analyzers
  • Profiling

Code Quality: Testing

If you love the standard library and hate external dependencies:

func TestDoSomething(t *testing.T) {

// "want" is your expected result

have, err := mypkg.DoSomething()

if err != nil {
t.Errorf("%v", err)
} else if !reflect.DeepEqual(have, want) {
t.Errorf("have %+v, want %+v", have, want)
}
}

​

Code Quality: Testing

If you love to “assert”: https://github.com/stretchr/testify

func TestDoSomething(t *testing.T) {

// "want" is your expected result

have, err := mypkg.DoSomething()

require.NoError(t, err)
assert.Equal(t, want, have)
}

​

Code Quality: Testing *

Additional topics:

Fuzzy testing *

  • My application works with a complex input
  • I validate it
  • I want to make sure that my validation is good enough
  • https://github.com/dvyukov/go-fuzz

Fuzzy testing can help you to check if your validation is good enough

Code Quality: Analysis

Static code analyzers:

  • Check code style
  • Detect potential bugs
  • Scan for security problems

​

Code Quality: Analysis

Tools:

Code Quality: Analysis *

When Develop/CI/CD-ing:

  • Local hooks (pre-commit)
  • Server hooks (push, pull request)
  • Separated Docker image
  • Continuous Integration stage

​

When one should run linters?

Code Quality: Runtime *

Profiler:

  • Simply import net/http/pprof to enable profiling handlers
  • Human-friendly interface: http://localhost:8080/debug/pprof
  • Tip: try github.com/google/pprof to check out new features

​

Code Quality: Runtime *

package main

​

import (

"log"

"net/http"

_ "net/http/pprof"

)

​

func main() {

log.Println(http.ListenAndServe(":8080", nil))

}

​

​

Hello, pprof!

Observability & Operability

This and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Observability

Logs:

  • Log errors or strange situations!
  • Log “stages” of your application
  • Try multiple log levels
  • Try “tags” to classify your logs
  • Tip: if something goes wrong, check performance of your logger

​

Observability

Hello, logs!

​

package main

import (
log "github.com/sirupsen/logrus"
)

func main() {
log.WithFields(log.Fields{
"animal": "walrus",
}).Info("A walrus appears")
}

Observability *

Metrics:

  • Define, measure and report business and SLA metrics
  • Business metrics might be useful as well!

Traces:

  • Tracing is a technique that shows how a single request goes through your service or services

How to start? Check out OpenCensus: https://opencensus.io

Operability

Health checks:

  • “healthy/unhealthy”: application state, business and performance metrics
  • “readiness”: report if application is ready to handle requests

​

​

Operability: Diagnostics Server *

Prepare a dedicated http.Server to handle health checks and pprof endpoints, it will listen to some other port (e.g. we can set it via DIAG_PORT).

Graceful Shutdown

  • Recover panics
  • Don’t panic, return errors
  • Handle errors
  • Listen to OS signals
  • Provide shutdown for your functional units

Cheatsheet: https://github.com/rumyantseva/advent-2017/blob/master/main.go#L29

Versioning

Use the linker to provide a version, commit hash or any other data to identify the version:
go build -ldflags "-X main.version=put_version_here"

package main

import (
"fmt"
)

var (
version = "unset"
)

func main() {
fmt.Printf("The version is: %s\n", version)
}

Configuration

  • Set configuration via env
  • Try github.com/kelseyhightower/envconfig as a systems solution
  • Keep secrets safe
  • Prepare a systems solution if you need to deal with secrets on the application level

Automation

Repeatable Actions

  • Define repeatable actions and prepare a tool to call them fast:
    • The actions you call when CI/CD-ing
    • Checkers and tests
    • Application building
    • Dealing with container images
  • GNU Make as a classic approach
  • A fancy alternative: https://github.com/go-task/task

Automate it *

  • Prepare a typical “Cloud-Native Hello World” service
  • Define templates based on it
  • Use code generation to produce new services from the templates
  • Consider the details specific for your infrastructure or project

For inspiration: https://github.com/takama/caldera

Dockerization

Builds

  • Compile a binary on the CI/CD node & copy it
  • Multi-staging builds

Simplest “mono-staging” image

FROM scratch

​

ENV PORT 8080

​

COPY bin/myapp /myapp

EXPOSE $PORT

​

CMD ["myapp"]

​

​

Multi-staging builds: GOPROXY

# Initial stage: download modules

FROM artifactory/golang:1.12

​

ADD go.mod go.sum /m/

RUN cd /m && go mod download

​

While go.mod and go.sum are not changed, the stage will be cached

Multi-staging builds: Toolchain

# Intermediate stage: Build the binary

FROM artifactory/golang:1.12 as builder

​

# add a non-privileged user

RUN useradd -u 10001 myapp

​

RUN mkdir -p /go/src/github.com/rumyantseva/myapp

ADD . /go/src/github.com/rumyantseva/myapp

WORKDIR /go/src/github.com/rumyantseva/myapp

​

# Build the binary with go build

RUN CGO_ENABLED=0 go build \

-o bin/myapp github.com/rumyantseva/myapp/cmd/myapp

​

# Stage 1. Run the binary

…

​

​

Multi-staging builds: The binary

# Final stage: Run the binary

FROM scratch

​

ENV PORT 8080

​

# certificates to interact with other services

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

​

# don't forget /etc/passwd from previous stage

COPY --from=0 /etc/passwd /etc/passwd

USER myapp

​

# and finally the binary

COPY --from=0 /go/src/github.com/rumyantseva/myapp/bin/myapp /myapp

EXPOSE $PORT

​

CMD ["myapp"]

​

​

Let’s deploy!

This and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Hello, Kubernetes!

https://slides.com/danielbln/kubernets-history-basics-beyond-12

Prepare kubectl

Deploy “Hello, World!”

  • Let’s try a “Hello Application” example:
    NAMESPACE=your_github_user
    kubectl run hello-app --image=gcr.io/google-samples/hello-app:1.0 \
    --port=8080     -n ${NAMESPACE}
  • To check Deployment and Pod:
    kubectl get deployments -n ${NAMESPACE}
    kubectl get pods -n ${NAMESPACE}
  • Create a Service resource:
    kubectl expose deployment hello-app -n ${NAMESPACE}
  • Check it:
    kubectl get services -n ${NAMESPACE}

Make “Hello, World!” accessible from Internet

  • Create a manifest to define Ingress resource (store it as a file):
    - see the next slide -

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

name: hello-app-ingress

namespace: rumyantseva

annotations:

kubernetes.io/ingress.class: nginx

nginx.ingress.kubernetes.io/rewrite-target: /$2

spec:

rules:

- host: services.k8s.community

http:

paths:

- path: /rumyantseva/hello(/|$)(.*)

backend:

serviceName: hello-app

servicePort: 8080

tls:

- hosts:

- services.k8s.community

secretName: tls-secret

​

Make “Hello, World!” accessible from Internet

  • Create an Ingress resource (from manifest):
    kubectl apply -f ./hello-app-ingress.yaml
  • Check how it works:
    curl -i https://services.k8s.community/${namespace}/hello

If you prefer UI

https://dash.k8s.community

  • You can login with the token you got at ui.k8s.community
  • Please note that your permissions allows you to work with resources within your namespace
  • Set your namespace - github username (lowercased) - in the field on the left to navigate to your namespace

Let’s prepare charts

https://github.com/rumyantseva/tenerife

Ready for integration

  • Install GitHub App which will listen to the events in your repo: github.com/apps/k8s
  • Push something to master
  • Check the results
  • If everything is ok, your app should be available within your namespace
  • If something is wrong, check the logs, fix the problem and try again

Summary

These and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Additional resources to try and play

  • https://keel.sh
  • https://k3s.io

Summary

  • Think about structure before you start writing the application
  • Write unit tests
  • Try TDD to make your structure better and define how separated modules should interact to each other
  • Check your code not only for its style but also for potential bugs and security problems

Summary

  • Add a profiler before you released your first version to have the profile prepared as soon as you need it
  • Log errors, stages and events produced by your application
  • Define, measure and report metrics
  • Try tracing
  • Shutdown gracefully
  • Design configuration in advance

Summary

  • Keep version info inside the build
  • Manage dependencies
  • Think about security when containerize
  • Separate a container where you check the code and a container where you run the application
  • Define and describe repeatable actions
  • Automate

Bonus Topic: a bit of Security

These and other wonderful gophers: https://github.com/ashleymcnamara/gophers

The role of Security in DevOps

Stage

Examples of Security Input

Define Way of Working

General Requirements, Policies

Design

Threat Modelling, Design Guidelines

Code

Coding Rules, Codereview, Tools

Test

Testing Plan, Toolkits

Deploy

Configuration, Checklists

Monitor

Incident Management, Scanning, Pen Test

Secrets

Our applications interact with a lot of sensitive data internally: a database password, or an API key are as much important as user data. As we can see from this example, some people don’t really care about their secrets.

Secrets

...Other people forget to clean their git history

Secrets: Kubernetes case study

  • How the data of secrets are stored?
  • Who can access and modify secrets?
  • Do you use secrets as ENV parameters?

So, I mustn’t store my secrets in source code.

Secrets: Kubernetes case study

  • How the data of secrets are stored?
  • Who can access and modify secrets?
  • Do you use secrets as ENV parameters?

So, I mustn’t store my secrets in source code...

Secrets: Solutions

Security: References and additional reading

Useful tools to check out

[GopherConEU 2019] Production Ready Cloud-Native Services - Google Slides