Production Ready
Cloud-Native Services
with Go and Kubernetes
Elena Grahovac & Daniel Mahlow
GopherCon Europe 2019
Production Ready
Cloud-Native Services
with Go and Kubernetes
Elena Grahovac & Daniel Mahlow
GopherCon Europe 2019
What should be prepared before the workshop begins
Agenda, part 1
09:00 - Warm Up
09:15 - Introduction. What we are going to achieve today
09:30 - Theory. How to be Cloud Native?
10:00 - Practice. Write a Go service from scratch
11:00 - Break & catch up (15 min)
11:20 - Practice. Continue preparing service
12:45 - Prepare your env for Kubernetes
13:00 - Lunch
Agenda, part 2
14:00 - Hello, Kubernetes!
14:30 - How we are going to deploy the services
14:45 - Dockerization
15:15 - Helm configuration
15:30 - Break & catch up (15 min)
15:45 - Configure GitHub integration to run CI/CD
16:00 - Deploy services
16:30 - Discuss questions and additional topics
Cleanup Notes
Lead TechOps Automation Engineer @N26
DevOps Enthusiast
hello@grahovac.pro
webdeva
rumyantseva
Co-Founder & TechOps lead @ Contiamo
dmahlow@gmail.com
danielbln
dmahlow
“Hello, World”
“Production ready”
How to be Cloud-Native?
DevOps
Cloud-Native
People and culture
Design and development practices
Cloud-Native Requirements
Production Readiness
These and other wonderful gophers: https://github.com/ashleymcnamara/gophers
Four-layer model of the microservice ecosystem
by Susan J. Fowler, Production-Ready Microservices
Production Readiness and Kubernetes
Structuring in Practice
Structuring in Practice
Kat Zień - How do you structure your Go apps: https://youtu.be/VQym87o91f8
Structuring in Practice
Structuring in Practice
cmd for small and simple entry points
If I have cmd/myctl and cmd/myserver
I can simply do go get cmd/… to have myctl and myserver binaries inside $GOPATH/bin
Structuring in Practice
internal for packages that should be available for current project only
If I don’t want a package being available for import by other projects, I’ll store it under internal
See also: ‘Go 1.4 Internal Packages’ - https://docs.google.com/document/d/1e8kOo3r51b2BWtTs_1uADIA5djfXhPT36s6eHVRIvaU/edit
Structuring in Practice
vendor to store dependencies :)
Other possible ideas (for inspiration): https://github.com/golang-standards/project-layout
Let’s start
Dependency Management
Dependencies: problem definition
Dependencies checklist
Dependencies: go mod cheatsheet
GOPROXY: pros
GOPROXY: cons
GOPROXY - Additional task *
Code Quality
This and other wonderful gophers: https://github.com/ashleymcnamara/gophers
Code Quality
Practices:
Code Quality: Testing
If you love the standard library and hate external dependencies:
func TestDoSomething(t *testing.T) {
// "want" is your expected result
have, err := mypkg.DoSomething()
if err != nil {
t.Errorf("%v", err)
} else if !reflect.DeepEqual(have, want) {
t.Errorf("have %+v, want %+v", have, want)
}
}
Code Quality: Testing
If you love to “assert”: https://github.com/stretchr/testify
func TestDoSomething(t *testing.T) {
// "want" is your expected result
have, err := mypkg.DoSomething()
require.NoError(t, err)
assert.Equal(t, want, have)
}
Code Quality: Testing *
Additional topics:
Fuzzy testing *
Fuzzy testing can help you to check if your validation is good enough
Code Quality: Analysis
Static code analyzers:
Code Quality: Analysis
Tools:
Code Quality: Analysis *
When Develop/CI/CD-ing:
When one should run linters?
Code Quality: Runtime *
Profiler:
Code Quality: Runtime *
package main
import (
"log"
"net/http"
_ "net/http/pprof"
)
func main() {
log.Println(http.ListenAndServe(":8080", nil))
}
Hello, pprof!
Observability & Operability
This and other wonderful gophers: https://github.com/ashleymcnamara/gophers
Observability
Logs:
Observability
Hello, logs!
package main
import (
log "github.com/sirupsen/logrus"
)
func main() {
log.WithFields(log.Fields{
"animal": "walrus",
}).Info("A walrus appears")
}
Observability *
Metrics:
Traces:
How to start? Check out OpenCensus: https://opencensus.io
Operability
Health checks:
Operability: Diagnostics Server *
Prepare a dedicated http.Server to handle health checks and pprof endpoints, it will listen to some other port (e.g. we can set it via DIAG_PORT).
Graceful Shutdown
Cheatsheet: https://github.com/rumyantseva/advent-2017/blob/master/main.go#L29
Versioning
Use the linker to provide a version, commit hash or any other data to identify the version:
go build -ldflags "-X main.version=put_version_here"
package main
import (
"fmt"
)
var (
version = "unset"
)
func main() {
fmt.Printf("The version is: %s\n", version)
}
Configuration
Automation
Repeatable Actions
Automate it *
For inspiration: https://github.com/takama/caldera
Dockerization
Builds
Simplest “mono-staging” image
FROM scratch
ENV PORT 8080
COPY bin/myapp /myapp
EXPOSE $PORT
CMD ["myapp"]
Multi-staging builds: GOPROXY
# Initial stage: download modules
FROM artifactory/golang:1.12
ADD go.mod go.sum /m/
RUN cd /m && go mod download
While go.mod and go.sum are not changed, the stage will be cached
Multi-staging builds: Toolchain
# Intermediate stage: Build the binary
FROM artifactory/golang:1.12 as builder
# add a non-privileged user
RUN useradd -u 10001 myapp
RUN mkdir -p /go/src/github.com/rumyantseva/myapp
ADD . /go/src/github.com/rumyantseva/myapp
WORKDIR /go/src/github.com/rumyantseva/myapp
# Build the binary with go build
RUN CGO_ENABLED=0 go build \
-o bin/myapp github.com/rumyantseva/myapp/cmd/myapp
# Stage 1. Run the binary
…
Multi-staging builds: The binary
# Final stage: Run the binary
FROM scratch
ENV PORT 8080
# certificates to interact with other services
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
# don't forget /etc/passwd from previous stage
COPY --from=0 /etc/passwd /etc/passwd
USER myapp
# and finally the binary
COPY --from=0 /go/src/github.com/rumyantseva/myapp/bin/myapp /myapp
EXPOSE $PORT
CMD ["myapp"]
Let’s deploy!
This and other wonderful gophers: https://github.com/ashleymcnamara/gophers
Hello, Kubernetes!
Prepare kubectl
Deploy “Hello, World!”
Make “Hello, World!” accessible from Internet
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: hello-app-ingress
namespace: rumyantseva
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: services.k8s.community
http:
paths:
- path: /rumyantseva/hello(/|$)(.*)
backend:
serviceName: hello-app
servicePort: 8080
tls:
- hosts:
- services.k8s.community
secretName: tls-secret
Make “Hello, World!” accessible from Internet
If you prefer UI
Let’s prepare charts
Ready for integration
Summary
These and other wonderful gophers: https://github.com/ashleymcnamara/gophers
Additional resources to try and play
Summary
Summary
Summary
Bonus Topic: a bit of Security
These and other wonderful gophers: https://github.com/ashleymcnamara/gophers
The role of Security in DevOps
Stage | Examples of Security Input |
Define Way of Working | General Requirements, Policies |
Design | Threat Modelling, Design Guidelines |
Code | Coding Rules, Codereview, Tools |
Test | Testing Plan, Toolkits |
Deploy | Configuration, Checklists |
Monitor | Incident Management, Scanning, Pen Test |
Secrets
Our applications interact with a lot of sensitive data internally: a database password, or an API key are as much important as user data. As we can see from this example, some people don’t really care about their secrets.
Secrets
...Other people forget to clean their git history
Secrets: Kubernetes case study
So, I mustn’t store my secrets in source code.
Secrets: Kubernetes case study
So, I mustn’t store my secrets in source code...
Secrets: Solutions
Security: References and additional reading
Useful tools to check out