Best Practices
for Cloud-Native Go Services

Elena Grahovac

Lead TechOps Automation Engineer @N26

Podcast co-host @GolangShow

​

​

hello@grahovac.pro

webdeva

rumyantseva

“Hello, World”

“Production ready”

How to be Cloud-Native?

• Operate managed infrastructure
• Develop isolated containerized services
• Communicate through network
• Follow microservice design principles
• Use DevOps techniques

DevOps

Cloud-Native

People and culture

Design and development practices

DevOps is not a one-way road!

“

Cloud-Native Requirements

• 12 factor
• Fault tolerance
• Reliability
• Maintainability
• Operability
• Observability
• Security
• Automation

Structuring
in Practice

“

Structuring
in Practice

• Flat
• Layer-based
• Module-based
• Context-based
• “Hexagonal”

​

Kat Zień - How do you structure your Go apps: https://youtu.be/VQym87o91f8

Structuring
in Practice

• cmd for small and simple entry points

If I have cmd/myctl and cmd/myserver
I can simply do go get cmd/… to have myctl and myserver binaries inside $GOPATH/bin

Structuring
in Practice

• internal for packages that should be available for current project only

If I don’t want a package being available for import by other projects, I’ll store it under internal

See also: ‘Go 1.4 Internal Packages’ - https://docs.google.com/document/d/1e8kOo3r51b2BWtTs_1uADIA5djfXhPT36s6eHVRIvaU/edit

Structuring
in Practice

• vendor to store dependencies :)

Other possible ideas (for inspiration): https://github.com/golang-standards/project-layout

Code Quality

This and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Code Quality

Practices:

• Testing
• Static code analyzers
• Profiling

Code Quality:

Testing

If you love the standard library and hate external dependencies:

func TestDoSomething(t *testing.T) {

// "want" is your expected result

have, err := mypkg.DoSomething()

if err != nil {
t.Errorf("%v", err)
} else if !reflect.DeepEqual(have, want) {
t.Errorf("have %+v, want %+v", have, want)
}
}

​

Code Quality:

Testing

If you love the standard library and hate external dependencies:

func TestDoSomething(t *testing.T) {

// "want" is your expected result

have, err := mypkg.DoSomething()

if err != nil {
t.Errorf("%v", err)
} else if !reflect.DeepEqual(have, want) {
t.Errorf("have %+v, want %+v", have, want)
}
}

​

Code Quality:

Testing

If you love to “assert”: https://github.com/stretchr/testify

func TestDoSomething(t *testing.T) {

// "want" is your expected result

have, err := mypkg.DoSomething()

require.NoError(t, err)
assert.Equal(t, want, have)
}

​

Code Quality:

Testing

Additional topics:

• Why there is no “assert” in the standard library: https://golang.org/doc/faq#testing_framework
• Learn testing: https://github.com/golang/go/wiki/LearnTesting
• Advanced testing: https://about.sourcegraph.com/go/advanced-testing-in-go

Code Quality:

Analysis

Static code analyzers:

• Check code style
• Detect potential bugs
• Scan for security problems

​

Code Quality:

Analysis

Tools:

• Run ~30 most popular linters: golangci-lint https://github.com/golangci/golangci-lint
• If you want more: go-critic (~60 checkers) https://github.com/go-critic/go-critic

Code Quality:
Analysis

When Develop/CI/CD-ing:

• Local hooks (pre-commit)
• Server hooks (push, pull request)
• Separated Docker image
• Continuous Integration stage

​

Code Quality:
Runtime

Profiler:

• Simply import net/http/pprof to enable profiling handlers
• Human-friendly interface: http://localhost:8080/debug/pprof
• Tip: try github.com/google/pprof to check out new features

​

Code Quality:
Runtime

package main

​

import (

"log"

"net/http"

_ "net/http/pprof"

)

​

func main() {

log.Println(http.ListenAndServe(":8080", nil))

}

​

​

Hello, pprof!

Code Quality:
Runtime

Want to use custom URLs instead of debug/pprof?

​

r := mux.NewRouter()

r.HandleFunc("/my-prefix/pprof", pprof.Index)

r.HandleFunc("/my-prefix/cmdline", pprof.Cmdline)

r.HandleFunc("/my-prefix/profile", pprof.Profile)

r.HandleFunc("/my-prefix/symbol", pprof.Symbol)

r.HandleFunc("/my-prefix/trace", pprof.Trace)

r.Handle(
"/my-prefix/goroutine", pprof.Handler("goroutine"),
)

r.Handle(
"/my-prefix/heap", pprof.Handler("heap"),
)

r.Handle(
"/my-prefix/threadcreate",
pprof.Handler("threadcreate"),
)

r.Handle("/my-prefix/block",pprof.Handler("block"))

​

Observability
&
Operability

This and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Observability

Logs:

• Log errors or strange situations!
• Log “stages” of your application
• Try multiple log levels
• Try “tags” to classify your logs
• Tip: if something goes wrong, check performance of your logger

​

Observability

Hello, logs!

​

package main

import (
log "github.com/sirupsen/logrus"
)

func main() {
log.WithFields(log.Fields{
"animal": "walrus",
}).Info("A walrus appears")
}

Observability

Metrics:

• Define, measure and report business and SLA metrics
• Business metrics might be useful as well!

Traces:

• Tracing is a technique that shows how a single request goes through your service or services

How to start? Check out OpenCensus: https://opencensus.io

Operability

Health checks:

• “healthy/unhealthy”: application state, business and performance metrics
• “readiness”: report if application is ready to handle requests

​

​

Graceful Shutdown

• Recover panics
• Don’t panic, return errors
• Handle errors
• Listen to OS signals
• Provide shutdown for your functional units

Versioning

Use the linker to provide a version, commit hash or any other data to identify the version:
go build -ldflags "-X main.version=put_version_here"

package main

import (
"fmt"
)

var (
version = "unset"
)

func main() {
fmt.Printf("The version is: %s\n", version)
}

Configuration

• Set configuration via env
• Try github.com/kelseyhightower/envconfig as a systems solution
• Keep secrets safe
• Prepare a systems solution if you need to deal with secrets on the application level
• Consider keeping configuration for the platform you use

Dependency Management

Dependency Management

• Want to feel safe? Use dep!
• Commit, push and review vendor to keep the code as secure as it possible
• Try go modules
• Be flexible and production-ready with GOPROXY

Dockerization

Dockerization

Simplest “monostage” image

FROM scratch

​

ENV PORT 8080

​

COPY bin/myapp /myapp

EXPOSE $PORT

​

CMD ["/myapp"]

​

​

Dockerization

Multi-stage “all-in-one” builds

# Stage 1. Build the binary

FROM golang:1.11

​

# add a non-privileged user

RUN useradd -u 10001 myapp

​

RUN mkdir -p /go/src/myteam/myapp

ADD . /go/src/myteam/myapp

WORKDIR /go/src/myteam/myapp

​

# build the binary with go build

...

Dockerization

Multi-stage “all-in-one” builds

# Stage 2. Run the binary
FROM scratch

ENV PORT 8080

COPY --from=0 /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

COPY --from=0 /etc/passwd /etc/passwd
USER myapp

COPY --from=0 /go/src/myteam/myapp/bin/myapp /myapp
EXPOSE $PORT

CMD ["/myapp"]

Automation

Repeatable Actions

• Define repeatable actions and prepare a tool to call them fast:
• The actions you call when CI/CD-ing
• Checkers and tests
• Application building
• Dealing with container images
• GNU Make as a classic approach
• A fancy alternative: https://github.com/go-task/task

Automate it!

• Prepare a typical “Cloud-Native Hello World” service
• Define templates based on it
• Use code generation to produce new services from the templates
• Consider the details specific for your infrastructure or project

For inspiration: https://github.com/takama/caldera

Summary

These and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Summary

• Think about structure before you start writing the application
• Write unit tests
• Try TDD to make your structure better and define how separated modules should interact to each other
• Check your code not only for its style but also for potential bugs and security problems

Summary

• Add a profiler before you released your first version to have the profile prepared as soon as you need it
• Log errors, stages and events produced by your application
• Define, measure and report metrics
• Try tracing
• Shutdown gracefully
• Design configuration in advance

Summary

• Keep version info inside the build
• Manage dependencies
• Think about security when containerize
• Separate a container where you check the code and a container where you run the application
• Define and describe repeatable actions
• Automate

What’s Next?

• Discover, try and adopt new practices
• Share your experience with the community
• See you as a speaker at GoWayFest 3.0!

hello@grahovac.pro

webdeva

rumyantseva

These slides: https://gowayfest.grahovac.pro