Best Practices
for Cloud-Native Go Services

Elena Grahovac

Video: https://youtu.be/rRSkbOMTHLw

Lead TechOps Automation Engineer @N26

Podcast co-host @GolangShow

hello@grahovac.pro

webdeva

rumyantseva

“Hello, World”

“Production ready”

How to be Cloud-Native?

  • Operate managed infrastructure
  • Develop isolated containerized services
  • Communicate through network
  • Follow microservice design principles
  • Use DevOps techniques

DevOps

Cloud-Native

People and culture

Design and development practices

DevOps is not a one-way road!

Cloud-Native Requirements

  • 12 factor
  • Fault tolerance
  • Reliability
  • Maintainability
  • Operability
  • Observability
  • Security
  • Automation

Structuring
in Practice

Structuring
in Practice

  • Flat
  • Layer-based
  • Module-based
  • Context-based
  • “Hexagonal”

Kat Zień - How do you structure your Go apps: https://youtu.be/VQym87o91f8

Structuring
in Practice

  • cmd for small and simple entry points

If I have cmd/myctl and cmd/myserver
I can simply do go get cmd/… to have myctl and myserver binaries inside $GOPATH/bin

Structuring
in Practice

  • internal for packages that should be available for current project only

If I don’t want a package being available for import by other projects, I’ll store it under internal

See also: ‘Go 1.4 Internal Packages’ - https://docs.google.com/document/d/1e8kOo3r51b2BWtTs_1uADIA5djfXhPT36s6eHVRIvaU/edit

Structuring
in Practice

  • vendor to store dependencies :)

Other possible ideas (for inspiration): https://github.com/golang-standards/project-layout

Code Quality

This and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Code Quality

Practices:

  • Testing
  • Static code analyzers
  • Profiling

Code Quality:

Testing

If you love the standard library and hate external dependencies:

func TestDoSomething(t *testing.T) {

// "want" is your expected result

have, err := mypkg.DoSomething()

if err != nil {
t.Errorf("%v", err)
} else if !reflect.DeepEqual(have, want) {
t.Errorf("have %+v, want %+v", have, want)
}
}

Code Quality:

Testing

If you love the standard library and hate external dependencies:

func TestDoSomething(t *testing.T) {

// "want" is your expected result

have, err := mypkg.DoSomething()

if err != nil {
t.Errorf("%v", err)
} else if !reflect.DeepEqual(have, want) {
t.Errorf("have %+v, want %+v", have, want)
}

}

Code Quality:

Testing

If you love to “assert”: https://github.com/stretchr/testify

func TestDoSomething(t *testing.T) {

// "want" is your expected result

have, err := mypkg.DoSomething()

require.NoError(t, err)
assert.Equal(t, want, have)
}

Code Quality:

Testing

Additional topics:

Code Quality:

Analysis

Static code analyzers:

  • Check code style
  • Detect potential bugs
  • Scan for security problems

Code Quality:

Analysis

Tools:

Code Quality:
Analysis

When Develop/CI/CD-ing:

  • Local hooks (pre-commit)
  • Server hooks (push, pull request)
  • Separated Docker image
  • Continuous Integration stage

Code Quality:
Runtime

Profiler:

  • Simply import net/http/pprof to enable profiling handlers
  • Human-friendly interface: http://localhost:8080/debug/pprof
  • Tip: try github.com/google/pprof to check out new features

Code Quality:
Runtime

package main

import (

"log"

"net/http"

_ "net/http/pprof"

)

func main() {

log.Println(http.ListenAndServe(":8080", nil))

}

Hello, pprof!

Code Quality:
Runtime

Want to use custom URLs instead of debug/pprof?

r := mux.NewRouter()

r.HandleFunc("/my-prefix/pprof", pprof.Index)

r.HandleFunc("/my-prefix/cmdline", pprof.Cmdline)

r.HandleFunc("/my-prefix/profile", pprof.Profile)

r.HandleFunc("/my-prefix/symbol", pprof.Symbol)

r.HandleFunc("/my-prefix/trace", pprof.Trace)

r.Handle(
"/my-prefix/goroutine", pprof.Handler("goroutine"),
)

r.Handle(
"/my-prefix/heap", pprof.Handler("heap"),
)

r.Handle(
"/my-prefix/threadcreate",
pprof.Handler("threadcreate"),
)

r.Handle("/my-prefix/block",pprof.Handler("block"))

Observability
&
Operability

This and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Observability

Logs:

  • Log errors or strange situations!
  • Log “stages” of your application
  • Try multiple log levels
  • Try “tags” to classify your logs
  • Tip: if something goes wrong, check performance of your logger

Observability

Hello, logs!

package main

import (
log "github.com/sirupsen/logrus"
)

func main() {
log.WithFields(log.Fields{
"animal": "walrus",
}).Info("A walrus appears")
}

Observability

Metrics:

  • Define, measure and report business and SLA metrics
  • Business metrics might be useful as well!

Traces:

  • Tracing is a technique that shows how a single request goes through your service or services

How to start? Check out OpenCensus: https://opencensus.io

Operability

Health checks:

  • “healthy/unhealthy”: application state, business and performance metrics
  • “readiness”: report if application is ready to handle requests

Graceful Shutdown

  • Recover panics
  • Don’t panic, return errors
  • Handle errors
  • Listen to OS signals
  • Provide shutdown for your functional units

Versioning

Use the linker to provide a version, commit hash or any other data to identify the version:
go build -ldflags "-X main.version=put_version_here"

package main

import (
"fmt"
)

var (
version = "unset"
)

func main() {
fmt.Printf("The version is: %s\n", version)
}

Configuration

  • Set configuration via env
  • Try github.com/kelseyhightower/envconfig as a systems solution
  • Keep secrets safe
  • Prepare a systems solution if you need to deal with secrets on the application level
  • Consider keeping configuration for the platform you use

Dependency Management

Dependency Management

  • Want to feel safe? Use dep!
  • Commit, push and review vendor to keep the code as secure as it possible
  • Try go modules
  • Be flexible and production-ready with GOPROXY

Dockerization

Dockerization

Simplest “monostage” image

FROM scratch

ENV PORT 8080

COPY bin/myapp /myapp

EXPOSE $PORT

CMD ["/myapp"]

Dockerization

Multi-stage “all-in-one” builds

# Stage 1. Build the binary

FROM golang:1.11

# add a non-privileged user

RUN useradd -u 10001 myapp

RUN mkdir -p /go/src/myteam/myapp

ADD . /go/src/myteam/myapp

WORKDIR /go/src/myteam/myapp

# build the binary with go build

...

Dockerization

Multi-stage “all-in-one” builds

# Stage 2. Run the binary
FROM scratch

ENV PORT 8080

COPY --from=0 /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/


COPY --from=0 /etc/passwd /etc/passwd
USER myapp

COPY --from=0 /go/src/myteam/myapp/bin/myapp /myapp
EXPOSE $PORT


CMD ["/myapp"]

Automation

Repeatable Actions

  • Define repeatable actions and prepare a tool to call them fast:
    • The actions you call when CI/CD-ing
    • Checkers and tests
    • Application building
    • Dealing with container images
  • GNU Make as a classic approach
  • A fancy alternative: https://github.com/go-task/task

Automate it!

  • Prepare a typical “Cloud-Native Hello World” service
  • Define templates based on it
  • Use code generation to produce new services from the templates
  • Consider the details specific for your infrastructure or project

Summary

These and other wonderful gophers: https://github.com/ashleymcnamara/gophers

Summary

  • Think about structure before you start writing the application
  • Write unit tests
  • Try TDD to make your structure better and define how separated modules should interact to each other
  • Check your code not only for its style but also for potential bugs and security problems

Summary

  • Add a profiler before you released your first version to have the profile prepared as soon as you need it
  • Log errors, stages and events produced by your application
  • Define, measure and report metrics
  • Try tracing
  • Shutdown gracefully
  • Design configuration in advance

Summary

  • Keep version info inside the build
  • Manage dependencies
  • Think about security when containerize
  • Separate a container where you check the code and a container where you run the application
  • Define and describe repeatable actions
  • Automate

What’s Next?

  • Discover, try and adopt new practices
  • Share your experience with the community
  • See you as a speaker at GoWayFest 3.0!

hello@grahovac.pro

webdeva

rumyantseva

[GoWayFest 2018] - Best Practices for Cloud-Native Go Services - Google Slides