Goroutines:

Under the Hood

Vicki Niu

@vickiniu

we ❤️ Go’s concurrency primitives

but what is a goroutine?

“goroutines are lightweight threads”

vs

the lightweight goroutine

the OS thread

the lightweight goroutine

leverages

the OS thread

Fast facts about goroutines

• Started and managed by the Go runtime
• Goroutines start with just 2kb of memory, but have growable stacks
• Seamless context-switching that maximizes efficiency, thanks to the magic of the Go runtime scheduler
• Native inter-goroutine communication with channels

In general, trivial to run millions of goroutines on a given machine!

👀 so... what is a goroutine?

type g struct {

stack stack // stack offsets

m *m // current m (os thread)

sched gobuf // saves context of g for recovering after context switch

atomicstatus uint32 // current status of g (runnable, running, waiting)

​

// activeStackChans indicates that there are unlocked channels

// pointing into this goroutine's stack. If true, stack

// copying needs to acquire channel locks to protect these

// areas of the stack.

activeStackChans bool

​

// parkingOnChan indicates that the goroutine is about to

// park on a chansend or chanrecv. Used to signal an unsafe point

// for stack shrinking. It's a boolean value, but is updated atomically.

parkingOnChan uint8

}

goroutine stack

All goroutines are initially allocated 2kb of memory

Each go function has a small preamble, which calls morestack if it runs out of memory

Then, the runtime allocates a new memory segment with double the size, and copies over the old segment and restarts execution

Effectively, makes goroutines infinitely growable! with efficient shrinking

type g struct {

stack stack // stack offsets

m *m // current m (os thread)

sched gobuf // saves context of g for recovering after context switch

atomicstatus uint32 // current status of g (runnable, running, waiting)

​

// activeStackChans indicates that there are unlocked channels

// pointing into this goroutine's stack. If true, stack

// copying needs to acquire channel locks to protect these

// areas of the stack.

activeStackChans bool

​

// parkingOnChan indicates that the goroutine is about to

// park on a chansend or chanrecv. Used to signal an unsafe point

// for stack shrinking. It's a boolean value, but is updated atomically.

parkingOnChan uint8

}

🧠 so, when does a goroutine run?

✨ enter the go scheduler ✨

P

M

G

goroutine

OS thread

logical processor

P

M

G

goroutine: the code to execute

OS thread: where to execute it

logical processor: rights + resources to execute it

M:N scheduler

G

G

G

M

M

M

G

G

G

goroutine: the code to execute

G

goroutine: the code to execute

can be in one of three states:

1. blocked
2. runnable
3. running

the scheduler’s job:

running goroutines as efficiently as possible

G

goroutine: the code to execute

can be in one of three states:

• blocked
• runnable
• running

P

M

G

goroutine: the code to execute

OS thread: where to execute it

logical processor: rights & resources to execute it

main() runs on the main execution thread

P₀

M

G

func main() {...}

Now, we create our processors

P₀

M

G

func main()

idle processors

P₁

P₂

P₃

Now, we create our processors

P₀

M

G

func main()

# of processors is equal to the number of logical cores, or GOMAXPROCS

idle processors

P₁

P₂

P₃

main() spawns a goroutine!

P₀

M

G

func main()

idle processors

P₁

P₂

P₃

G₁

G₁ wakes up an idle P

P₀

M

G

func main()

idle processors

P₁

P₂

P₃

G₁

G₁ wakes up an idle P

P₀

M

G

func main()

idle processors

P₁

P₂

P₃

G₁

P₃ creates an M, with an OS thread to run G₁

P₀

M

G

func main()

idle processors

P₁

P₂

P₃

M

G₁

P₃ creates an M, with an OS thread to run G₁

P₀

M

G

func main()

idle processors

P₁

P₂

P₃

M

G₁

G₁ finishes executing, now P₃ and M are idle

P₀

M

G

func main()

idle processors

P₁

P₂

P₃

M

G₁ finishes executing, now P₃ and M are idle

P₀

M

G

func main()

idle processors

P₁

P₂

P₃

M

idle threads

what do we really need P for?

P₀

M

G

func main()

P₁

M

G₁

P₀

M

G

func main()

P₁

M

G₁

G₂

P₀

M

G

func main()

P₁

M

G₁

G₂

P₀

M

G

func main()

P₁

M

G₁

G₂

LRQ

G₂ gets added to the

local run queue of the current P

P₀

M

G

func main()

P₁

M

G₁

G₂

LRQ

When P₁ becomes available again, looks in LRQ for a G

P₀

M

G

func main()

P₁

M

G₂

LRQ

When P₁ becomes available again, looks in LRQ for a G

P₀

M

G

func main()

P₁

M

G₂

LRQ

When P₁ becomes available again, looks in LRQ for a G

P₀

M

G

func main()

P₁

M

G₂

LRQ

P knows what G’s are available for an M to run

🤔 what about blocked goroutines?

💡the scheduler should maximize execution time on OS threads

what blocks a goroutine?

• system calls (file I/O)
• network calls
• channels

blocking system call

P₀

M₀

G₀

P₁

idle processors

idle threads

M₁

LRQ

G₀ makes a blocking system call, invoking the scheduler

P₀

M₀

G₀

P₁

idle processors

idle threads

M₁

LRQ

G₀ makes a blocking system call, invoking the scheduler

P₀

M₀

G₀

P₁

idle processors

idle threads

M₁

LRQ

P₀ releases M₀ while G₀ is still blocking

P₀

M₀

G₀

P₁

idle processors

idle threads

M₁

LRQ

P₀ wakes up M₁ to continue executing G’s

P₀

M₀

G₀

P₁

idle processors

idle threads

M₁

G₁

LRQ

G₂

P₀ wakes up M₁ to continue executing G’s

P₀

M₀

G₀

P₁

idle processors

idle threads

M₁

G₁

LRQ

G₂

P₀ wakes up M₁ to continue executing G’s

P₀

M₀

G₀

P₁

idle processors

idle threads

M₁

G₁

LRQ

G₂

P₀ wakes up M₁ to continue executing G’s

P₀

M₀

G₀

P₁

idle processors

idle threads

M₁

G₂

LRQ

G₁

When G₀ unblocks, M₀ attempts to find a P

P₀

M₀

G₀

P₁

idle processors

idle threads

M₁

G₂

LRQ

G₁

When G₀ unblocks, M₀ attempts to find a P

P₀

M₀

G₀

P₁

idle processors

idle threads

M₁

G₂

LRQ

G₁

When G₀ unblocks, M₀ attempts to find a P

P₀

M₀

G₀

P₁

idle processors

idle threads

M₁

G₂

LRQ

G₁

G execution continues!

P₀

M₀

G₀

P₁

idle processors

idle threads

M₁

G₂

LRQ

G₁

If M₀ can’t find a P, then M₀ goes to sleep

P₀

G₀

idle processors

idle threads

M₁

G₂

LRQ

G₁

M₀

If M₀ can’t find a P, then M₀ goes to sleep

P₀

G₀

idle processors

idle threads

M₁

G₂

LRQ

G₁

M₀

If M₀ can’t find a P, then M₀ goes to sleep

P₀

G₀

idle processors

idle threads

M₁

G₂

LRQ

G₁

M₀

If M₀ can’t find a P, then M₀ goes to sleep

If M₀ can’t find a P, then M₀ goes to sleep

​

G₀ gets added to GRQ

(global run queue)

P₀

G₀

idle processors

idle threads

M₁

G₂

LRQ

G₁

M₀

If M₀ can’t find a P, then M₀ goes to sleep

​

G₀ gets added to GRQ

(global run queue)

P₀

idle processors

idle threads

M₁

G₂

LRQ

G₁

M₀

G₀

GRQ

blocking system call

• Allows the blocking G to hang onto the M executing the system call, but avoid consuming P resources to run other goroutines!
• Some system calls should always be quick! Go runtime does a slight optimization, only context-switching for expensive syscalls

​

blocking network call

blocking network call

• net/http package provides a default of spawning a goroutine for each incoming connection
• Within Go, we can interact with network requests as if they are blocking, which makes our lives as Go developers easy!
• In the runtime, this is taken care of by the netpoller: netpoller’s job is to schedule goroutines waiting on asynchronous system calls, and provide a blocking interface

​

P₀

M₀

G₀

G₁

LRQ

G₂

G₀ wants to make a network call

P₀

M₀

G₀

G₁

LRQ

G₂

G₀ wants to make a network call

P₀

M₀

G₀

G₁

LRQ

G₂

G₀ wants to make a network call

P₀

M₀

G₀

G₁

LRQ

G₂

Net Poller

G₀ wants to make a network call, so G₀ is moved to the net poller!

Net poller has its own OS thread, and handles events from goroutines doing network I/O

​

Interfaces with the OS to poll the appropriate network sockets

​

Re-schedules goroutines when their network resource is available!

P₀

M₀

G₀

G₁

LRQ

G₂

Net Poller

netpoll interface

// func netpollinit()

//

// func netpollopen(fd uintptr, pd *pollDesc) int32

// Arm edge-triggered notifications for fd. The pd argument is to pass

// back to netpollready when fd is ready. Return an errno value.

//

// func netpoll(delta int64) gList

// Poll the network. If delta < 0, block indefinitely. If delta == 0,

// poll without blocking. If delta > 0, block for up to delta nanoseconds.

// Return a list of goroutines built by calling netpollready.

//

// func netpollBreak()

// Wake up the network poller, assumed to be blocked in netpoll.

//

// func netpollIsPollDescriptor(fd uintptr) bool

// Reports whether fd is a file descriptor used by the poller.

netpoll interface

// func netpollinit()

//

// func netpollopen(fd uintptr, pd *pollDesc) int32

// Arm edge-triggered notifications for fd. The pd argument is to pass

// back to netpollready when fd is ready. Return an errno value.

//

// func netpoll(delta int64) gList

// Poll the network. If delta < 0, block indefinitely. If delta == 0,

// poll without blocking. If delta > 0, block for up to delta nanoseconds.

// Return a list of goroutines built by calling netpollready.

//

// func netpollBreak()

// Wake up the network poller, assumed to be blocked in netpoll.

//

// func netpollIsPollDescriptor(fd uintptr) bool

// Reports whether fd is a file descriptor used by the poller.

// poll network if not polled for more than 10ms

lastpoll := int64(atomic.Load64(&sched.lastpoll))

if netpollinited() && lastpoll != 0 && lastpoll+10*1000*1000 < now {

atomic.Cas64(&sched.lastpoll, uint64(lastpoll), uint64(now))

list := netpoll(0) // non-blocking - returns list of goroutines

if !list.empty() {

...

injectglist(&list)

}

}

​

G₀ wants to make a network call, so G₀ is moved to the net poller!

P₀

M₀

G₁

LRQ

G₂

Net Poller

G₀

G₀ wants to make a network call, so G₀ is moved to the net poller!

P₀

M₀

G₂

LRQ

Net Poller

G₀

G₁

When network call is complete, net poller moves G₀ back onto LRQ

P₀

M₀

G₂

LRQ

Net Poller

G₀

G₁

When network calls is complete, net poller moves G₀ back onto LRQ

P₀

M₀

G₂

LRQ

Net Poller

G₁

G₀

P₀

M₀

G₂

LRQ

Net Poller

G₁

G₀

blocking on channels

P₀

M₀

G₀

G₁

LRQ

G₂

G₀ wants to send on a full channel

ch <- foo

P₀

M₀

G₀

G₁

LRQ

G₂

G₀ wants to send on a full channel, and blocks

ch <- foo

P₀

M₀

G₀

G₁

LRQ

G₂

G₀ wants to send on a full channel, and blocks, adding G₀ to ch’s recvq

ch <- foo

ch

P₀

M₀

G₀

G₁

LRQ

G₂

Scheduler removes G₀ from M₀, executing the next G

ch <- foo

P₀

M₀

G₀

G₁

LRQ

G₂

Scheduler removes G₀ from M₀, executing the next G

ch <- foo

P₀

M₀

G₂

LRQ

G₀

G₁

Now, G₁ is our savior!

ch <- foo

bar := <-ch

P₀

M₀

G₂

LRQ

G₀

G₁

Now, G₁ is our savior!

​

On the channel receive, ch looks into its recvq.

​

G₁ then calls into the scheduler, making G₀ runnable.

ch <- foo

bar := <-ch

ch

P₀

M₀

G₂

LRQ

G₀

G₁

G₀ is added to the LRQ.

bar := <-ch

P₀

M₀

G₂

LRQ

G₁

G₀

work stealing between threads

P₀

M₀

G₀

G₁

LRQ

G₂

G₃

G₄

LRQ

G₅

G₆

G₇

P₁

M₁

P₀

M₀

LRQ

G₄

G₅

LRQ

G₆

G₇

P₁

M₁

P₀ tries to steal work from another P.

P₀

M₀

LRQ

G₄

G₅

LRQ

G₆

G₇

P₁

M₁

P₀ tries to steal work from another P.

P₀

M₀

LRQ

G₄

G₅

LRQ

G₆

G₇

P₁

M₁

P₀ tries to steal work from another P.

P₀

M₀

LRQ

G₄

G₅

LRQ

P₁

M₁

G₆

G₇

P₀ tries to steal work from another P.

P₀

M₀

LRQ

G₅

LRQ

P₁

M₁

G₇

G₄

G₆

Ensures that threads don’t remain idle while there is work to do!

P₀

M₀

LRQ

G₅

LRQ

P₁

M₁

G₇

G₄

G₆

🙇‍♀️ now, we’re goroutine experts!

goroutine stack is lightweight & growable

M:N scheduling allows for efficient use of OS threads

scheduler maximizes OS thread resources for execution, context switching for

• long OS syscalls,
• networking syscalls, and
• channel blocking seamlessly!

exposes convenient blocking interfaces for us to use as Go developers!