2 of 93

Who am I?

Ethan Garofolo - ethan@suchsoftware.com

Software consultant - I help companies make the transition to microservices

Used to run Learn All the Nodes

3 of 93

What is the opposite of a services-based architecture?

4 of 93

The monolith!

Image credit: https://upload.wikimedia.org/wikipedia/en/5/53/African_monolith_2001.jpg

5 of 93

What’s a monolith?

6 of 93

kloc < 300

Running on different machines

7 of 93

kloc < 300

Running on different machines

8 of 93

What is a monolith?

An entity focus
Implicit coupling at the database level

9 of 93

What is an entity focus?

What’s the first thing you do in a new Rails app?
Before anything else, you start planning out database tables
Before you know WHAT your app does, you start thinking about how you’re going to store data in a database

10 of 93

Implicit coupling in the database

Example: “the” users table
Everything in the system that needs up-to-date information about a user is now coupled to wherever that table lives
User Model is wont to accumulate a large number of methods

11 of 93

The User Model

12 of 93

“Extract microservices”

13 of 93

users

products

14 of 93

users

products

These lines are function calls

15 of 93

users

products

16 of 93

users

products

17 of 93

users

products

What are these lines now?

18 of 93

users

products

19 of 93

users

products

20 of 93

users

products

21 of 93

users

products

22 of 93

What defines a service?

24 of 93

Services are autonomous

No one asks them for anything (you ask databases for things, and that’s called querying)
They don’t ask anyone else for things (otherwise they’re dependent on those other things)
They are oblivious to the existence of other things

25 of 93

How do you get services to do anything?

https://xkcd.com/1312/

26 of 93

“Extract microservices”

Take Two

28 of 93

“JUST EXTRACT KITTENS FROM THIS SLIDE.” -- The Blogosphere

29 of 93

users

products

30 of 93

Asynchronous events

31 of 93

Case Study: HPC resource management

HPC assets used by scientists

Needed to manage access to them

Existing application could not adapt to the needs

32 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

33 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

34 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

Job requested!

35 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

36 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

I started moving the files

yawn

noted

37 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

38 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

noted

39 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

I finished moving the files

40 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

41 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

noted

42 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

I scheduled the job to run

43 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

44 of 93

Big win #1: Testing

45 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

46 of 93

Test harness

send-email

47 of 93

Big win #2: Other teams not moving fast enough

48 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

49 of 93

API

Event bus

dispatch

simulate-jobs

send-email

move-files

50 of 93

Big win #3: New features

51 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

52 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

send-sms

What changes were required everywhere else?

53 of 93

Big win #4: Leveraged existing system

54 of 93

And this was all cool, but...

55 of 93

API

Event bus

Legacy runtime

dispatch

New runtime

send-email

move-files

What happened to the events themselves?

send-sms

56 of 93

Event Sourcing

57 of 93

Traditional way of storing data

58 of 93

Web server

Biz

logic

buttons

id	name	clicks

59 of 93

Web server

req.body = {

}

Biz

logic

buttons

id	name	clicks

POST /buttons

60 of 93

Web server

req.body = {

}

Biz

logic

buttons

id	name	clicks

61 of 93

Web server

req.body = {

}

Biz

logic

buttons

id	name	clicks
1	foo	0

62 of 93

Web server

Biz

logic

id	name	clicks
1	foo	0

buttons

63 of 93

Web server

Biz

logic

id	name	clicks
1	foo	1

buttons

req.body = {

buttonId: 1,

}

POST /buttons/1/click

64 of 93

Web server

Biz

logic

id	name	clicks
1	foo	2

buttons

req.body = {

buttonId: 1,

}

POST /buttons/1/click

65 of 93

Web server

Biz

logic

id	name	clicks
1	foo	3

buttons

req.body = {

buttonId: 1,

}

POST /buttons/1/click

66 of 93

Implications

The database is a current snapshot of state
Reading and writing from the same data model
History is lost
At any point in time, we’re only storing what we’re aware we need to store

id	name	clicks
1	foo	3

buttons

67 of 93

Event Sourcing

68 of 93

Web server

Biz

logic

69 of 93

Web server

Biz

logic

POST /buttons

{

type: “buttonCreated”,

streamId: 1,

}

70 of 93

Web server

Biz

logic

POST /buttons

{

type: “countIncremented”,

streamId: 1,

}

{

type: “buttonCreated”,

streamId: 1,

}

71 of 93

Web server

Biz

logic

POST /buttons

{

type: “countIncremented”,

streamId: 1,

}

{

type: “buttonCreated”,

streamId: 1,

}

{

type: “countIncremented”,

streamId: 1,

}

72 of 93

Web server

Biz

logic

POST /buttons

{

type: “countIncremented”,

streamId: 1,

}

{

type: “buttonCreated”,

streamId: 1,

}

{

type: “countIncremented”,

streamId: 1,

}

{

type: “countIncremented”,

streamId: 1,

}

73 of 93

Does that feel weird?

74 of 93

What are events?

76 of 93

Anotomy of an Event

Events are interesting occurrences in your application’s domain
Event types are past tense
Events are things that have happened - you can’t argue with them
streamId is globally unique
`correlationId` links events in a distributed system to each other

{

type: ‘buttonCreated’,

streamId: ‘cj2qaa7sm0016muuaa1icupmb’,

streamType: ‘button’,

streamVersion: 1,

userId: ‘cj2qaa7smcanyouseemeaa1icupmb’

correlationId: ‘cj2qaa7k50012muuas9oh7g8k’,

payload: { ‘arbitrary’: ‘things’ },

createdAt: 1494917942033,

}

77 of 93

HOw do we make a useful user interface?

{

type: “countIncremented”,

streamId: 1,

}

{

type: “buttonCreated”,

streamId: 1,

}

{

type: “countIncremented”,

streamId: 1,

}

{

type: “countIncremented”,

streamId: 1,

}

78 of 93

We’d prefer something like

79 of 93

Event stream vs. displayable data

What we have:

[� { type: 'buttonCreated', streamId: '1', streamType: 'button' },� { type: 'buttonClicked', streamId: '1', streamType: 'button' },� { type: 'buttonCreated', streamId: '2', streamType: 'button' },� { type: 'buttonCreated', streamId: '3', streamType: 'button' },�]

What we’d like to have:

{

‘1’: { clicks: 1 },

‘2’: { clicks: 0 },

‘3’: { clicks: 0 },

}

Do we have something that can take a collection of things and turn it into a single thing?

80 of 93

“Current State is a

Left Fold of previous behaviours.”

-- Greg Young

81 of 93

“Current State is a

Left Fold of previous behaviours.”

-- Greg Young

“Left fold” is what reduce is called in functional programming circles

82 of 93

To the code!

Let’s reduce some things.

83 of 93

API

send-email

84 of 93

API

send-email

event stream (*not an Airstream)

85 of 93

Different views of the same state

86 of 93

{

type: “countIncremented”,

streamId: 1,

}

{

type: “buttonCreated”,

streamId: 1,

}

{

type: “countIncremented”,

streamId: 1,

}

{

type: “countIncremented”,

streamId: 1,

}

For sending to the client

[

{ id: ‘1’, clicks: 3 }

]

87 of 93

{

type: “countIncremented”,

streamId: 1,

}

{

type: “buttonCreated”,

streamId: 1,

}

{

type: “countIncremented”,

streamId: 1,

}

{

type: “countIncremented”,

streamId: 1,

}

For sending to the client

[

{ id: ‘1’, clicks: 3 }

]

For dealing with emails

[

{

id: ‘1’,

needsEmail: true,

emailTriggerCorrelationId: ‘foo’

}

]

88 of 93

Almost a free lunch

89 of 93

Multiple streams

90 of 93

Multiple streams

This app happened to put all events in a single table
But conceptually it’s still different streams
Streams are identified by a stream_id
They don’t necessarily correspond to single entities in the system

This app that happened to be the case
But imagine a bank account-- transactions on an account could be in the same stream as the account’s creation, but they’re entities in their own right

91 of 93

Some key points

Probably wouldn’t want to roll these up on the fly
Can snapshot state along the way
I’ll see your “you ain’t gonna need it” and raise you “1 premature optimization”

92 of 93

Conclusions

Implicit coupling is what makes a monolith
Implicit coupling across machine boundaries (the distributed monolith) is still a monolith
Services are autonomous (they don’t take questions, ask them, or know anyone else is even out there)
I don’t find that the “micro” in “microservices” adds much meaning
Life is an append-only series of events
Event-based architecture !== message brokers

1 of 93

2 of 93

3 of 93

4 of 93

5 of 93

6 of 93

7 of 93

8 of 93

9 of 93

10 of 93

11 of 93

12 of 93

13 of 93

14 of 93

15 of 93

16 of 93

17 of 93

18 of 93

19 of 93

20 of 93

21 of 93

22 of 93

23 of 93

24 of 93

25 of 93

26 of 93

27 of 93

28 of 93

29 of 93

30 of 93

31 of 93

32 of 93

33 of 93

34 of 93

35 of 93

36 of 93

37 of 93

38 of 93

39 of 93

40 of 93

41 of 93

42 of 93

43 of 93

44 of 93

45 of 93

46 of 93

47 of 93

48 of 93

49 of 93

50 of 93

51 of 93

52 of 93

53 of 93

54 of 93

55 of 93

56 of 93

57 of 93

58 of 93

59 of 93

60 of 93

61 of 93

62 of 93

63 of 93

64 of 93

65 of 93

66 of 93

67 of 93

68 of 93

69 of 93

70 of 93

71 of 93

72 of 93

73 of 93

74 of 93

75 of 93

76 of 93

77 of 93

78 of 93

79 of 93

80 of 93