Discussion about managing shared states in distributed systems

Semih Ünaldı

Engineering Manager @ Delivery Hero - talabat

How we make architectural decisions?

Every single decision might result in potential technical debt in the future

​

• “Good” tech debt that you purposely left behind

​

• The “Evil” one that you haven’t even considered

​

​

​

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

2

Monoliths

Gains

​

• Cross-cutting concerns
• Simple Testing
• Less confused developers
• Direct communication
• No moving parts to secure

​

Pains

​

• Operational Scaling
• Risk of Change
• Risk of Deployment
• Onboarding Velocity
• Monoliths get old, fast...

​

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

3

Microservices

Gains

​

• Increased scalability
• Better resiliency
• Modularity
• Business agility
• Tech agnostic

​

​

Pains

​

• Higher complexity
• Network traffic
• Global testing and debugging
• Complex deployment

​

​

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

4

When do we say we successfully designed our architecture

• It works!

​

• It is easily maintainable (bugs won’t cause huge headaches)

​

• New joiners can adapt in their onboarding process

​

• New requirements can be absorbed easily

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

5

Fallacies of distributed systems

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

6

CAP Theorem

• Consistency
• Every read receives the most recent write or an error

​

• Availability
• Every request receives a (non-error) response, without the guarantee that it contains the most recent write

​

• Partition Tolerance
• The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

7

How do we introduce states to a system?

• Efficiency
• Not to do the same job twice

​

• Correctness
• Not to mess up the data when there are concurrent operations

​

​

​

​

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

8

How do we introduce states to a system?

​

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

9

How do we introduce states to a system?

​

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

10

How do we introduce states to a system?

​

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

11

What can go wrong?

​

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

12

What can go wrong?

​

ZooKeeper can be an option

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

13

Pains of Stateful Services

• State Consistency (Synchronization)
• Freshness of the State
• Duplicate Calls/Messages/Overwrites
• Introduces Complexity

​

​

​

​

​

​

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

14

Ease of Stateless Services

• Fire & forget
• Minimize Resources

​

​

​

​

• Ease of Scaling
• Less complexity/bugs

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

15

What to watch out for?

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

16

About talabat

• Part of the Delivery Hero network, sister company of Yemeksepeti
• Leading online food and groceries ordering platform in MENA region
• Teams in 13 countries and operations in 40+ countries
• Product & Tech organization consists of 350+ individuals
• More than 800,000 orders a day
• Quality enables speed

Delivery Hero Proprietary & Confidential. 2022 All Rights Reserved

17

Thank you!