1 of 64

Kafka tips for

the Devops Trenches

StevenLeRoux

@StevenLeRoux

@FinistDevs

2 of 64

$ whoami

    • Former network & infra eng. At Crédit Mutuel Arkéa
    • Anti-DDoS Detection system at OVH
    • Big Data Architect at OVH
    • @OvhMetrics
    • Principal Engineer @OVH
    • User of Kafka 0.6 (according to Horacio ;) )

To change the footer

  • Select the text block on the bottom of the slide
  • Write the name of the chapter

@FinistDevs

3 of 64

Streaming

​

Platform

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

@FinistDevs

4 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Who needs streaming?

  • PubSub
  • Stream processing
  • Real Time insights / statistics
  • … Actually who doesn’t need it?

​

@FinistDevs

5 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

@FinistDevs

6 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

What is Kafka?

Kafka is a Commit Log

​

​

@FinistDevs

7 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

What’s a Commit Log

@FinistDevs

8 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

What’s a commit log?

user

@FinistDevs

9 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

What’s a commit log?

user

txn

@FinistDevs

10 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

What’s a commit log?

user

txn

WAL

@FinistDevs

11 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

What’s a commit log?

user

txn

WAL

...

@FinistDevs

12 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

What’s a commit log?

user

txn

WAL

...

@FinistDevs

13 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

What’s a commit log?

@FinistDevs

14 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

What’s a commit log?

@FinistDevs

15 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

What is Kafka?

Kafka is a Distributed Commit Log

​

@FinistDevs

16 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

What is Kafka?

@FinistDevs

17 of 64

/ kafka / topic

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

@FinistDevs

18 of 64

/ kafka / topic / replicas

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

@FinistDevs

19 of 64

/ kafka / topic / replicas / factor / 3

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

@FinistDevs

20 of 64

/ kafka / topic

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

@FinistDevs

21 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Kafka Partitions

@FinistDevs

22 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Kafka Scaling

#partitions

#consumers

@FinistDevs

23 of 64

/ kafka / customer

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

@FinistDevs

24 of 64

/ kafka / cluster

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

@FinistDevs

25 of 64

/ kafka

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

@FinistDevs

26 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

What’s your use case ?

@FinistDevs

27 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Customer driven

Generics vs Purpose built

​

@FinistDevs

28 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Customer driven

It Starts with

Hardware consideration

@FinistDevs

29 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

Why?

@FinistDevs

30 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

my use cases with Apache Kafka

​

Stream Processing………………………....OK

​

Messaging / PubSub…………………..….OK

​

Asynchronous process……….….........OK

​

Whatsoever……….…..................................OK

​

​

@FinistDevs

31 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

Do you need to replay your data?

@FinistDevs

32 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

my use cases with Apache Kafka

​

​

Whatsoever………...….…..................................OK

​

With replay.…..................................DEPENDS

​

​

@FinistDevs

33 of 64

Metrics Data Platform - @OvhMetrics

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

@FinistDevs

34 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image
  • 5-10 ms
  • 200 move/s

​

Seek Time

@FinistDevs

35 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

HDD vs SSD vs BS

@FinistDevs

36 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Data Transfer Rates

Mbps 100 1000 10000

MB/s 12.5 125 1250

​

1 Mo 80ms 8ms 0.8ms

1 CD (700 Mo) 56s 5.6s 0.56s

1 Go (1000 Mo) 1m20 8s 0.8s

1 DVD (4700 Mo) 6m16 37.6s 3.76s

1 To (1000 Go) 22h13 2h13m 13m

@FinistDevs

37 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Data Transfer Rates

Mbps 100 1000 10000

MB/s 12.5 125 1250

​

1 min 750 MB 7.5 GB 75 GB

15 min 11 GB 112 GB 1 TB

1 hour 45 GB 450 GB 4.5 TB

1 day 1TB 10.8 TB 108 TB

@FinistDevs

38 of 64

Metrics Data Platform - @OvhMetrics

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

@FinistDevs

39 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

You need an OS

for your hardware

@FinistDevs

40 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

OS Considerations

​

  • File System choice (Ext4 vs XFS vs … )
  • File System Tuning
    • Noatime
    • buffer cache
    • fd limits
    • vm.swappiness
    • vm.max_map_count
  • Dedicated disks

@FinistDevs

41 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

Now we can focus on

Apache Kafka

itself

@FinistDevs

42 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

Availability

Consistency

Partitioning

CA

AP

CP

@FinistDevs

43 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Consistency : Producer side

Setting

Desc

Consistency

Performance

Acks = 0

“Fire and forget”

-

++

Acks = 1

Wait for leader acknowlegement

+

+

Acks = all

Wait for followers acknowlegements

++

-

@FinistDevs

44 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

Clients (producers) may have

no consideration for

your cluster topology

@FinistDevs

45 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Consistency : Cluster side

Remember : Purpose Built

@FinistDevs

46 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Consistency : Cluster side

@FinistDevs

47 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Consistency : Cluster side

ISR : In Sync Replicas

​

  • min.insync.replicas
  • unclean.leader.election.enable
  • replica.lag.max.messages

@FinistDevs

48 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Global Consistency : The Three Axis Rule

#Consistency

#Availability

#Performance

Acks = 0

unclean.leader.election.enable=true

min.insync.replicas

acks = all

unclean.leader.election.enable=false

min.insync.replicas=2

acks = 0/1

unclean.leader.election.enable=true

min.insync.replicas

@FinistDevs

49 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

Store Streams

@FinistDevs

50 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

Storing Considerations

​

  • Message size
  • Message rate
  • #partitions
  • #topics
  • Partition dispatch over borkers

​

=> log.retention.(bytes|time)

@FinistDevs

51 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

Performance considerations

@FinistDevs

52 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Performance Considerations

batch.size

linger.ms

Buffer memory

@FinistDevs

53 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Performance Considerations

batch.size

linger.ms

latency

thoughput

@FinistDevs

54 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Performance Considerations

batch.size

linger.ms

latency

thoughput

@FinistDevs

55 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

And More….

​

  • GC
  • Time-outs
  • Sockets buffers
  • Threads
  • Zookeeper
  • Compression

@FinistDevs

56 of 64

To add an image:

  • Click on the image icon �in the center of the block �on the right
  • Choose your image

Performance

Scale out infrastructure

@FinistDevs

57 of 64

Wrap up

    • OVH ❤ Kafka
    • Kafka is
        • a rock solid piece of software
        • a strong community
        • an OSS with commercial support and training
    • A “fire and forget” kind of software

To change the footer

  • Select the text block on the bottom of the slide
  • Write the name of the chapter

@FinistDevs

58 of 64

Thank you!

?

@StevenLeRoux

@FinistDevs

59 of 64

Metrics Data Platform - @OvhMetrics

To add an image:

  • Click on the image icon �in the center of the block �on the left
  • Choose your image

@FinistDevs

60 of 64

ICONS

  • TITLE OF THE CHAPTER

How to change the size of the icons:

  • If you want to make an icon bigger or smaller select it and drag it up or down

How to change the color of the icons:

  • Righ click on an icon and change the filling color

@FinistDevs

61 of 64

ICONS

  • TITLE OF THE CHAPTER

How to change the size of the icons:

  • If you want to make an icon bigger or smaller select it and drag it up or down

How to change the color of the icons:

  • Righ click on an icon and change the filling color

@FinistDevs

62 of 64

ICONS

  • TITLE OF THE CHAPTER

How to change the size of the icons:

  • If you want to make an icon bigger or smaller select it and drag it up or down

How to change the color of the icons:

  • Righ click on an icon and change the filling color

@FinistDevs

63 of 64

ICONS

  • TITLE OF THE CHAPTER

How to change the size of the icons:

  • If you want to make an icon bigger or smaller select it and drag it up or down

How to change the color of the icons:

  • Righ click on an icon and change the filling color

@FinistDevs

64 of 64

ICONS

  • TITLE OF THE CHAPTER

How to change the size of the icons:

  • If you want to make an icon bigger or smaller select it and drag it up or down

How to change the color of the icons:

  • Righ click on an icon and change the filling color

@FinistDevs