1 of 35

Fluentd / Cloud Native Logging

Fluentd / Cloud Native Logging�Eduardo Silva @edsiper�eduardo@treasure-data.com

2 of 35

“Logging... Why?”�

​

3 of 35

Logging: Why ?

Analyze Applications Behavior

4 of 35

Logging: Why ?

5 of 35

Logging: Why ?

6 of 35

Logging Pipeline�

​

7 of 35

Logging Pipeline

8 of 35

It’s a mess

9 of 35

Logging has�Never Been Easier

10 of 35

11 of 35

About Fluentd

  • Invented by� �

​

  • Now hosted at �

�

​

12 of 35

Logging Pipeline

13 of 35

How Fluentd Simplifies Logging Pipeline

14 of 35

How Fluentd Simplifies Unified Logging Layer

15 of 35

About Fluentd

  • More than 600 plugins available�
  • Pluggable Architecture�
  • Built-in Reliability�
  • Full integration with Docker and Kubernetes�
  • Written in Ruby + C�

�

​

16 of 35

Fluentd Plugin Ecosystem

17 of 35

Fluentd Modular Architecture

18 of 35

Fluentd at Scale

19 of 35

Extending the Fluentd Ecosystem�

​

20 of 35

Fluentd Modes

  • Log Forwarder�
  • Log Aggregator�

�

​

21 of 35

Log Aggregator = �(Forwarder + Buffering Capabilities)�

​

22 of 35

Edge Nodes / Forward to Aggregators

App

Node 1

DB

App

DB

23 of 35

Edge Nodes & Costs

24 of 35

Edge Nodes & Costs

  • Fluentd requires ~40MB as minimum�
  • Deploying a few hundred could be expensive�
  • Can we make Forward cheaper ?��

�

​

25 of 35

Forwarder & Aggregator

Log Forwarder

Log Aggregator

26 of 35

27 of 35

About Fluent Bit

  • Written in C�
  • Pluggable Architecture�
  • Built-in Reliability�
  • Event Driven - Async I/O�

�

​

28 of 35

Why Fluent Bit as a Forwarder

  • Features�
    • Input, Filter and Output Plugins�
    • Built-in parsing support�
    • Minimum memory required 450KB

29 of 35

Edge Nodes / Forward to Aggregators

30 of 35

Cheap Forwarding

31 of 35

Cloud Native Features

  • Backpressure Handling�
  • Security�
  • Kubernetes Metadata Filter�
  • Timestamps w/Nanoseconds (upcoming v0.12)��

32 of 35

Security

  • Native integration for TLS/SSL�
      • Optional for plugins with network I/O
      • Certificates support

�

​

​

33 of 35

Features, plugins and more

  • Plugins (~35)�
    • Input
      • cpu, disk, forward, mem, syslog, tcp, kmsg...

​

    • Filter
      • Grep, Kubernetes & Record Modifier�
    • Output
      • Elasticsearch, HTTP, InfluxDB, Forward, Treasure Data

​

​

34 of 35

Fluent Bit Kubernetes DaemonSet

Github Repository�

​

Docker Hub Image

​

​

​

​

�

​

​

�

​

35 of 35

More Information

Links�

​

Contact�

  • Slack http://slack.fluentd.org
  • Twitter @fluentd

​

​

​

​

​

�

​

​

�

​