1 of 5

Fluentd: Overview and History

  • https://github.com/fluent/fluentd/
  • http://www.fluentd.org/
  • Started at Treasure Data in 2011
    • Sadayuki “Sada” Furuhashi: MessagePack and Embulk creator
  • Solve MxN problem in log collection reliably
    • M(any) inputs & N(umerous) outputs make log delivery hard

​

​

1

2 of 5

Fluentd: Milestones

  • 118 releases & 2,362 commits, 704 PRs (17 open)
    • 90 contributors
    • 50% of top 10 contributors commercially sponsored
  • 4,368 stars, 518 forks
  • 625 plugins
  • In production@Atlassian, Nintendo, LINE, Microsoft OMS, etc.

2

3 of 5

Fluentd: Key Features

  • Structural logging + Tag-based routing
    • parse data as JSON (msgpack internally)
    • Tags for declarative event routing
  • Built-in buffering
    • No external queue required for retries
  • Plugin system
    • Input/Parser/Filter/Buffer/Output/Formatter all pluggable
  • High performance
    • Mostly in Ruby, performance critical part in C

3

4 of 5

Fluentd: Architecture

4

TCP/UDP

Syslog

HTTP(S)

File

​

HDFS

Elasticsearch

S3

GCS

Azure Blob

Treasure Data

SaaS API

etc.

5 of 5

Fluentd: Docker Setup

  • Supported by Docker Log Driver API
    • Stream container logs directly to Fluentd (running on host or container)
  • Kubernetes Logging Pod

5