1 of 32

Apache Pinot

Fasih Khatib / fasih@startree.ai

powering real-time analytics

2 of 32

Outline

  • An introduction to real-time analytics.
  • An introduction to Apache Pinot
    • What is Apache Pinot?
    • What use-cases does Pinot enable?
    • Who uses Pinot?
    • Pinot 101 — architecture, ingestion, querying
  • Demo

3 of 32

What is real-time analytics?

4 of 32

Real-time analytics is all about going from event to action as quickly as possible.

💥

💭

🏃‍➡️

Event

Insight

Action

5 of 32

Real-time analytics is all about going from event to action as quickly as possible.

💭

🏃‍➡️

Buy

Find the trending product

Make sure it is in stock

6 of 32

Why real-time?

7 of 32

Analytics quadrant

8 of 32

What is Apache Pinot?

9 of 32

“Apache Pinot is a realtime, distributed, OLAP database built for low-latency, high-throughput analytics.”

10 of 32

Realtime

The amount of time taken to execute a query.

​

As low as 10ms.

The ability of the system to handle multiple queries simultaneously.

​

As high as 100k QPS.

The up-to-date nature of data in the system.

​

Seconds from event-time till queryable in Pinot.

Latency

Concurrency

Freshness

11 of 32

Distributed

  • Traditional relational databases colocate compute and storage.
  • Pinot is built from the ground up to be distributed.
  • It chooses tightly-coupled storage and pre-allocated compute to minimise query latency.
  • Storage and compute can be scaled semi-independently so that clusters can be customised for specific workloads.

12 of 32

OLAP

Transaction-focused.

Aggregation-focused.

Write-heavy workloads.

Read-heavy workloads.

Often involves a single row per operation.

Often involves many records in one operation.

OLTP

OLAP

13 of 32

Use cases

  • User-facing analytics. Build in-app analytics.
  • Business intelligence. Build dashboards for internal users.
  • Ad-hoc exploration. Write queries to analyse your data.
  • Anomaly detection. Detect outliers in the data.

14 of 32

Getting started with Pinot

15 of 32

Data Model

  • Pinot uses tables to store data.
  • Tables are created using JSON.
  • Queries are expressed in SQL.

16 of 32

Architecture —

Tables and Segments

17 of 32

Tables

  • Basic unit of storage in Pinot
  • Composed of rows and columns.
  • Can store arbitrarily large row counts.
  • Defined using JSON schema.
  • Three types — realtime, offline, hybrid.
  • Every column is either a metric, a dimension, or a date time.

18 of 32

Segments

  • Tables are made up of units of storage called segments.
  • Similar to shards, and completely transparent to the user.
  • For offline tables, segments are created outside of Pinot and pushed into the cluster using the REST API.
  • For realtime tables, segments are created automatically from events sourced from the event streaming system. (eg. Kafka)

19 of 32

Architecture — Cluster

20 of 32

Cluster

  • The cluster consists of the following components.
    • Controller
    • Broker
    • Server
    • Minion

21 of 32

Controller

  • Coordination.
  • The controller is responsible for
    • Managing the brokers, minions, and servers.
    • Keeping track of which servers are responsible for which segments.
    • Providing endpoints for segment uploads used in offline data pushes.

22 of 32

Broker

  • Query processing.
  • The broker is responsible for
    • Accepting queries from users, scattering them to the servers, gathering the results, and returning the final result to the user.
    • Determining which segments are responsible for required in a given query, and which servers host these segments.
    • For single-stage queries, they collect results from the servers and consolidate them into a single result.
    • For multi-stage queries, they issue a query plan to the servers, which eventually returns a single result set to the issuing brokers.

23 of 32

Server

  • Storage and computation.
  • Server is responsible for
    • Storing the segments.
    • Performing the computation required to execute queries.

24 of 32

Minion

  • Executing background tasks.
  • Tasks performed by the minions include
    • Converting batch data from Avro or JSON into segment files.
    • Rewriting existing segment files to purge records as required by data privacy laws like GDPR.
  • Tasks can be run once or periodically.

25 of 32

Ingestion —�Batch and Realtime.

26 of 32

Batch Ingestion

  • Allows ingesting data from a file system like S3, HDFS, Blob store, ADLS, etc.
  • The following steps are involved in batch ingestion:
    • Create a schema.
    • Create a table configuration.
    • Upload table and schema.
    • Spin up a minion task.

27 of 32

Streaming Ingestion

  • Allows ingesting data from a file system like Kafka, Kinesis, and Pulsar.
  • The following steps are involved in stream ingestion:
    • Create a schema.
    • Create a table configuration.
    • Specify the ingestion config.
    • Upload table and schema.

28 of 32

Querying

29 of 32

Querying

  • Tables can be queried using SQL.
  • There are two query engines.
    • Single-stage query engine.
    • Multi-stage query engine.
  • Single-stage query engine allows simple SQL queries.
  • Multi-stage query engine allows complex SQL queries.

30 of 32

Single-stage Query Engine

  • Suitable for simpler queries that perform aggregations.
  • Scatters the query to relevant servers.
  • Gathers the results.
  • Performs a reducing operation to compute the final result.
  • The result is then sent back to the user.

31 of 32

Multi-stage Query Engine

  • Suitable for more complex queries involving joins, subqueries, CTE.
  • Scatters the query execution plan to the relevant servers.
  • Includes a data exchange layer which gathers the partial results from the servers.
  • Partial data is sent to one of the servers which performs any remaining computation to generate the final result.
  • The result is then sent back to the user.

32 of 32

Demo