2 of 17

Collection of Elasticsearch, Logstash, Kibana from Elastic

Search engines that scan and process the entire text in a document prior to indexing, are so-called full-text search engines. Lucene, which Elasticsearch and Solr are built on top of, is an example of a full-text search engine.

�

https://www.elastic.co/blog/found-indexing-for-beginners-part2/

4 of 17

Elasticsearch

Easy to deploy (minimum configuration)
Scales vertically and horizontally
Easy to use API
Written in Java (beware of jvm heap, current support java 11)
Modules for most programming/scripting languages
Actively developed with good online documentation from its website
High availability
It’s free
Scoring…

5 of 17

Elasticsearch elements

Shards

• Single instance of Lucene on a node

• Can be primary or replica

Index

Mapping of shards to nodes
Like a database within a relational database

Nodes

• Keeps a copy of the index

Maintain primary and replica shards

https://www.elastic.co/blog/found-indexing-for-beginners-part1/

Data in Elasticsearch is organized into indices. Each index is made up of one or more shards. Each shard is an instance of a Lucene index, which you can think of as a self-contained search engine that indexes and handles queries for a subset of the data in an Elasticsearch cluster.

As data is written to a shard, it is periodically published into new immutable Lucene segments on disk, and it is at this time it becomes available for querying. This is referred to as a refresh. How this works is described in greater detail in Elasticsearch: the Definitive Guide.

As the number of segments grow, these are periodically consolidated into larger segments. This process is referred to as merging. As all segments are immutable, this means that the disk space used will typically fluctuate during indexing, as new, merged segments need to be created before the ones they replace can be deleted. Merging can be quite resource intensive, especially with respect to disk I/O.

The shard is the unit at which Elasticsearch distributes data around the cluster. The speed at which Elasticsearch can move shards around when rebalancing data, e.g. following a failure, will depend on the size and number of shards as well as network and disk performance.

6 of 17

https://www.elastic.co/blog/found-indexing-for-beginners-part3

9 of 17

Terminologies

10 of 17

Installation type

Bare install
Docker
Kube

11 of 17

Various data sources

Kafka
Log
Api
CSV

12 of 17

Logstash

Log parsing

15 of 17

Kibana

front end for visualizing dashboard for elasticsearch
Realtime search and index

http://localhost:5601

16 of 17

access1M.log

client_hostname.csv

17 of 17

Exercise –web log visualization

Given: A dump file contains web log data

Goal: Visualizations in Kibana
Use logstash to filter data
Create a dashboard with various visualizations
Example of visualization

Count only the number of GET /product/* by day
Count the frequency of accesses by types eg. Mozzila, chrome, android, etc.
Count the frequency of accesses by hostname

https://www.elastic.co/blog/importing-csv-and-log-data-into-elasticsearch-with-file-data-visualizer

https://www.bmc.com/blogs/elasticsearch-load-csv-logstash/

1 of 17

2 of 17

3 of 17