Log Analytics with ELK Stack�

(Architecture for aggressive cost optimization and infinite data scale)

Denis D’Souza

DevOps Engineer

About me...

• Currently a DevOps engineer at Moonfrog Labs

​

• 6 + years working as DevOps Engineer, SRE and Linux administrator

Worked on a variety of technologies in both service-based and product-based organisations

​

• How do I spend my free time ?

Learning new technologies and Playing PC Games

​

​

www.linkedin.com/in/denis-dsouza

​

Who we are?

• A Mobile Gaming Company making mass market social games
• More than 5M+ Daily Active, 15M+ Weekly Active Users
• Real time, Cross platform games optimized for Primary Market(s) - India and subcontinent
• Profitable!

​

Current Scale

Peak Concurrent Playing Users : 500,000 (Half million)

Our problem statement

“ We were looking for a Log Analytics Platform that fits our requirements.

We evaluated, compared and decided to build a self managed ELK stack ”

​

1. Our business requirements
2. Choosing the right option
3. ELK Stack overview
4. Our ELK architecture
5. Optimizations we did
6. Backup and restore
7. Disaster recovery
8. Cost savings
9. Key takeaways

Our business requirements

• Log analytics platform (Web-Server, Application, Database logs)
• Data Ingestion rate: ~300GB/day
• Frequently accessed data: last 8 days
• Infrequently accessed data: 82 days (90 - 8 days)

• Uptime: 99.90
• Hot Retention period: 90 days
• Cold Retention period: 90 days (with potential to increase)
• Simple and Cost effective solution
• Fairly predictable concurrent user-base
• Not to be used for storing user/business data

Choosing the right option

* values are estimations taken from the ‘product pricing web-page’ of the respective products, they may not represent the actual values and are meant for the purpose of comparison only.�References:�https://www.splunk.com/en_us/products/pricing/calculator.html#tabs/tab2�https://www.sumologic.com/pricing/apac/�

ELK Stack overview

ELK Stack overview: Terminologies

• Index
• Shard
• Primary
• Replica
• Segment
• Node

References:

https://www.elastic.co/guide/en/elasticsearch/reference/5.6/_basic_concepts.html

Our ELK architecture

Our ELK architecture: Hot-Warm-Cold data storage (infinite scale)

Our ELK architecture: Size and scale

Optimizations we did

Application Side

• Logstash
• Elasticsearch

Infrastructure Side

• EC2
• EBS
• Data transfer

​

Optimizations we did: Application side

Optimizations we did: Logstash

Pipeline Workers:

• Adjusted "pipeline.workers" to x4 the number of Cores to improve CPU utilisation on Logstash server (as threads may spend significant time in an I/O wait state)

### Core-count: 2 ###�...�pipeline.workers: 8�...

Configuration Optimizations

logstash.yml

Workers: 8

Workers: 2

References:

https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html

Optimizations we did: Logstash

'info' logs:

• Separated application 'info' log to be store in a different index with retention policy of fewer days

​

​

if [sourcetype] == "app_logs" and [level] == "info" {

elasticsearch {

index => "%{sourcetype}-%{level}-%{+YYYY.MM.dd}"

...

​

if [sourcetype] == "nginx" and [status] == "200"

{

elasticsearch {

index => "%{sourcetype}-%{status}-%{+YYYY.MM.dd}"

...

'200' response-code logs:

• Separated Access log with '200' response-code be store in a different index with retention policy of fewer days

​

Filter Optimizations

filters.conf

References:

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

Optimizations we did: Logstash

Log ‘message’ field:

• Removed "message" field if there were no 'grok-failures' in logstash while applying grok patterns�(reduced storage footprint by ~30% per doc)

​

​

if "_grokparsefailure" not in [tags] {

mutate {

remove_field => ["message"]

}

}

​

Filter Optimizations

filters.conf

Eg:

Nginx Log-message: 127.0.0.1 - - [26/Mar/2016:19:09:19 -0400] "GET / HTTP/1.1" 401 194 "" "Mozilla/5.0 Gecko" "-"

​

Grok Pattern: %{IPORHOST:clientip} (?:-|(%{WORD}.%{WORD})) %{USER:ident} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{QS:forwarder}

​

Optimizations we did: Application side

Optimizations we did: Elasticsearch

JVM heap vs non-heap memory:

• Optimised JVM heap-size by monitoring the GC interval, this helped in efficient utilization of system Memory (33% for JVM, 66% for non-heap) *

​

### Total system Memory 15GB ###

-Xms5g

-Xmx5g

​

Configuration Optimizations

jvm.options

​

Heap too small

Heap too large

Optimised Heap

* Recommended heap-size settings by Elastic:�https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html

Optimizations we did: Elasticsearch

Shards:

• Created templates with number of shards which are multiples of the number of Elasticsearch nodes �(helps fix issues with shards distribution imbalance which resulted in uneven disk, compute resource usage)

​

### Number of ES nodes: 5 ###

{

...

"template": "appserver-*",

"settings": {

"number_of_shards": "5",

"number_of_replicas": "0",

...

}

}'

​

Template configuration

Trade-offs:

• Removing replicas will result in search queries running slower as replicas are used while performing search operations
• It is not recommended to run production clusters without replicas

Replicas:

• Removed replicas for the required indexes (50% savings on storage cost, ~30% reduction in compute resource utilization)

Optimizations we did: Infrastructure side

AWS

• EC2
• EBS
• Data transfer (Inter AZ)

Spotinst platform allows users to reliably leverage excess capacity, simplify cloud operations and save 80% on compute costs.

Optimizations we did: Infrastructure side

Optimizations we did: EC2 and spot

- Stateful EC2 Spot instances:

• Moved all ELK nodes to run on spot instances �(Instances maintain IP address, EBS volumes)

Recovery time: < 10 mins

Trade-offs:

• Prefer using previous generation instance types to reduce frequent spot take-backs

Optimizations we did: EC2 and spot

Auto-Scaling:

• Performance/time based auto-scaling for Logstash Instances

Optimizations we did: Infrastructure side

Optimizations we did: EBS

- "Hot-Warm" Architecture:

• "Hot" nodes: store active indexes, use GP2 EBS-disks (General purpose SSD)�
• "Warm" nodes: store passive indexes, use SC1 EBS-disks (Cold storage)�(~69% savings on storage cost)

​

node.attr.box_type: hot

...

"template": "appserver-*",�"settings": {

"index": {

"routing": {

"allocation": {

"require": {

"box_type": "hot"}

}

}

},

...�

elasticsearch.yml

Template configuration

Trade-offs:

• Since "Warm" nodes are using SC1 EBS-disks,�they have lower IOPS, throughput this will result in search operations being comparatively slower

References:

https://cinhtau.net/2017/06/14/hot-warm-architecture/

Optimizations we did: EBS

Moving indexes to "Warm" nodes:

• Reallocated indexes older than 8 days to "Warm" nodes
• Recommended to perform this operation during off-peak hours as it is I/O intensive

​

actions:

1:

action: allocation

description: "Move index to Warm-nodes after 8 days"

options:

key: box_type

value: warm

allocation_type: require

wait_for_completion: true

timeout_override:

continue_if_exception: false

filters:

- filtertype: age

source: name

direction: older

timestring: '%Y.%m.%d'

unit: days

unit_count: 8

...�

Curator configuration

References:

https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x

Optimizations we did: Inter-AZ data transfer

Single Availability Zone:

• Migrated all ELK node to a single availability zone �(reduce inter AZ data transfer cost for ELK nodes by 100%)
• ​
• Data transfer/day: ~700GB�(Logstash to Elasticsearch: ~300GB,�Elasticsearch inter-communication: ~400GB)

​

​

Trade-offs:

• It is not recommended to run production clusters in a single AZ as it will result in downtime and potential data loss in case of AZ failures

Data backup and restore

Using S3 for index Snapshots:

• Take snapshots of indexes and store them in S3

curl -XPUT "http://<domain>:9200/_snapshot/s3_repository/snap1?pretty?wait_for_completion=true" -d'

{

"indices": "index_1,index_2",

"ignore_unavailable": true,

"include_global_state": false

}

​

​

Backup:

References:�https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

https://medium.com/@federicopanini/elasticsearch-backup-snapshot-and-restore-on-aws-s3-f1fc32fbca7f

Data backup and restore

curl -s -XPOST --url "http://<domain>:9200/_snapshot/s3_repository/snap1/_restore" -d'

{

"indices": "index_1,index_2",

"ignore_unavailable": true,

"include_global_state": false,

}'

​

​

On-demand Elasticsearch cluster:

• Launching a on demand ES cluster and importing the snapshots from S3

Existing Cluster:

• Restore the required snapshots to existing cluster

​

Restore:

Disaster recovery

Data corruption:

• List out indexes with status as ‘Red’
• Deleted the corrupted indexes
• Restore indexes from S3 snapshots
• Recovery time: depends of size of data

​

​

​

Node failure due to AZ going down:

• Launch a new ELK cluster using AWS cloud formation templates
• Do the necessary config changes in Filebeat, Logstash etc.
• Restore the required indexes from S3 snapshots
• Recovery time: depends on provisioning time and size of data

Node failures due to underlying hardware issue:

• Recycle node in Spotinst console �(will take AMI of root volume, launch new instance, attach EBS volumes, maintain private IP)
• Recovery time: < 10 mins/node

Snapshot restore time (estimates):

• < 4mins for a 20GB snapshot (test-cluster: 3 nodes, multiple indexes with 3 primary shards each, no replicas)

Cost savings: EC2

Cost savings: Storage

Total Savings

* Total savings are exclusive of some of the application-level optimizations done

Our Costs vs other Platforms

Savings over traditional ELK stack: 71% *

* values are an estimation taken from the ‘product pricing web-page’ of the respective products, they may not represent the actual values and are meant for the purpose of comparison only

* Total savings are exclusive of some of the application-level optimizations done

Key Takeaways

ELK Stack Scalability:

• Logstash: auto-scaling
• Elasticsearch: overprovisioning (nodes run at 60% capacity during peak load), predictive vertical/horizontal scaling

Handling potential data-loss while AZ is down:

• DR mechanisms in place, daily/hourly backups stored in S3, Potential chances of data loss of about 1 hour
• We do not store user-data or business metrics in ELK, users/business will not be impacted

Handling potential data-corruptions in Elasticsearch:

• DR mechanisms in place, recover index from S3 index-snapshots

​

Managing downtime during spot take-backs:

• Logstash: multiple nodes, minimal impact
• Elasticsearch/Kibana: < 10min downtime per node
• Use previous generation instance types as spot take-back chances are comparatively low

Key Takeaways

Handling back-pressure when a node is down:

• Filebeat: will auto-retry to send old logs
• Logstash: use ‘date’ filter for document timestamp, auto-scaling
• Elasticsearch: overprovisioning

​

Other log analytics alternatives:

• We have only evaluated ELK, Splunk and Sumo Logic

​

ELK stack upgrade path:

• Blue Green deployment for major version upgrade

​

Reflection

• We built a platform tailored to our requirements, yours might be different...

​

• Building a log analytics platform is not rocket science, but it can be painfully iterative if you are not aware of the options

​

• Be aware of the trade-offs you are ‘OK with’ and you can roll out a solution optimised for your specific requirements

​

Thank you!

Denis D’Souza

DevOps Engineer

Happy to take your questions..

Copyright Disclaimer: All rights to the materials used for this presentation belongs to their respective owners..

www.linkedin.com/in/denis-dsouza