Things I Wish I'd Known About Elasticsearch When I Started Use It As NoSQL

Jimin Hsieh

Agenda

• Summary
• What is Elasticsearch?
• What I wish I’d known since the beginning.

Summary

• Think hard (and write down!) how you will access your data, then carefully model your table to satisfy those access patterns.
• Semi-structured data doesn’t mean you can put arbitrary data into it.

What is Elasticsearch?

• Distributed data store
• NoSQL
• JSON Document
• Semi-structured and schema free

Architecture

https://github.com/exo-archives/exo-es-search#es-architecture

Inverted Index

https://community.hitachivantara.com/s/article/search-the-inverted-index

Analogy between ES and SQL

• Index = database
• Type = table
• Document = row
• Field = column

• Mapping = schema

Document

{

"_index": "hanmvngj",

"_type": "stresstest",

"_id": "7y2lLXkB3FI9wq1OkmCn",

"_score": 1,

"_source": {

"f": "amhksvjeocyogkmlzibvmpakuc",

"i": "xcwclqmpniijqgeebapkspmcfvzqoypitjvzbcqtba",

"ak": "gtomqdwncpymlvfljsnhufojytjqouvxsuidoqwtttplvrm",

"qnxxdsf": "gqrktshotoqijetacvbnqbpdhbumhomuutbiqdqjsfqros",

"qhxcumzqc": "kuiwcgnrcpzggeldypdvnijq",

"hwx": "cyexgvrrcdismskdcflwgytcijkoibuvsqxijlgwlxjv",

"eih": "x"

}

}

Mapping

{

"hanmvngj": {

"mappings": {

"properties": {

"afh": {

"type": "text",

"fields": {

"keyword": {

"type": "keyword",

"ignore_above": 256

}

}

}

}

}

}

}

Data Type

• JSON data type
• String
• Number
• Boolean
• Null
• Array
• Object

• Elasticsearch data type
• Number
• Long
• Integer
• Short
• Byte
• Text
• Keyword
• …ect

Mapping

• Don’t use default dynamic mappings.
• Wrong data type
• Don’t use default dynamic string mappings.

Mapping

• Dynamic field mapping
• Pick up the type of data for you
• Explicit mapping
• Decide on your own
• Define your dynamic templates
• Define your properties

Text vs Keyword

Analyzer

• Default Analyzer
• Standard analyzer
• Standard tokenizer
• Lowercase token filter
• Stop token filter

Pagination

• Don’t use from-size
• Scroll search
• Sliced Scroll
• Point in time + search after
• You can use it only if you pay for the commercial version and after 7.10.

Must vs Filter

• Filter will ignore scoring.
• Filter could be cached.

Increase write throughput

• Turn off replica
• index.refresh_interval

Turn Off Replica

https://learning.oreilly.com/library/view/elasticsearch-the-definitive/9781449358532/assets/elas_0401.png

Index Refresh Interval

https://tech.ebayinc.com/assets/Uploads/_resampled/ResizedImageWzgwMCw1MzZd/Picture23.png

Mapping Explosion

• NoSQL prefers flatten data model.
• When you have too many fields in your document
• index.mapping.total_fields.limit = 1000 (default)
• If your field mappings contain a large, arbitrary set of keys, consider using the flattened data type. By official tips
• But you have to pay for the commercial version.

Spark Elasticsearch

• Use a maximum limit of task writing to Elasticsearch
• coalesce
• Be aware imbalanced data

Java API

• Scrolled search
• Implementation
• https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-search-scrolling.html#java-search-scrolling
• https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-search-scroll.html#java-rest-high-search-scroll-example
• Clear search
• Reduced unnecessary return
• Source filtering

Reference

• Beginner's Crash Course to Elastic Stack
• Elasticsearch Do's, Don'ts and Pro-Tips - Itamar Syn Hershko
• Elasticsearch Performance Tuning Practice at eBay
• Operating Elasticsearch with Ease at Scale
• Elasticsearch from the bottom up
• Elasticsearch: The Definitive Guide
• elasticsearch.cn

FAQ

Thank you for your attention.