Art of Caching

Ways

Wins

Woes

Weird

Wisdom

Beauty of Caching: Trade offs

• A wise software engineer once said - “It Depends”

​

• How many of you implemented caching?

​

• How many of you said, lets not write caching logic here?

​

• How many of you faced, cache invalidation issues?

​

• How many of you built a backdoor to forcefully invalidate cache?

Intro

• Srinivas

​

• Founder @ Opti Owl

​

• Web Developer to SRE Lead @ Zomato

​

• 2 Mil Orders Per Month to 3+ Mil Orders Per Day

​

• Deployed Cultural Frameworks & Practices like Postmortems & FMEA at Zomato

Definition

Caching the process of

Storing a transformed copy in a storage system

Which is more efficient than the�Source storage system

How many here have seen

​

Dr Strange

-

Marvel Cinematic Universe ?

TOC

• Definition
• Ways & Wins
• Woes & Weird
• Wisdom
• Characteristics

Ways & Wins

Saving Latency

• Most Common Win
• Cache SQL Queries
• Cache Images on CDN

Saving Vendor API Cost

​

• Who heard about Ola Maps
• Google Maps Cost
• Caching Data (Complying to T&C)

Saving Database Cost

• Caching DynamoDB Reads
• Caching DynamoDB Writes (10x costly)
• DAX

​

Database Latency Distribution

​

​

​

​

Redis Latency Distribution

Saving Latency Distribution

Improving Resiliency

• Caching Expensive Queries
• Story: Deepi’s Followers

​

• Fallback on DB Failure

​

Caching Booleans

• Bloom Filters
• Caching Serviceability
• Caching Restaurant Open / Close
• Bit Set (Array)

Layered Caching

• Power Law Distribution

Layered Caching

• Central Distributed Cache (Redis / Memcache)
• Local Cache (Container Level - Sidecar or App Global)
• Request / Thread Local Cache (Thread / Go Routine Level)

Woes & Weird

​

Bill always comes Due

When Cache Increases Latency

• Batch GETs
• When fetching 1000+ items the probability of a single cache miss can cause additional latency than reduction in latency

Caching Null Values

Caching Null Values

• Inserts don’t delete cache, only updates delete cache
• So even after Insert, the reads will respond that row doesn’t exist

Not Caching Null Values

Not Caching Null Values

• Every query would go to the Database
• In high frequency scenarios even if the load on database should be low, it remains high

Thrashing

• Data is being removed due to eviction causing high miss rate
• High Eviction Fetch Ratio

To Evict or Not to Evict

• Variable Performance
• Harder to predict what is the ideal hit ratio you could achieve
• Perfer Expiry Tuning compared to Eviction Tuning

Compute Hot Nodes

• Power Law
• Too much throughput on specific keys
• Delhi, Bangalore Orders

Memory Hot Nodes

• Rare but happens
• Grouped Keys
• Hash Tags
• `user_{1234}_feat1`, `user_{1234}_feat2`

Bandwidth Hot Nodes

• Throughput so high that hot nodes form
• Power Law
• Levelled Caching
• App Level Replicated Cache
• user_123_c1, user_123_c2 etc,...

Bi Modal Behaviour

• Behaviour When cache is online is quite different from when cache is offline
• System is more fragile because of how it behaves differently for different cases
• Constant Work model to control bi-modal behaviour

Cache Poisoning

Cache Poisoning

• Incorporate serialisation, compression, encoding inbuilt into the caching package
• Write time validations

Backwards Compatibility

X Broke Prod

• Pre Prod breaks Prod
• Canary breaks Prod

Revert Breaks Prod

Timeout Propagation

Timeout Propagation

• Remove timeout on cache sets
• Don’t propagate timeout, set independent timeout

Irrecoverable Cache Wipe

• Timeout Propagation causes cache to be never filled

Cache Error is a Cache Miss?

Self Immolating Cache

External Cache Faster than In App Cache

• Fetching from a local cache sidecar is faster than fetching cache from memory
• Most In-memory cache implementations are not as concurrent as external caches

Not Serializing Local Cache

• Unsafe Modifications to the cached objects
• Error Prone Estimation of Total Cache Size
• Immutable Objects FTW

Serializing Local Cache

• High compute resources in serialization & deserialization
• High Garbage Collection due to serialization allocations.

Infinite Loop Caches

Object Size - Cascading Failure

• Local Cache Object Size > Remote Cache Object Size

Vs

• Local Cache Object Size < Remote Cache Object Size

Wisdom

• There is a reason for below saying

​

​

Cache Invalidation

• Separate Talk

​

​

Eviction Policy: LFU

• Most Ideal Solution
• Only use if you see strong Power Law Distribution
• Main Flaw: Localised Access Spikes skews the data
• Ensuring Decay of Frequency Counters

Eviction Policy: LRU

• Easier and More Performant Implementation than LFU
• LFU tree is modified on reads, while LRU tree is only modified on writes
• LFU lock times are higher than LRU
• LFU more memory than LRU (storing counters or approximate counters)

Eviction Policy: FIFO

• Even more easier and more performant implementation than LRU

​

• Newer variants like LP-DQ-FIFO are even more performant and achieves better hit ratios than LRU

Cache Schema Changes

• Incorporate version number hierarchically to upgrade the cache
• Schema Changes
• Cache Poisoning
• Example: `global_v1/user_v2/1234123`

Cache Consistency

• Consider Replica Lag
• Node Stickiness for a request
• Request Aware Local Cache Layer

Mind the Key Size, not just Value

• user_service_v1_location_module_v3_most_recent_location_feature_v4

Cache Miss Injection

• Finding Slow Queries
• Finding Queries that could cause you an outage during cache wipes

Retry Budgets

• Retry on errors
• Retry on timeouts
• Limit of 10-20% budget on retries to avoid overloading cache

Circuit Breakers

• Client Container protecting circuit breakers
• Server protecting circuit breakers

Mind Durability Guarantees

• Redis by default doesn’t support durability
• Memcache also doesn’t support durability

Tolerating Cache Errors

• Similar to retry budgets, designing error budgets i.e 1% of the errors are treated as misses rather than errors

Serialization

• Use Binary Serialisation techniques like Arrow or Protobuf
• Schema Data is often too much bloat

Compression

• As with any other data, duplication is often high in bigger objects
• Use different compression trade offs at different levels

Characteristics

Efficiency

• Latency
• Cost
• Storage Cost
• Access Cost
• Vendor Cost
• Fault Tolerance
• CPU
• Network

Efficiency Trade offs

• Latency
• Cost
• Storage Cost
• Access Cost
• Vendor Cost
• Fault Tolerance
• CPU
• Network

Transientness

• Not Necessarily Transient

​

• Especially if the storage cost of cache storage system is extremely cheap

Durability

• Durability of a cache system adds additional complexity

​

• Failure Modes

Source of Truth

• Cache System is not the source of truth

​

• Sometimes it might contain a new data, but it’s still a transformation of source data rather than source of truth

• More than 36% in Cost Optimization
• Book a Meeting to discuss on any problems you are facing in Platform / SRE / Devops / Cost

Opti Owl

Thank you

sri@optiowl.cloud

Srinivas Devaki

Schedule Meeting: https://cal.com/optiowl/srinivas

https://www.linkedin.com/in/eightnoteight/