Redis server wiped out entire DB when “maxmemory” is met with multiple slaves connected.
Yes. command not allowed when used memory > 'maxmemory'
No, but this error msg will appear many times.
All data erased! Quite significant.
Enable maxmemory configuration and multiple slaves
1. Enable maxmemory configuration (config change)
2. Attach multiple slaves (add node)
In this order
Yes --- slaves connections
Data loss + “command not allowed when used memory > ‘maxmemory’”...
From the code that prints the log:
if (server.maxmemory) freeMemoryIfNeeded();
if (server.maxmemory && (c->cmd->flags & REDIS_CMD_DENYOOM) &&
zmalloc_used_memory() > server.maxmemory)
addReplyError(c,"command not allowed when used memory > 'maxmemory'");
---- Once it’s triggered, something must be wrong!
We can see the error happens after freeMemoryIfNeeded did not successfully free any memory...
When maxmemory is met, Redis tries to expire keys. But the memory usage calculation accounted the output buffer size for the slave, which keeps growing with the “DEL” command sending to the slaves. Therefore the more Redis tries to expire keys, the larger the memory consumption is, and therefore in the end, Redis deleted all the keys!!!
The issue happens for the following reason:
Incorrect error handling (handled)
--- the initial error is not handled correctly.
Quite complicated. Basically when free-ing the memory, make sure the output buffer size is not counted torwards memory usage!