Memtables are a cache to SSTable on disk. Only dirty memtable needs to be flushed to disk. Only the dirty memtables are used to check if a commit log segment is obsolete. A rare race condition can result in the log segments not be deleted even though the data has been flushed.
Commit log reaches maximum allowed size. Logs inside commitlog is permanently dirty and cannot be removed.
IOException (out of disk space)
no (just lots of repetition of the same kind of failure)
single node (for the affected node)
No. There is no dataloss and can be manually recovered
No special configuration
1. Write to cf1 and flush and do not write to it again (file write)
2. Replay commit log (file read)
3. Write to CF2 and flush. (file write)
After the 3 steps, CF1’s commit log entry will remain in commit log forever
We first noticed that the commit log directory is filled with 7GB+ (20hrs) of commit log on one of the node. This symptom is not supposed to happen.
When taking a closer look, we see that memtables corresponding to entries in the commit log has already been purged to disk. This means there is no data loss and the entries in the commit log is obsolete. With a bit of domain knowledge, we determined that there is a race condition with purging the obsolete commit log entry and flushing the memtable.
race condition with purging the obsolete commit log entry and flushing the memtable.
Added two non-perisistent fields:
1) most recent write at
2)oldest write/replay position at
By checking these fields, we can determine whether if commit log entry is obsolete automatically and remove them when its not needed.
Scope of the failure
all nodes which are affected which the commit log directory reached maximum allowed size