CASSANDRA-2829

1. Symptom

Background:

Memtables are a cache to SSTable on disk. Only dirty memtable needs to be flushed to disk. Only the dirty memtables are used to check if a commit log segment is obsolete. A rare race condition can result in the log segments not be deleted even though the data has been flushed.

Commit log reaches maximum allowed size. Logs inside commitlog is permanently dirty and cannot be removed.

Category (in the spreadsheet):

early termination,

1.1 Severity

Critical

1.2 Was there exception thrown? (Exception column in the spreadsheet)

IOException (out of disk space)

 

1.2.1 Were there multiple exceptions?

no

 

1.3 Was there a long propagation of the failure?

no (just lots of repetition of the same kind of failure)

 

1.4 Scope of the failure (e.g., single client, all clients, single file, entire fs, etc.)

single node (for the affected node)

 

Catastrophic? (spreadsheet column)

No. There is no dataloss and can be manually recovered

2. How to reproduce this failure

2.0 Version

0.8.3

2.1 Configuration

No special configuration

 

# of Nodes?

1

2.2 Reproduction procedure

1. Write to cf1 and flush and do not write to it again (file write)

2. Replay commit log (file read)

3. Write to CF2 and flush. (file write)

After the 3 steps, CF1’s commit log entry will remain in commit log forever

 

Num triggering events

3

 

2.2.1 Timing order (Order important column)

yes

2.2.2 Events order externally controllable? (Order externally controllable? column)

yes

2.3 Can the logs tell how to reproduce the failure?

yes

2.4 How many machines needed?

1

 

3. Diagnosis procedure

Error msg?

yes

3.1 Detailed Symptom (where you start)

We first noticed that the commit log directory is filled with 7GB+ (20hrs) of commit log on one of the node. This symptom is not supposed to happen.

3.2 Backward inference

When taking a closer look, we see that memtables corresponding to entries in the commit log has already been purged to disk. This means there is no data loss and the entries in the commit log is obsolete. With a bit of domain knowledge, we determined that there is a race condition with purging the obsolete commit log entry and flushing the memtable.

 

4. Root cause

race condition with purging the obsolete commit log entry and flushing the memtable.

4.1 Category:

Semantic

4.2 Are there multiple fault?

no

4.2 Can we automatically test it?

yes

5. Fix

5.1 How?

Added two non-perisistent fields:

1) most recent write at

2)oldest write/replay position at

By checking these fields, we can determine whether if commit log entry is obsolete automatically and remove them when its not needed.

Scope of the failure

all nodes which are affected which the commit log directory reached maximum allowed size