CASSANDRA-5256

1. Symptom

Getting memory was freed assertion during major compaction of SSTables.

 

Category (in the spreadsheet):

wrong computation

1.1 Severity

critical        

1.2 Was there exception thrown? (Exception column in the spreadsheet)

yes java.lang.AssertionError: Memory was freed

 

1.2.1 Were there multiple exceptions?

no

 

1.3 Was there a long propagation of the failure?

no

 

1.4 Scope of the failure (e.g., single client, all clients, single file, entire fs, etc.)

single file

 

Catastrophic? (spreadsheet column)

no 

2. How to reproduce this failure

2.0 Version

1.2.2

2.1 Configuration

standard configuration

# of Nodes?

1

2.2 Reproduction procedure

1) initiate a compaction with nodetool h localhost compaction command (feature start)

 

Num triggering events

1

 

2.2.1 Timing order (Order important column)

NA

2.2.2 Events order externally controllable? (Order externally controllable? column)

No. Datarace.

2.3 Can the logs tell how to reproduce the failure?

yes

2.4 How many machines needed?

1

3. Diagnosis procedure

Error msg?

yes

3.1 Detailed Symptom (where you start)

Memory was freed assertion was raised when doing a compaction.

3.2 Backward inference

After takling a look at the logs, we find that a race condition is allowing SSTable to participate in multiple concurrent compactions. The the frist compaction finishes, it frees the compression metadata. When the second compaction operation checks the meatadata, we cause the assertion error.

3.3 Are the printed log sufficient for diagnosis?

yes

 

4. Root cause

race conditon which allows a single sstable to participate in multiple concurrent compaction.

4.1 Category:

semantic

4.2 Are there multiple fault?

no

4.2 Can we automatically test it?

yes

5. Fix

5.1 How?

make sure we don’t allow compaction of a single SSTable in multiple compactor concurrently.

5.2 Exception behavior?

no more exception