CASSANDRA-1477

1. Symptom

We drop a column family, recreate it, then read data from a key that existed before dropping, it shouldn’t exist now on cassandra. If we do those operation fast enough, we get old data. If the procedure happen slow enough, we get this exception: org.apache.thrift.TApplicationException: Internal error processing get_slice

Category (in the spreadsheet):

wrong computation

1.1 Severity

blocker

1.2 Was there exception thrown? (Exception column in the spreadsheet)

yes.

On client: "org.apache.thrift.TApplicationException: Internal error processing

get_slice"

On server: [org.apache.cassandra.thrift.Cassandra$Processor] (pool-1-thread-4 -

Internal error processing get_slice

java.lang.RuntimeException: java.util.concurrent.ExecutionException:

java.io.IOError: java.io.FileNotFoundException:

1.2.1 Were there multiple exceptions?

yes. See above section.

1.3 Was there a long propagation of the failure?

1.4 Scope of the failure (e.g., single client, all clients, single file, entire fs, etc.)

single file

Catastrophic? (spreadsheet column)

2. How to reproduce this failure

2.0 Version

0.7 beta 2

2.1 Configuration

standard configuration

# of Nodes?

2.2 Reproduction procedure

1) drop column family (file write)

2) recreate the dropped column family (file write)

3) read data from a key that existed before dropping, but doesn’t exist now (file read)

Num triggering events

2.2.1 Timing order (Order important column)

yes (operations must be performed very fast or very slow)

2.2.2 Events order externally controllable? (Order externally controllable? column)

Yes

2.3 Can the logs tell how to reproduce the failure?

yes

2.4 How many machines needed?

3. Diagnosis procedure

Error msg?

yes. See exception section.

3.1 Detailed Symptom (where you start)

We performing these three actions in succession: drop column family, recreate it, read data from a key that existed before dropping, but doesn’t exist now. If done too fast, we receive stale data. If done too slow we get "org.apache.thrift.TApplicationException: Internal error processing get_slice" on client side and java.lang.RuntimeException: java.util.concurrent.ExecutionException:, java.io.IOError: java.io.FileNotFoundException: on the server side.

3.2 Backward inference

With some domain knowledge, the developer knows the way to cleanup after dropped column family has changed. Previous to the change, there is a block on delete. Now instead of blocking on delete, we mark the column family as compacted and waiting for normal cleanup to do the job. With this change, we introduce a race condition.

4. Root cause

race condition between deletion and creation of column family.

4.1 Category:

semantic

4.2 Are there multiple fault?

4.2 Can we automatically test it?

yes

5. Fix

5.1 How?

When creating a column family, ensure that its data directory, if it exists, is empty. This avoids the race condition.