On a system on which users frequently over-writes existing rows/columns, after upgrading Cassandra from 0.7.9 to 1.0.7, read/write performance is not optimal.
Single file with single client
Replication factor must be set greater than 1.
1. Update row on Cassandra 0.7.9 (file write)
2. Upgrade Cassandra 0.7.9 to 1.0.7 (feature start)
3. Read the updated row on Cassandra 1.0.7 (file read)
Yes. When reading the updated column, nodetool cfhistograms shows cassandra touching two SSTables.
The performance is lower than expected for read and write after upgrade.
Because the drop in performance noticed after upgrade, taking a look at the IO logs with the tool nodetool cfhistogram will be useful. From the IO logs, we can see for update row operation, multiple replicas are read from different nodes. This behavior is strange because because we are doing unnecessary reads. The correct way to do update row operation is to first check the max timestamp value (newest time). The max timestamp value serves as the index which points to the location of the newest replica. From that point, cassandra will only need to fetch that specific replica. After analysing the sequence of events, we noticed the max timestamp field is missing after upgrading Cassandra to newer version. Due to this failure, the next update row operation must first read in multiple replica for comparison and return the newest copy.
Note: Cassandra creates tombstones for every update operation. If max timestamp value is missing, not only does Cassandra need to compare the replicas, Cassandra may need to read in the tombstones for comparison as well. This would intensify the existing problem.
Max timestamp value is not carried over from previous version of Cassandra. So when reading the value, multiple replicas must be obtained, their timestamps compared, before returning the replica with the newest time stamp. We can’t get the newest replica right away thus must compare the returned result thus gives lower performance
The patch made sure max timestamp value gets carried over to upgraded cassandra version for all SSTable content (including tombstones).