Compaction of Cassandra hints can get stuck in a loop -- the thread doing compaction will hang (but directly user-visible).
No, there is no exception thrown
no, the problem does not have long propagation. It is immediately affected.
1.2.0 beta 1
Standard default configuration
1. start node
2. force compaction check in ACS kicks off
3. observe looping compaction on the hints, but only compacting the last SSTable iteratively
Yes, need to trigger compaction of hints
In the logs, we first see an iterative compaction on one file.
INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
for 1 keys at 5.912220MB/s. Time: 748ms.
Looking at the logs, we can see that the machine does not hand anything off, so everything in these SStables must be tombstones. Then CASSANDRA-3442 was believed to have caused the issue where it is performing single SSTable compaction if its droppable tombstone ratio is above threshold. However, there is a special case which compaction does not drop tombstone when the key in compacting sstable appears in another sstable.
The fix changed the way a SSTable will be purged. The old method is performing single SSTable compaction if its droppable tombstone ratio is above threshold. However, there is a case where compaction does not drop tombstone. The new method supposed to drop tombstone is only done when the key tombstone belongs to does not appear in other sstables that are not compacting.