When performing “trunk single-pass streaming” on very large column family will fail and that node raise out of memory exception.
yes, out of memory exception
basic configuration with streaming
1. write large amount of data into a column family (file write)
2. perform trunk single-pass stream on a column family that is larger than “inMemoryLimit” (feature start)
When performing trunk single-pass streaming operation on large column family, we run out of memory.
First we check whether the column family we want to stream is larger than our available memory (inMemoryLimit value). The answer is yes and the current code doesn’t handle this in a graceful way.
Streamed column family data cannot fit inside the memory and no graceful error handling is present.
Better error handling. Fix is to write a portion of the streamed data to disk and leave a portion of it in memory.