I moaned about buffered IO in a previous post. Turns out that there is another penalty. fsyncing every X number of bytes in order to bound memory usage falsely communicates to the OS that the IO is more important than it is. If the snapshot volume and log volume are the same then there will be two processes calling fsync concurrently and no priority difference between them from the OS perspective. The truth is that I would rather take a hit in throughput for one process and get better fsync latency in the other. The interaction between the two in terms of how their contention effects throughput is also pretty muddy.
Maybe the flaw lies with fsync as an API? Right now it means flush immediately, when sometimes all you want to know is when something has been flushed. The latter meaning is only relevant because I need to rate limit, and my need to rate limit is caused by buffered IO. I can't use O_DIRECT from Java, but I wonder how that would differ from doing a regular write and then calling fsync?