Export overflow

VoltDB allows you to declare export only tables. These are tables that you can insert a row into, and the row is not actually stored in the database. If you attempt to select from an export only table you will get a compilation error.

Rows that are inserted into an export only table are batched together and streamed to a companion system that speaks the export protocol. These rows are streamed transactionally so the data is not removed from VoltDB until it is acknowledged as durable by the companion system. One concern is what happens when the companion system is unavailable or unable to keep up?

Pre 1.3 the data would be buffered in memory on the Java heap. Post 1.3 the data is stored on disk with the head of the stream stored in memory.

If you take a snapshot then all of the export data is flushed from memory to disk. This means you can safely shutdown VoltDB while the companion system is unavailable and not lose any export data. When you reload a snapshot you are guaranteed to have available any unconsumed export data generated before the snapshot. If you are restoring a snapshot after a system crash you may have export data that was generated from after the snapshot in which case the export data will be truncated to the txnid of the snapshot.

This means that your export data loss in a crash will match your regular dataloss in 1.3 which gives you something of a sane starting point. Once command logging is available you will be able to regenerate the truncated export data by replaying the command log.