After some discussion the proposed plan is to implement our own agreement service using the existing VoltDB plumbing. We will create a new type of site called an agreement site that contains an initiator and a restricted priority queue instead of just a restricted priority queue. These agreement sites will communicate via the existing mailbox system and will detect failures using the mailbox system. A mutation against agreement state is identical to a multi-partition txn so they will use the same failure handling code as regular execution sites.
I will configure the initiators so they don't send heartbeats and we will instead rely on the heartbeats requests from the RPQs
I plan to reuse org.apache.zookeeper.server.DataTree for the data and execution model as well as the existing ZooKeeper wire protocol and client library. A plus here is that someone can point us to an existing ZK cluster or use VoltDB as a ZK cluster.