After some discussion the proposed plan is to implement our own agreement service using the existing VoltDB plumbing. We will create a new type of site called an agreement site that contains an initiator and a restricted priority queue instead of just a restricted priority queue. These agreement sites will communicate via the existing mailbox system and will detect failures using the mailbox system. A mutation against agreement state is identical to a multi-partition txn so they will use the same failure handling code as regular execution sites.
I will configure the initiators so they don't send heartbeats and we will instead rely on the heartbeats requests from the RPQs
I plan to reuse org.apache.zookeeper.server.DataTree for the data and execution model as well as the existing ZooKeeper wire protocol and client library. A plus here is that someone can point us to an existing ZK cluster or use VoltDB as a ZK cluster.
Since you are poking around with Zookeeper, you may want to give Doozer a quick look. Not sure if it supports re-configuration yet, but its basically another reliable configuration service like Zookeeper. It is written in Go, but compiles to a single executable with no dependencies, so if it were useful, you could probably still make it work.
ReplyDeletehttps://github.com/ha/doozerd#readme
Good luck heading down this path. These things are tricky to get right!