Zookeeper As A Service For Shard Assignments

How zookeeper could be used in a shard assignment like service, allowing the “application to automatically recover from faults without human intervention”:

Another example arises when you have some partitioned resource (database, message streams, file storage, distributed actor system, etc.) and need to decide which partition to assign to which node. As new nodes join the cluster, some of the partitions need to be moved from existing nodes to the new nodes in order to rebalance the load. As nodes are removed or fail, other nodes need to take over the failed nodes’ work. These kinds of tasks can be achieved by judicious use of atomic operations, ephemeral nodes 202211232312, and notifications in ZooKeeper. If done correctly, this approach allows the application to automatically recover from faults without human intervention. It’s not easy, despite the appearance of libraries such as Apache Curator that have sprung up to provide higher-level tools on top of the ZooKeeper client API—but it is still much better than attempting to implement the necessary consensus algorithms from scratch, which has a poor success record.

Created from: Zookeeper 202211232246

uid: 202211232309 tags: #distributed-systems

Date

February 22, 2023

Up next

202211241007 The benefits of keeping a done list Source: https://www.oliverburkeman.com/donelist This is why I’m such an enthusiastic proponent of keeping a “done list”, which starts empty,

Previously

Atomic Commits How to do atomic commits? Two-phase commit. Two-phase commit has a single coordinator and multiple participants. First phase: The coordinator sends