Apache ZooKeeper

Apache ZooKeeper provides naming, configuration management, synchronisation and group services for distributed systems, presented through a hierarchical namespace modelled on a filesystem. It is replicated across an ensemble of servers, survives node failures without data loss, handles tens of thousands of requests per second with sub-millisecond in-memory messaging, and stamps every update with a monotonically increasing transaction identifier that gives clients a total ordering guarantee.

Being a coordination kernel rather than an application is the thing to understand first. Very few organisations set out to buy ZooKeeper; they adopt Kafka, Hadoop or HBase and inherit it, which makes it one of the most widely operated pieces of infrastructure that almost nobody chose deliberately.

Total ordering is the guarantee everything above it is built on. Because every update carries an increasing identifier, clients can reason about the sequence of changes rather than merely their content, which is what makes leader election and distributed locking implementable at all.

Operationally the ensemble is the thing to plan for. A quorum has to be maintained, which means an odd number of servers and enough separation that a single failure domain cannot take the majority, and getting that wrong is the usual cause of an outage attributed to the software.