ZooKeeper Architecture in KubeBlocks

This page describes how KubeBlocks deploys an Apache ZooKeeper ensemble on Kubernetes — covering the resource hierarchy, pod internals, the ZAB consensus protocol, and traffic routing.

Resource Hierarchy

KubeBlocks models a ZooKeeper ensemble as a hierarchy of Kubernetes custom resources:

Cluster  →  Component  →  InstanceSet  →  Pod × N

ZooKeeper requires an odd number of members (3, 5, or 7) to maintain a voting quorum. KubeBlocks assigns a unique myid to each pod, derived from its ordinal, which persists across restarts.

Containers Inside Each Pod

Every ZooKeeper pod runs one container:

Each pod mounts two PVCs:

• data PVC → /bitnami/zookeeper/data — snapshot files and the ZooKeeper data tree
• snapshot-log PVC → /bitnami/zookeeper/log — transaction log files

Node Roles

High Availability via ZAB Protocol

ZooKeeper provides HA through the ZooKeeper Atomic Broadcast (ZAB) protocol, which guarantees total order of updates and crash-recovery:

A 3-member ensemble tolerates 1 failure; a 5-member ensemble tolerates 2 failures.

Leader Election Process

When the leader becomes unavailable:

1. All remaining servers detect the missing heartbeat and enter leader election mode
2. Each server votes for the candidate with the highest zxid (transaction ID) and myid
3. The server that collects a quorum of votes becomes the new leader
4. The new leader synchronizes followers before resuming write operations
5. Leader election typically completes in 200 ms to 2 seconds under normal network conditions

Traffic Routing

KubeBlocks creates three services for each ZooKeeper ensemble:

• Write / coordination traffic: connect to {cluster}-zookeeper:2181 — the Endpoints always point at the current leader. Write requests go directly to the leader without any forwarding overhead.
• Read-heavy workloads: connect to {cluster}-zookeeper-readable:2181 — distributes client connections across all ensemble members (leader and followers). Followers serve read requests locally; write requests are still forwarded to the leader by the follower that receives them.

Port 7000 (Prometheus metrics) is exposed on each pod but is not included in any ClusterIP service. To scrape metrics, connect directly to the pod on port 7000, or configure a PodMonitor targeting port 7000 on the headless service.

Quorum and leader election traffic (ports 2888 and 3888) uses the headless service, where each ensemble member is individually addressable by its stable pod DNS name:

{pod-name}.{cluster}-zookeeper-headless.{namespace}.svc.cluster.local

The zoo.cfg configuration file references all peer addresses using these stable DNS names, ensuring correct cluster membership after pod restarts or rolling updates.

Automatic Failover

When a ZooKeeper ensemble member fails:

1. Member goes offline — peers detect the missing heartbeat within the session timeout (default 2× tick time)
2. Leader election (if the lost member was the leader) — surviving members elect a new leader in milliseconds to seconds
3. Write continuity — as long as a quorum remains available, all write and read operations continue normally
4. Pod recovery — when the failed pod restarts, it reads its myid from the PVC, contacts the leader, and syncs any missed transactions before rejoining the ensemble

System Accounts

KubeBlocks manages the following ZooKeeper system account. The password is auto-generated and stored in a Secret named {cluster}-{component}-account-admin (replace {cluster} and {component} with your Cluster metadata.name and the ZooKeeper component name, typically zookeeper).