Zombie Leaders

The hardest part of being a distributed leader isn’t getting elected, it’s realizing when to step down.

In distributed control planes, we often reach for CAS + TTL to implement leader election without the operational overhead of a dedicated Raft/Paxos ring. It’s a pragmatic choice for many services.

But there is a subtle liveness vs. safety gap that often creates “Zombie Leaders.”

The mechanism relies on a Lease (TTL). The assumption is that if the TTL expires, the leader has stepped down. But in a distributed system, time is not a reliable clock.

We’ve seen this failure mode in production:

Leader acquires lease. Leader hits a Stop-the-World GC pause or a storage stall. TTL expires during the pause. New leader is elected. Old leader wakes up, unaware time has passed, and flushes a stale write. If your storage layer doesn’t support fencing tokens (like a strictly increasing epoch number), your “single leader” guarantee just became a race condition.

Implementing CAS election is easy. Ensuring the leader is fenced off effectively when it loses that election is where the actual engineering happens.

If you are relying on wall-clock time for correctness in a distributed system, you are likely already incorrect.

Hrishikesh Alshi