Distributed locks feel like a solved problem. Grab a lock from Zookeeper, etcd, or Redis, do your work, release it. Simple, until the moment a client holding the lock stalls for a few seconds during a GC pause, or a network blip makes the lock service think a client died when it didn’t. That’s when a “safe” lock stops being safe, and the fix isn’t a better lock. It’s a fencing token.

Why Locks Alone Aren’t Enough

A distributed lock is really just a promise: “while I hold this lock, no one else is touching the resource.” The lock service enforces that promise by giving out leases with timeouts. If a client doesn’t renew its lease in time, the service assumes it’s dead and hands the lock to someone else.

The trouble is that assumption can be wrong. A client can be alive but unresponsive, stuck behind a long garbage collection pause, a slow disk write, or a delayed network packet. From the lock service’s point of view, the client is gone. From the client’s point of view, nothing happened, it’s still holding what it believes is a valid lock. It resumes execution, finishes its write to the shared resource, completely unaware that a second client has since acquired the same lock and is now writing too.

This is the classic split-brain moment for locks: two processes both convinced they have exclusive access, both acting on that belief. The lock service did its job correctly by its own rules. The problem is that “correctly acquired a lock” and “safe to act on the protected resource” are not the same guarantee, and most systems quietly assume they are.

What a Fencing Token Actually Does

A fencing token is a number that increases every time the lock changes hands. The lock service hands out this token along with the lock itself, and the resource being protected, whether that’s a database, a file, or a storage service, is made to check the token on every write and reject any write carrying a token older than the highest one it’s already seen.

So the sequence looks like this: client A acquires the lock with token 33, gets delayed, and client B later acquires the same lock with token 34. If A eventually wakes up and tries to write with token 33, the resource rejects it because it has already accepted writes tagged with 34. A’s write fails loudly instead of silently corrupting data. The lock service doesn’t need to know anything went wrong. The enforcement moves to the thing actually being protected.

This shifts the burden in an important way. The lock coordinator only needs to guarantee the token strictly increases. The resource, not the lock, becomes the final arbiter of which write is legitimate. That’s a much easier property to guarantee than trying to make the lock service somehow immune to network delays and slow clients, which isn’t achievable in an asynchronous network anyway.

Where This Shows Up in Practice

Storage systems that support conditional writes, like version checks or generation numbers on cloud object stores, are effectively using fencing tokens even if nobody calls them that. Distributed databases that use epoch numbers or term numbers on leader changes are applying the same idea to leader election: a new leader’s term number fences off any writes from a leader that doesn’t know it’s been replaced. Even primary-backup replication schemes often bake in a monotonically increasing generation number for exactly this reason.

The pattern generalizes beyond locks entirely. Anywhere you have “exactly one thing should be in charge at a time,” you have the same underlying risk: the thing in charge might not know it’s been replaced. A fencing token, or something equivalent to it, is the mechanism that makes that risk survivable instead of catastrophic.

The Lesson Underneath

The real takeaway isn’t “use fencing tokens for locks.” It’s that mutual exclusion guaranteed by a coordinator is only as strong as the enforcement at the point of action. If you can’t stop a stale actor from acting, you have to make sure its actions are rejected downstream. That’s a more modest, and more honest, guarantee than pretending the lock itself can never be wrong, and it’s the reason fencing tokens show up quietly in so many systems that need to survive real-world timing failures.