Ask any two computers what time it is and you’ll usually get two different answers. Not wildly different, but enough of a gap that a lot of distributed systems logic breaks if you assume otherwise. This turns out to be one of the more persistently annoying problems in distributed systems design: you need to know the order in which things happened across multiple machines, but the tool everyone reaches for first, the system clock, is not built for that job.

The Problem With Wall Clocks

Every server has a physical clock, and those clocks drift. Even with network time protocol (NTP) keeping them roughly in sync, “roughly” can mean tens of milliseconds of skew, sometimes more under load or network issues. That sounds small until you realize a single database transaction can complete in microseconds. If node A timestamps a write at 10:00:00.001 and node B timestamps a dependent write at 10:00:00.000, a naive system might conclude B happened first, even though A’s write is what triggered B in the first place.

This matters constantly in distributed databases, message queues, and replicated systems. If you’re resolving conflicting writes, deciding which version of a record is “latest,” or just trying to debug a sequence of events across services, you need a notion of ordering that doesn’t depend on clocks agreeing perfectly.

Logical Clocks: Order Without Time

The earlier fix, dating back to Leslie Lamport’s foundational work in the late 1970s, was to throw out physical time entirely and use logical clocks instead. A Lamport clock is just a counter each process increments on every event, and passes along with every message it sends. When a process receives a message, it bumps its counter to be at least one higher than the sender’s. This guarantees that if event A causally influenced event B, A’s logical timestamp is smaller than B’s.

The catch is that Lamport clocks only give you a partial answer. Two unrelated events can get assigned timestamps that look ordered even though neither one actually influenced the other. Vector clocks fix that by having each process track a whole vector of counters, one per process in the system, so you can definitively tell whether two events are causally related or truly concurrent. The tradeoff is overhead: the vector grows with the number of processes, which gets unwieldy in large or dynamic clusters.

Hybrid Logical Clocks: Getting the Best of Both

This is where hybrid logical clocks (HLCs) come in, and why they’ve become popular in modern distributed databases. An HLC combines a physical timestamp with a logical counter. Under normal conditions, it behaves like a regular wall clock, timestamps roughly track real time, which is genuinely useful for humans reading logs or reasoning about “when” something happened. But whenever causality requires it, the logical component ticks forward to preserve ordering guarantees, even if the physical clocks briefly disagree.

The practical benefit is that you get timestamps that are both causally consistent and close enough to real time to be useful for things like TTL expiration, human-readable audit trails, or bounding how stale a read might be. Systems like CockroachDB and MongoDB use HLC-style approaches for exactly this reason: they need ordering guarantees for correctness, but they also don’t want every timestamp to be logical gibberish disconnected from wall-clock reality.

Why This Still Matters

None of this is a solved problem in the sense of being invisible to engineers. Clock skew, NTP hiccups, and leap seconds still cause real incidents. Systems that rely on tightly bounded clock uncertainty, like Google Spanner’s TrueTime, invest heavily in specialized hardware (GPS and atomic clock references) specifically to shrink that uncertainty window and make stronger consistency guarantees possible.

The broader lesson is that “what happened first” is a much harder question in a distributed system than it is on a single machine, and the tools for answering it, Lamport clocks, vector clocks, hybrid logical clocks, are really different tradeoffs between overhead, precision, and human readability. Anyone building or debugging distributed systems eventually runs into a bug that only makes sense once you stop trusting the clock on the wall and start thinking in terms of causality instead.