Time, clocks and ordering

Every service has a clock, every log line has a timestamp, and every engineer assumes that two timestamps can be compared to say which event happened first. Inside one process that is nearly true. Across two machines it is false in ways that produce the bugs this lesson is about: the write that "happened before" the read it does not contain, the lock that expired on one clock and not the other, the sort by timestamp that puts the reply before the message. The fix is not a better clock; it is knowing which questions a clock can answer.

#Wall clocks drift, and are corrected by jumping

A machine's wall clock (System.currentTimeMillis(), Instant.now()) is a quartz oscillator that drifts — tens of parts per million, which is seconds per day — corrected by NTP, which slews it gently when the error is small and steps it when the error is large. A step is a jump: the clock reads 10:00:05, then 10:00:02. Between machines, NTP keeps clocks within milliseconds on a good network and tens or hundreds of milliseconds on a bad one, a virtual machine that was paused, or a container whose host was under load; and an NTP outage lets them drift apart unbounded.

So a comparison of timestamps from two machines carries an error bar you do not know. if (event.timestamp > lastSeen) apply(event) drops events from a machine whose clock is behind; a token that expires at now + 30s on the issuer is already expired on a validator whose clock is 31 seconds ahead; and "the latest write wins" by wall-clock timestamp — which is what Cassandra does by default and what many "last writer wins" caches do — lets a machine with a fast clock overwrite newer data from a machine with a slow one, silently and forever. Wall time is for humans and logs, and even in logs the correlation id from the observability course is what orders the lines, not the timestamp.

#Monotonic time: for durations, on one machine

System.nanoTime() reads a different clock: one that only moves forward, at a steady rate, unaffected by NTP steps — and whose value means nothing except as a difference from another nanoTime() on the same JVM. It is the clock for every duration: a timeout, a retry backoff, a latency measurement, a cache TTL. Instant.now() for those is a bug that surfaces once a year, when a clock step turns a 30-second timeout into a negative one that never fires, or into a 31-minute one; the JDK's own ScheduledExecutorService, Thread.sleep and CompletableFuture.orTimeout use the monotonic clock for exactly this reason, and the microservices course's retry budgets should too.

The rule is two clocks for two jobs: wall time to say when something happened for a person, monotonic time to say how long something took for a program. Neither says which of two events on two machines came first.

#Ordering without a clock: happens-before

Lamport's observation is that the order we actually need is causal: event A happened before event B if A could have influenced B — the same process did A then B, or A was the sending of a message and B its receipt, or there is a chain of those. Two events with no such chain are concurrent, and asking which came first is asking a question with no answer; any system that gives one (by wall clock) is guessing.

The concurrency course's Java Memory Model is this idea inside one JVM: synchronized and volatile establish happens-before between threads, and without them two threads' writes have no order. Across machines the same is true with messages instead of monitors, and the clocks below are the tools that make happens-before computable.

#Lamport clocks: a counter that respects causality

Each process keeps a counter. On every local event, increment it. On every send, increment and attach the value. On every receive, set the counter to max(local, received) + 1. The result: if A happened before B, then L(A) < L(B), always. So sorting events by Lamport timestamp (with a process id to break ties) gives a total order that never contradicts causality — the order a log, an audit trail or a replicated state machine needs, from a number that costs nothing.

What it does not give: the converse. L(A) < L(B) does not mean A caused B; two concurrent events get some order, arbitrarily. That is fine for "put these in a consistent sequence" and not fine for "detect that these two writes conflicted", which is the next clock's job.

#Vector clocks: detecting concurrency

A vector clock is a counter per process, carried as a map: {A: 3, B: 1, C: 7}. Each process increments its own entry on an event and a send; on receive, take the element-wise maximum and increment your own. Now comparison is three-way: V(A) < V(B) (every entry less or equal, at least one less) means A happened before B; the reverse means B before A; and neither means they are concurrent — two writes that neither knew about the other, which is a conflict the application must resolve rather than a winner a clock can pick. That is what Riak and the original Dynamo used to return siblings instead of silently dropping a write, and what a CRDT uses to merge deterministically.

The cost is the vector's size: one entry per process that has ever written, which for a system with many short-lived clients grows without bound and needs pruning; version vectors (per replica rather than per client) and dotted version vectors are the practical forms. For most services the pragmatic middle is a single version number per object with optimistic concurrency — the JPA course's @Version, the MongoDB lesson's expected-version check, the HTTP ETag with If-Match — which detects the conflict between two writers to one row without ordering anything else.

#Where this lands in a service

• Ordering events: a per-partition sequence from Kafka (the offset) or a per-aggregate version from the outbox, never the producer's wall-clock timestamp. Two events from two producers about different aggregates have no order and should not need one.
• Expiry and leases: a lease issued for 30 seconds is checked against the issuer's clock, or with a margin large enough to cover the drift you have measured; the failure-detection lesson makes this concrete with fencing tokens.
• Timeouts and retries: nanoTime, always.
• Idempotency and conflicts: a version or a vector on the object, and a merge rule written down, not "the later timestamp wins".
• Logs and traces: the correlation id orders them; the timestamp decorates them. A trace whose child span starts before its parent is clock skew, not time travel.

The one place wall clocks are made trustworthy at scale is by bounding their error explicitly: Google's TrueTime returns an interval [earliest, latest] from GPS and atomic clocks and Spanner waits out the uncertainty before committing, which is the engineering behind "external consistency" and the reason it needs hardware most systems do not have.