Why "What Happened First?" Is One of the Hardest Questions in Large-Scale Systems

Imagine you and your friend are in different cities. You both look at your watches and agree to clap at exactly 1:00 PM. You clap, but did you both clap at the exact same instant? Probably not. Your watch might be a few seconds faster than your friend's, or maybe it runs a tiny bit slower. Computers in a large system face the same problem, but on a massive scale. A system like FAANG’s runs on thousands of computers spread across the globe. Each computer has its own internal clock, just like you have your own watch. These clocks are never perfectly in sync. This creates a big problem: if two events happen on two different computers, it's hard to say with 100% certainty which one happened first just by looking at their timestamps. The obvious solution seems to be to just synchronize all the computer clocks. We have protocols like the Network Time Protocol (NTP) that try to do this by connecting to ultra-precise atomic clocks. However, even with NTP, achieving perfect synchronization is i

Imagine you and your friend are in different cities. You both look at your watches and agree to clap at exactly 1:00 PM. You clap, but did you both clap at the exact same instant? Probably not. Your watch might be a few seconds faster than your friend's, or maybe it runs a tiny bit slower. Computers in a large system face the same problem, but on a massive scale. A system like FAANG’s runs on thousands of computers spread across the globe. Each computer has its own internal clock, just like you have your own watch. These clocks are never perfectly in sync. This creates a big problem: if two ev