Designing High-Availability Identity Systems Processing Billions of API Calls
It's Tuesday at 3 a.m., and the engineer on call receives a page: there's an issue with the entire service authenticating. After an hour and a half, it turns out to be an incorrect time-to-live (TTL) for the cache, which allowed stale data about token information to "break" all the validation checks. There were no warnings. No circuit breaker tripped. It just didn't scale. Identity is a "fragile" area of software systems. It is the conduit for all user authentication, token renewals, and authorization. With billions of calls per day, a 0.01% error rate means tens of thousands of authentication failures. The design of systems to handle this without performance or correctness penalties is a design trade-off which must be weighed in three aspects: observability, latency engineering, and failure isolation. Observability is needed to do latency engineering, and latency engineering is needed to do failure isolation.
Observability: You Cannot Fix...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
