Even the fastest AI clusters can be held back by their release pipelines
A firmware rollout I was running had been moving zone by zone for six days when it stopped itself. The page always comes as a silent vibration first, but at 1:14 AM it feels like a fire alarm. In one zone, the share of hosts that rebooted uncleanly during the update had crossed the safety threshold, so the automation froze the rollout and flagged it for a person.
The first assumption in a moment like that is that we broke something. We hadn't. The hosts that failed had been stuck for hours before the update reached them, on a hardware hang we already knew about. The reboot in the update just made it visible. We changed the check to leave out hosts that were already unhealthy before the update, resumed, and finished the rollout 9 days later.
That incident taught me something I still apply: A health signal has to...
Copyright of this story solely belongs to venturebeat.com. To see the full text click HERE