That Time a Rolling Deployment Almost Corrupted Our Data
During what should have been a routine deployment, our pipeline briefly ran two copies of itself at the same time. Nothing failed. No alerts fired. For that window, though, the system was producing output that could not be trusted.
Some context. We had recently migrated a critical data pipeline off a managed Python ETL service and onto long-running Rust containers on a standard orchestration platform. The performance improvement was significant: a run that took 160 seconds in Python finished in 20 seconds in Rust, on a pipeline that processes millions of records per execution and fires every two and a half minutes. But the migration story that stayed with me was not about performance. It was about a bug we almost did not find.
The Setup
The pipeline generates periodic file datasets consumed by a downstream control system. Every two and a half minutes it produces a batch of files...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
