How We Stabilized a Bazel CI Runner by Fixing Resource Limits and Container Cleanup

https://hackernoon.imgix.net/images/QEQBwoE0mIVieVY9Bb1uvEcDMVd2-m083mgp.jpeg

It all started as a routing investigation.

The monorepo (Go, nodejs, C++ with shared proto files) pipeline had been running slowly for some time, and then it started to fail. Bazel builds were being OOM-killed during working hours, the retries lengthened the queue, the longer queue reduced the frequency of pushes, and the CI issues quietly escalated into engineering problems. Upon finally logging into the runner host, I found that the machine, with 6 cores and 12 threads, had a load average of 32.7. Swap space was being used. There was nothing wrong with the box in the conventional sense. It was working exactly as instructed, but the instructions themselves were bizarre.

The cause was a comment in the .bazelrc file describing a different machine. The CI configuration block briefly stated that the runner could run a maximum of two Bazel containers simultaneously, and therefore each build could consume five...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE