How a RabbitMQ TLS mismatch burned one CPU core before AMQP negotiation

https://hackernoon.imgix.net/images/ATuRYnOn0QNfPpDbuAtt2kf63vE2-z2d39hb.png

Three .NET services on an AKS cluster each pinned their CPU limit while doing no useful work. They targeted a TLS-only RabbitMQ listener with a plaintext client. Their configuration omitted the AMQP TLS flag, and the older framework package did not wire that flag into the connection factory at all. The application never logged the failure, so the proof came from seven bytes on the wire.

The investigation worked only after I separated CPU behaviour, network reachability, protocol negotiation and application startup into different tests.

What the cluster looked like

Three backend services ran at roughly 100% of their CPU limit. The kernel throttled them for 100.0%, 97.6% and 95.5% of CFS periods against a 500m limit. Two comparable .NET backends on the same chart, node, and framework idled at 1m to 2m, and the kernel throttled them for 9.1% and 7.4% of periods.

The busy three had no inbound...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE

Read more