Debugging connection count to RabbitMQ from Kubernetes
This is going to be a short one. Have you ever needed to connect which consumer of RabbitMQ is which pod in Kubernetes? We recently had an issue that sparked our interest. The connection count was rising in thousands, and we had limited possibilities to see which connection belonged to which pod. The connections tab in RabbitMQ administration does not really say anything constructive:
You can always get the list by asking the API, but even that could be a problem for the RabbitMQ cluster, which is under a heavy load. My suggestion is to save the list once you get it. In case you are debugging a steadily rising connection count, the list won’t change from one moment to the next:
The same should be done for Kubernetes pods. As we were debugging connection counts in thousands, we suppose there are thousands of pods. Saving the list of pods for later can also be helpful:
And now we can connect the pods and the connections. Making two for loops and comparing those two lists makes Bash run really slowly. It’s much faster to awk the second list for each pod:
This will reasonably quickly get you a list of pods and their connections to the RabbitMQ cluster. I have this script ready in case of emergencies. Needless to say, having a large number of connections does not usually pose any problem. It’s important to understand that the connection count is represented by file descriptors in the hosting node. If your node configuration does not enable large amounts of opened file descriptors, you might have to check that.
The whole script:
