## Agenda 1. Who framed RabbitMQ? 1. Troubleshoot in Wonderland 1. Under the hood: oslo.messaging 1. Follow the *right* rabbit 1. Lessons learned from this journey

# 1. Who framed RabbitMQ?

## Cluster unresponsive RabbitMQ server and/or cluster is unresponsive - stuck rabbitmqctl / api - a lot of error in rabbitmq logs - no more data from prom exporter -- ## Missed heartbeats A lot of connection closed because of missed heartbeats - not something new - seems accepted by the community -- ## oslo.messaging RPC timeout A lot of oslo.messaging RPC timeout in all openstack component logs ```bash MessagingTimeout: Timeout while waiting on RPC response... ``` -- ## Network partition RabbitMQ cluster partition without any visible reason -- ## Our (dirty) solution Reset cluster -- ## Bad effects - events loss ➜ inconsistencies on openstack - loss 1 node ➜ non HA queue are lost (*transient* queues) -- ## You said large-scale? Most of our region are bigger than 1k computes nodes Some regions have more than 2k computes nodes. ➜ We know that at large scale, it's not easy... **But we need to do better**

# 2. Troubleshoot

## Improve our observability - Prometheus exporter + grafana dashboards - oslo.metrics - smokeping between cluster nodes Note: - screenshot of tuned grafana dashboard + oslo.metrics - talk about patch we introduced in oslo.metrics -- ## Take a look at logs *that sounds obvious* Add some logs with: ```shell # Publisher 2024-05-20 17:18:37.824 235838 INFO oslo_messaging._drivers.amqpdriver [-] [unique_id:f68f648121954a30897b836098675fa6] Calling RPC method report_state on exchange 'neutron' topic 'q-reports-plugin' - Expecting reply to msg 116cdbdf52334a00979224c4e99a706a in queue reply_host299939.9023.dev.cloud.ovh.net:neutron-ovh-vrack-agent:1 # Subscriber 2024-05-20 17:18:37.832 4581 INFO oslo.messaging._drivers.impl_rabbit [-] [unique_id:f68f648121954a30897b836098675fa6 req:req-086f6872-15d6-40f5-9262-902331f3c5ed] Acknowledge message from queue q-reports-plugin - Must reply to msg_id 116cdbdf52334a00979224c4e99a706a in queue reply_host299939.9023.dev.cloud.ovh.net:neutron-ovh-vrack-agent:1 # Publisher receive ack 2024-05-20 17:18:37.864 235838 INFO oslo.messaging._drivers.impl_rabbit [-] [unique_id:276845328c72417d8baef0e9988cbe02] Acknowledge message from queue reply_host299939.9023.dev.cloud.ovh.net:neutron-ovh-vrack-agent:1 - Received reply to msg 116cdbdf52334a00979224c4e99a706a ... ``` -- ## Talk with upstream community *that sounds also obvious* -- ## Improve our tooling - rabbitmqctl - rabbitmq-queues - rabbitmq-diagnostics - etc. Note: - rabbitmq-diagnostic observer is useful to know which erlang function is consuming cpu - a tons of events in a rabbit cluster is not normal -- ## What we learned? Neutron is the bad guy * From oslo.metrics: most of messages in neutron are sent in fanouts * 1 message published ➜ replicated to N queues * neutron-server flood resources update to agents (agent notifier api, resource cache) * Each compute is having at least 4 neutron agents (bgp, vrack, l3, metadata, [dhcp]) * 1 OVS agent = ~14 tcp connections on rabbit * 1 OVS agent = +30 queues * Most of them are *transient* queues -- ## What we learned? Nova is not as bad as neutron * Only one agent (nova-compute) * Less queues / TCP connections * More messages (5x) -- ## What we learned? RabbitMQ perftest is a good tool to reproduce workload - rabbitmq does not like at all handling tons of internal events (connection, channel, queue create/delete, etc..) - it's designed to handle high msg/s - it's a message broker, not a queue/connection broker! ```bash ❭ docker run -it --net=host --rm pivotalrabbitmq/perf-test:latest -H amqp://rabbit:xxx@10.69.212.68 \ --size 5000 --json-body \ --publishing-interval 5 \ --consumers 1000 \ --producers 3000 \ -f persistent \ -queue-pattern "ha_queue_%s" \ --queue-pattern-from 1 \ --queue-pattern-to 30000 ```

# 3. Under the hood: oslo.messaging

## RPC server - 1 RPC server declare and listen on 3 queues: - topic - topic.server - topic_fanout_[rand] - 1 RPCServer = 1 tcp connection -- ## Publisher - one rpc method (ex: report_state) - one oslo.messaging Target - one of those send methods: - call - reply sent to a dedicated reply queue (reply_[rand]) - cast with fanout=False - cast with fanout=True -- ## Publisher: oslo.message ? ```yaml method: report_state version: '1.2' args: agent_state: agent_state: agent_type: DHCP agent availability_zone: nova binary: neutron-dhcp-agent configurations: {dhcp_driver: neutron.agent.linux.dhcp.Dnsmasq, dhcp_lease_duration: 86400, log_agent_heartbeats: false, networks: 1, ports: 5, subnets: 1} host: snat306422.9023.dev.cloud.ovh.net topic: dhcp_agent uuid: c1505bd8-da04-49ab-86a4-42f37142a7cb time: '2024-05-20T12:06:06.504363' _context_XXX: ... _msg_id: c2fee6ad20a84cb38aec390089b43856 _reply_q: reply_a098743aef84321bdcaa15851145542 _timeout: null _unique_id: d4b6abcd263d4596b9a6f851145dd99e ```

# 4. Follow the *right* rabbit

## RabbitMQ upgrade 3.8 ➜ 3.12 ➜ 4.1 -- Better, but not enough  -- ## Avoid missed heartbeats A lot of connections are closed because of missed heartbeats * not something new * seems accepted by the community Let's fix it anyway * Switch apache mpm: worker ➜ event * fixed green threads (some fix were already done by RedHat) * fixed pyamqp (timeout not respected) -- Better, but not enough  -- ## Introducing quorum queues * Quorum are replacing classic HA (removed in rabbitmq 4) * oslo.messaging partial support * missing support for *transient* queues ```shell [oslo.messaging] rabbit_quorum_queue = True rabbit_transient_quorum_queue = True ``` -- ## Good results All queues are now HA We support a node loss! -- Better, but not enough  -- ## Bad results * more CPU * more network * more RAM Queue churn is killing servers/cluster * Rabbit unresponsive / slow * Cluster partition * Exhaustion on Erlang Atom (1M) -- ## Avoid queue churn Queue churn is due to randomness in *transient* queues * reply\_[rand] * \*\_fanout\_[rand] ```shell [oslo.messaging] use_queue_manager = True hostname = yourhostname processname = yourprocessname ``` -- ## Good results ```bash reply_4170059e-e2d2-4291-94ce-2c765437b3b0 ``` to ```bash reply_host1234:neutron-bgp-agent:1 ``` On service restart: **no queue churn** -- Better, but not enough  -- ## Introducing stream queues Still too many queues Most of them are Fanouts As RabbitMQ doc says: > Use cases for using streams: > > large fan-outs -- ## Stream queues you said ?  -- From thousands ``` xyz_fanout_[rand] ``` to **only one**, shared by consumers ``` xyz_fanout ``` Using: ```shell [oslo.messaging] rabbit_stream_fanout = True ``` Note: In rabbitmq, each consumer declare it's own queue. with stream, same queue can be shared with N consumers. -- ## Good results (GRA3)  -- Now we are good!  -- ## Going further * reduce nb queues created by neutron-ovs-agent * a lot of queues are unused (3 queues declared per RPC server) * reduce nb connections created by neutron-ovs-agent * each RPC server have a dedicated connection (~10 per agent) * reuse same connection for multiple RPC server (e.g. ressource cache)

# 5. Lessons learned

### Lessons learned - Stay up to date with RabbitMQ - help being more stable - Follow RabbitMQ implementation recomendations - We spent long time to tweak RabbitMQ, but most of our issues were related to messaging implementation - community is helpful (ml, discord)

# QUESTIONS