## Who are we ? Julien Arnaud

## What do we do? 

## Agenda 1. Who framed RabbitMQ? 1. Troubleshoot in Wonderland 1. Under the hood: oslo.messaging 1. Follow the *right* rabbit 1. Lessons learned from this journey

# 1. Who framed RabbitMQ?

## Cluster unresponsive RabbitMQ server and/or cluster is unresponsive - stuck rabbitmqctl / api - a lot of error in rabbitmq logs - no more data from prom exporter -- ## Missed heartbeats A lot of connection closed because of missed heartbeats - not something new - seems accepted by the community -- ## oslo.messaging RPC timeout A lot of oslo.messaging RPC timeout in all openstack component logs ```bash MessagingTimeout: Timeout while waiting on RPC response... ``` -- ## Network partition RabbitMQ cluster partition without any visible reason -- ## Our (dirty) solution Reset cluster -- ## Bad effects - events loss ➜ inconsistencies on openstack - loss 1 node ➜ non HA queue are lost (*transient* queues) -- ## You said large-scale? Most of our region are bigger than 1k computes nodes Some regions have more than 2k computes nodes. ➜ We know that at large scale, it's not easy...

# 2. Troubleshoot

## Improve our observability - Prometheus exporter + grafana dashboards - oslo.metrics - smokeping between cluster nodes Note: - screenshot of tuned grafana dashboard + oslo.metrics - talk about patch we introduced in oslo.metrics -- ## Take a look at logs *that sounds obvious* -- ## Add some useful logs Add some logs with: - message ID - consumer ID - RPC called - etc. ```shell # Publisher 2024-05-20 17:18:37.824 235838 INFO oslo_messaging._drivers.amqpdriver [-] [unique_id:f68f648121954a30897b836098675fa6] Calling RPC method report_state on exchange 'neutron' topic 'q-reports-plugin' - Expecting reply to msg 116cdbdf52334a00979224c4e99a706a in queue reply_host299939.9023.dev.cloud.ovh.net:neutron-ovh-vrack-agent:1 # Subscriber 2024-05-20 17:18:37.832 4581 INFO oslo.messaging._drivers.impl_rabbit [-] [unique_id:f68f648121954a30897b836098675fa6 req:req-086f6872-15d6-40f5-9262-902331f3c5ed] Acknowledge message from queue q-reports-plugin - Must reply to msg_id 116cdbdf52334a00979224c4e99a706a in queue reply_host299939.9023.dev.cloud.ovh.net:neutron-ovh-vrack-agent:1 # Publisher receive ack 2024-05-20 17:18:37.864 235838 INFO oslo.messaging._drivers.impl_rabbit [-] [unique_id:276845328c72417d8baef0e9988cbe02] Acknowledge message from queue reply_host299939.9023.dev.cloud.ovh.net:neutron-ovh-vrack-agent:1 - Received reply to msg 116cdbdf52334a00979224c4e99a706a ... ``` -- ## Talk with upstream community *that sounds also obvious* -- ## Improve our tooling - rabbitmqctl - rabbitmq-queues - rabbitmq-diagnostics - etc. Note: - rabbitmq-diagnostic observer is useful to know which erlang function is consuming cpu - a tons of events in a rabbit cluster is not normal -- ## Reproduce production workload RabbitMQ perftest is a good tool for that ```bash ❭ docker run -it --net=host --rm pivotalrabbitmq/perf-test:latest -H amqp://rabbit:xxx@10.69.212.68 \ --size 5000 --json-body \ --publishing-interval 5 \ --consumers 1000 \ --producers 3000 \ -f persistent \ -queue-pattern "ha_queue_%s" \ --queue-pattern-from 1 \ --queue-pattern-to 30000 ``` -- ## What we learned? Neutron is the bad guy - From oslo.metrics: most of messages in neutron are sent in fanouts - 1 message published ➜ replicated to N queues - neutron-server flood resources update to agents (agent notifier api, resource cache) -- ## What we learned? Neutron is the bad guy * Each compute is having at least 4 neutron agents (bgp, vrack, l3, metadata, [dhcp]) * 1 agent = 14 tcp connections on rabbit * 1 agent = at least N queues * Most of them are *transient* queues -- ## What we learned? * Neutron create a lot of queues and connections! * Rolling update redeclare tons of queues * anormal CPU usage on rabbitmq servers -- ## What we learned? Nova is not as bad as neutron * Only one agent (nova-compute) * Less queues / TCP connections * More messages (5x) -- ## What we learned? perftest: - rabbitmq does not like at all handling tons of internal events (connection, channel, queue create/delete, etc..) - it's designed to handle high msg/s - it's a message broker, not a queue/connection broker! -- ## What we learned? Grafana: - bottleneck on file descriptor usage - a persistent queue use 1 fd stored on disk

# 3. Under the hood: oslo.messaging

## Publish/Subcribe mechanism - Subscriber: oslo_messaging.RPCServer - create and listen on *topic* queues - process rpc message - Publisher - Send oslo.message to target(s) *topic* queue -- ## RPC server - RPCServer declaration: - one oslo.messaging Target - a list of endpoint class with rpc methods ```python target = oslo_messaging.Target( topic=name, server=hostname, fanout=fanout ) endpoints = [RPCMethods()] return oslo_messaging.get_rpc_server( TRANSPORT, target, endpoints, ... ) ``` -- ## RPC server - 1 RPC server declare and listen on 3 queues: - topic - topic.server - topic_fanout_[rand] - 1 RPCServer = 1 tcp connection -- ## Publisher - one rpc method (ex: report_state) - one oslo.messaging Target - one of those send methods: - call - reply sent to a dedicated reply queue (reply_[rand]) - cast with fanout=False - cast with fanout=True -- ## Publisher: oslo.message ```yaml method: report_state version: '1.2' args: agent_state: agent_state: agent_type: DHCP agent availability_zone: nova binary: neutron-dhcp-agent configurations: {dhcp_driver: neutron.agent.linux.dhcp.Dnsmasq, dhcp_lease_duration: 86400, log_agent_heartbeats: false, networks: 1, ports: 5, subnets: 1} host: snat306422.9023.dev.cloud.ovh.net topic: dhcp_agent uuid: c1505bd8-da04-49ab-86a4-42f37142a7cb time: '2024-05-20T12:06:06.504363' _context_XXX: ... _msg_id: c2fee6ad20a84cb38aec390089b43856 _reply_q: reply_snat306422.9023.dev.cloud.ovh.net:neutron-dhcp-agent:1 _timeout: null _unique_id: d4b6abcd263d4596b9a6f851145dd99e ``` --  -- ## Policy and queues Recommended HA policy: ```bash `^(?!(amq\.)|(.*_fanout_)|(reply_)).*` ``` * No HA on *transient* queues: * reply\_[rand] * \*\_fanout\_[rand]

# 4. Follow the *right* rabbit

## RabbitMQ upgrade 3.8 ➜ 3.12 -- Better, but not enough  -- ## Avoid missed heartbeats A lot of connections are closed because of missed heartbeats * not something new * seems accepted by the community Let's fix it anyway -- ## Switch apache mpm worker ➜ event -- Better, but not enough  -- ## Avoid missed heartbeats on compute nodes * fixed green threads (some fix were already done by RedHat) * fixed pyamqp (timeout not respected) -- Better, but not enough  -- ## Introducing RPC pinger A new endpoint to help operators monitoring agents liveness -- ## Configuring RPC pinger ```shell [oslo.messaging] rpc_ping_enabled = True ``` \+ a tool that actually send call RPC ping on all agents every 5 minutes (ovh-rpc-pinger) -- ## Good results Very precise monitoring of which neutron / nova agent is dead or not -- ## Bad results Too many messages sent over RabbitMQ wires (x10) Not as reliable as expected: * each RPC server run in a separate thread -- Better, but not enough  -- ## Support a node loss * Recommended policy is not HA for all queues * node loss ➜ queues loss ➜ messages loss * Classic HA are deprecated -- ## Introducing quorum queues * Quorum are replacing classic HA * RabbitMQ community recommands quorum * oslo.messaging partial support * missing support for *transient* queues -- ## Configuring quorum queues ```shell [oslo.messaging] rabbit_quorum_queue = True rabbit_transient_quorum_queue = True ``` -- ## Good results All queues are now HA We support a node loss! -- Better, but not enough  -- ## Bad results * more CPU * more network * more RAM -- ## Very bad results Queue churn is killing servers/cluster * Rabbit unresponsive / slow * Cluster partition * Exhaustion on Erlang Atom (1M) -- ## Avoid queue churn Queue churn is due to randomness in *transient* queues * reply\_[rand] * \*\_fanout\_[rand] -- ## Introducing queue manager ```shell [oslo.messaging] use_queue_manager = True hostname = yourhostname processname = yourprocessname ``` -- ## Good results ```bash reply_4170059e-e2d2-4291-94ce-2c765437b3b0 ``` to ```bash reply_host1234:neutron-bgp-agent:1 ``` On service restart: **no queue churn** -- Better, but not enough  -- ## Too many queues issue Too many queues on RabbitMQ **Side effect**: * Queue leader re-election is expensive * Can kill the cluster -- ## Introducing stream queues As RabbitMQ doc says: > Use cases for using streams: > > large fan-outs -- ## Stream queues you said ?  -- From thousands ``` xyz_fanout_[rand] ``` to **only one**, shared by consumers ``` xyz_fanout ``` Using: ```shell [oslo.messaging] rabbit_stream_fanout = True ``` Note: In rabbitmq, each consumer declare it's own queue. with stream, same queue can be shared with N consumers. -- ## Good results (GRA3)  -- Better, but not enough  -- ## OVS agent: unused queues ? Neutron creates a lot of queues, while it seems they are not used! Can we delete them / avoid creating them ? -- ## OVS agent: Stop creating unused queues 1 RPCServer = 3 queues (topic, topic.server, topic_fanout) But most of the time, only one of those is used. - Most of the time, only *topic_fanout* is used ➜ Good news, it's now shared queues (stream) -- **How ?** At RPCServer creation, idea is to declare and listen only on needed queues. We are lucky: oslo.messaging Target have all informations needed ! Using: ```shell [oslo.messaging] rabbit_declare_all_consumers = False ``` -- ## Good results (GRA3)  -- ## Reduce OVS agents TCP connections Neutron declare a lot of distinct RPC Servers for same purpose: - 5 connections for resource cache (neutron-vo-Port-1.9, neutron-vo-SecurityGroup-1.6, ...) - 2 connections for trunk extension (neutron-vo-SubPort-1.0 neutron-vo-Trunk-1.1) - 3 connections for agent notifier (q-agent-notifier-tunnel-update, tunnel-delete, dvr-update) -- **How ?** At RPCServer creation, idea is to pass a list of topics instead just 1 On OVS agent conf: ``` [agent] rpc_multiplex_agent_notifier_topics = True rpc_multiplex_resource_cache_topics = True ``` -- ## Good results (GRA3)  -- Now we are good! 

# 5. Lessons learned

### Lessons learned - Stay up to date with RabbitMQ - help being more stable - Follow RabbitMQ implementation recomendations - We spent long time to tweak RabbitMQ, but most of our issues were related to messaging implementation - community is helpful (ml, discord)