Day Two operations

[colemanserious]

The documentation for the operator indicates it as supporting "day two" operations. The setup of Prometheus definitely counts in that set. Are there other features?

Examples that I've been asked about but don't see evidence of:

• periodic rebalance of queues
• approaches for handling network partitions

For the two above, do folks have recommendations for how they've handled? Note that in our prod environment, only a few elevated privilege folks have access to kubectl exec, so I'm seeking ideas for jobs/tools/scripts that we can lay in place for these sorts of scenarios.

[mkuratczyk]

Periodic rebalance of queues

Can you provide a scenario where the distribution becomes unequal and rebalancing would be useful? The operator already rebalances queues after a rolling restart. Combined with queue_leader_locator = balanced, the cluster should never be significantly off balance in the first place.

Also, keep in mind that rebalancing is purely based on the number of queues/leaders right now. Given a single busy queue can consume much more resources than many idle queues, rebalancing just to make sure the numbers are roughly even will often not equalize resource conumption anyway.

Network partition handling

We pretty much consider the partition handling strategies a lost cause. There are a few, all with pros and cons. Rather than implementing automated resolutions to these strategies, we spent the last few years building https://github.com/rabbitmq/khepri/ to replace Mnesia, the currently used metadata store. With Khepri, all those partition handling strategy settings will go away and RAFT semantics will always be used.

[colemanserious]

Hmmm - we don't have queue_leader_locator = balanced (though I see that's apparently it's a default setting), but instead see the operator default of queue_master_locator = min-masters in 10-operatorDefaults.conf. From here, I believe master_locator is used for classic queue replication, which we are still making use of, and queue_leader_locator is used for streaming queues, which we are not making use of.

It sounds as a potential resolution for us, since we're not using stream queues, is to hold on the master_locator setting, but consider executing a rolling restart against the cluster when/if we recognize issues with queues or we think the cluster is evidencing disk or network issues. (Side note: we're working in an environment where we're still tuning our use of istiod, which has caused us more issues than we anticipate otherwise having longer-term. But that's being separately tuned and worked.)

Thanks for the efforts on khepri. It doesn't look like it's yet far enough along for us to be able to make use of it, but appreciate being able to track it forward.

[mkuratczyk]

queue_leader_locator affects quorum queues and streams, while queue_master_locator affects mirrored queues.

Khepri is not meant for production yet indeed. It will be with 4.0 later this year.

I'm not sure if that's what you meant, but a rolling restart of a cluster just to rebalance the queues would be an overkill. You can just execute rabbitmq-queues rebalance when you feel like you need that.