Messages lost when nodes restarts

[bdaoudtdc]

Describe the bug

We have a cluster with 3 nodes.
In the event of nodes restarts because of changing a config value in the YAML file , we see messages loss.
We use quorum queues.
We are using publish confirm & consumer ack.
We believe the messages are lost between RMQ & the consumer
If the consumer is down , we don't see loss of messages.

Expected behavior
We don't except any loss of messages

Version and environment information

• RabbitMQ: [3.8.12]
• RabbitMQ Cluster Operator: [ 1.1.7 ]
• Kubernetes: [e.g. 1.17.13]

[Zerpet]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[Zerpet]

Hi @bdaoudtdc 👋 Can you provide more details on how do you determine there's been message loss? How many messages are lost, on average?

Do you "force" a Pod to terminate when it stays for some time in "Terminanting" state?

Are you able to consistently reproduce this issue?

Any metrics, specially from Grafana dashboards, would be very helpful to determine what's going on, and the YAML manifest you use to deploy your cluster would be helpful too.

[bdaoudtdc]

Hello,
Each message is logged at the consumer and each message have a distinct ID, we count the number of distinct IDs.
On average we have 5-20 messages lost.
No we don't force any pod to terminate.
Yes I can reproduce the issue almost all times

We don't have grafana running.

Here is the YAML file

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rmq-uat-cluster
spec:
image: rabbitmq:3.8.12-management
replicas: 3
service:
type: NodePort
persistence:
storageClassName: rmq-data-uat-storage
storage: 10Gi
tls:
secretName: back-secret
disableNonTLSListeners: true
rabbitmq:
additionalPlugins:
- rabbitmq_auth_backend_ldap
additionalConfig: |
## ==============
## Security, Access Control
## ==============
## Select an authentication/authorisation backend to use
auth_backends.1 = internal
auth_backends.2 = ldap
## TLS configuration.
ssl_options.versions.1 = tlsv1.2
## ==============================
## Resource Limits & Flow Control
## ==============================
vm_memory_high_watermark_paging_ratio = 0.99
vm_memory_high_watermark.relative = 0.6
disk_free_limit.absolute = 1GB
## Clustering
### =====================
cluster_partition_handling = pause_minority
# =======================================
# Management section
# =======================================
## Supported TLS versions
management.ssl.versions.1 = tlsv1.2
## Logging to console
log.console.level = warning
# =======================================
# LDAP section
# =======================================
auth_ldap.servers.1 = dc1.example.com
auth_ldap.servers.2 = dc2.example.com
auth_ldap.use_starttls = true
auth_ldap.timeout = 15000
auth_ldap.idle_timeout = 300000
auth_ldap.log = false
auth_ldap.dn_lookup_attribute = sAMAccountName
auth_ldap.dn_lookup_base = dc=example,dc=com
auth_ldap.dn_lookup_bind.user_dn = cn=user1,cn=Users,dc=example,dc=com
auth_ldap.dn_lookup_bind.password = xxxxxxxx
advancedConfig: |
[
{rabbitmq_tracing,
[
{directory, "/var/log/rabbitmq/"},
{username, "tadmin"},
{password, "xxxxxxxxxxxx"}
]
},
{rabbitmq_auth_backend_ldap, [
{ssl_options, [{cacertfile,"/etc/rabbitmq/certs/mycert.crt"}]},

        {tag_queries,           [{administrator, {'or',[{in_group, "CN=admins,OU=SecurityGroups,DC=example,DC=com"},{in_group, "CN=admins1,OU=SecurityGroups,DC=example,DC=com"},{in_group, "CN=admins2,OU=SecurityGroups,DC=example,DC=com"} ]}},
                                {management, {'or',[{in_group, "CN=mgt1,OU=SecurityGroups,DC=example,DC=com"},{in_group, "CN=mgt2,OU=SecurityGroups,DC=example,DC=com"}]}}]}
        ]}
   ].

override:
service:
spec:
ports:
# - name: rmq-mgm-port #Assign NodePort
# protocol: TCP
# port: 15672
# nodePort: 30052
- name: rmq-mgm-tls-port #Assign NodePort
protocol: TCP
port: 15671
nodePort: 30051
# - name: rmq-amqp-port #Assign NodePort
# protocol: TCP
# port: 5672
# nodePort: 30042
- name: rmq-amqp-tls-port #Assign NodePort
protocol: TCP
port: 5671
nodePort: 30041
statefulSet:
spec:
template:
spec:
containers:
- name: rabbitmq
volumeMounts:
- mountPath: /etc/rabbitmq/certs
name: ldapcacert
volumes:
- name: ldapcacert
secret:
secretName: myldapcert

[mkuratczyk]

Can you tell me if this report comes from you or someone you work with https://groups.google.com/g/rabbitmq-users/c/bU9Ne0qgw3g/m/Dmit2o5yAwAJ? If not then we now have 2 reports of a similar issue and most likely the problem is unrelated to the Operator but potentially a bug in RabbitMQ itself.

Given you are able to reproduce it almost every time, it'd help tremendously if you could provide an executable test case - simple publisher and consumer apps that we can run to trigger the problem and see those missing IDs. Any chance you could provide that?

[bdaoudtdc]

Hello , yes, I checked the link , this is actually the same issue but reported by a member from a different team.

[kjnilsson]

@bdaoudtdc Can you test if terminating the consumer after the test returns the messages to the queue?

I can think of a scenario where this may happen. If that is the case then messages aren't lost but rather "stuck" (i.e. the queue has assigned the messages to the consumer but the delivery hasn't completed and may not if the consumer's prefetch is maxed out or if there are no further messages received by the queue).

[bdaoudtdc]

Actually messages are lost .
My test based on your suggestion , is that I sent the messages, i restarted the nodes one at a time , then I counted the messages at the consumer side, I saw that I lost messages , so I shutdown the consumer , but nothing happened , I still had the same lost messages.

Please advice what do you want me to do.
Thanks,

[kjnilsson]

Ah I see. logs would still be helpful but most likely this is an issue in the mule adapter or your application. Providing an application that we can use easily to reproduce is the best way forward. Ideally an application that directly uses one of our supported client libraries.

[shahzadlari]

We are using mule version 3.9 with amqp connector version 3.9.5 and Client library 5.8.0. Mule uses java client library which I think is supported client library. A very simple flow created in mule with above versions should be good to recreate the issue with rabbitMQ cluster.
I am pasting the connector setting and an empty flow consuming the message.. let me know if that is enough or need it as mule application.

<amqps:connector name="rabbitConnect" sslProtocol="TLS" ackMode="MULE_AUTO" activeDeclarationsOnly="false" virtualHost="${amqp.virtual.host}" numberOfChannels="20" host="${amqp.host}" port="${amqp.port}" username="${amqp.user}" password="${amqp.password}" doc:name="AMQP-0-9 Connector" prefetchCount="1">


</amqps:connector>

<flow name="some-name" doc:name="ServiceId-9" initialState="started"> 
    <amqps:inbound-endpoint queueName="${queue.name}" exchange-pattern="request-response" connector-ref="rabbitConnect"/>
    <logger message="#[payload]" level="INFO" doc:description="loggging message" doc:name="Logger"/>
    
	<!-- Message Processing -->
	<!-- Message Processing -->
	<!-- Message Processing -->
	
	<rollback-exception-strategy doc:name="Rollback Exception Strategy"> 
        <choice doc:name="acknowledge the message if amqp">
            <when expression="#[flowVars.'amqp.channel'!=null]">
                <logger message="Failed message"  level="INFO" doc:name="Logger"/>
                 <amqps:acknowledge-message />
            </when>
        </choice> 
    </rollback-exception-strategy>  
</flow>

[kjnilsson]

Are you using amqp transactions?

[shahzadlari]

No, shall we try with it? we did not use because at any point if message fails in processing we store the message in a database and acknowledge.

[bdaoudtdc]

Does anyone else have this problem .. if not .. may you please share which client libraries you are using.
Thanks,

[kjnilsson]

I’m away atm so can’t look at this issue but as we asked before if you can provide a simple runnable application we can use with reproduction steps we’ll be able to look into it further. Cheers Karl

On Sat, 4 Sep 2021 at 04:53, bdaoudtdc ***@***.***> wrote: Does anyone else have this problem .. if not .. may you please share which client libraries you are using. Thanks, — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#814 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAHFE4PTZ7KKN2I2WDSFDUAGC33ANCNFSM5CQULBZQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- *Karl Nilsson*