Support for liveness probe

[vinvli]

When deploying a rabbitmq cluster using the operator I see that a readiness probe is configured by default (amqp port check) but no liveness probe. Is that intentional? Because I notices an alivenesscheck is there in the go code? Is there a way to enable it?

[mkuratczyk]

We are definitely not against a liveness probe but the last time we discussed this we could not come up with a good one. Do you have any suggestions for what that probe should be? Erlang already restarts all the processes internally (within the running Erlang VM), so most (Erlang) process crashes are already handled. If Erlang itself crashes - that's handled by Kubernetes without a liveness probe. So what are the valid circumstances to kill Erlang VM when it's running?

[vinvli]

The issue with those liveness checks is that it eats up a considerable amount of CPU and is really noticeable on a small k8s cluster. So that's why I asked the question here.

[mkuratczyk]

Yes, using the CLI probe is just wrong, but that's an implementation detail - I see that currently an HTTP request is used which should be better (less overhead). But that doesn't answer the "why" - have you actually seen a scenario where this probe resolved an issue? What is the scenario in which the Erlang VM is running but we decide to kill it?

[vinvli]

No not really

[vinvli]

We just enabled it because it seemed like a good idea.

[mkuratczyk]

Well, that's why we currently don't define a livenessProbe. :) I'm not saying we never will but it needs to be carefully considered. There are situations in which killing Erlang could actually make a problem worse, or cause a problem when there wasn't any.

[polaris-alioth]

Sorry! How to check amqp port in readiness probe currently? Where i can view the readiness probe source code?

[mkuratczyk]

The easiest would be to deploy a cluster using the Operator and check the StatefulSet definition, but if you want the source code, it's here:

cluster-operator/internal/resource/statefulset.go

Lines 594 to 612 in e4e4271

	// Why using a tcp readiness probe instead of running `rabbitmq-diagnostics check_port_connectivity`?
	// Using rabbitmq-diagnostics command as the probe could cause context deadline exceeded errors
	// Pods could be stuck at terminating at deletion as a result of that
	// More details see issue: https://github.com/rabbitmq/cluster-operator/issues/409
	ReadinessProbe: &corev1.Probe{
	Handler: corev1.Handler{
	TCPSocket: &corev1.TCPSocketAction{
	Port: intstr.IntOrString{
	Type: intstr.String,
	StrVal: readinessProbePort,
	},
	},
	},
	InitialDelaySeconds: 10,
	TimeoutSeconds: 5,
	PeriodSeconds: 10,
	SuccessThreshold: 1,
	FailureThreshold: 3,
	},

[ccmcbeck]

If Erlang itself crashes - that's handled by Kubernetes without a liveness probe.

Can you explain a bit more? Is there some type of process monitor in the container that handles this? Today we were surprised to see (for the first time) that a rabbit-server-0 was unresponsive. Fortunately it wasn't PROD. There were no log errors that could shed any light. We had to delete the pod and let K8s recreate. We ended up here looking for info on liveness. Tks

[ccmcbeck]

For more context, here is the containerStatus

  "containerStatuses": [
    {
      "containerID": "docker://9f431eb251fa8f44df10d7d5a7599dbcd1e85efd027c2b4e17086f0e7736727c",
      "image": "bitnami/rabbitmq:3.11.16-debian-11-r3",
      "imageID": "docker-pullable://bitnami/rabbitmq@sha256:13582bf01e90508103c9a5e63643f519627119c96cfe62dd8428072ddbd37327",
      "lastState": {},
      "name": "rabbitmq",
      "ready": false,
      "restartCount": 0,
      "started": true,
      "state": {
        "running": {
          "startedAt": "2023-06-14T21:32:31Z"
        }
      }
    }
  ],

It is started but not ready and has been in that state for 2 weeks.

[ccmcbeck]

kubectl describe shows this

Events:
  Type     Reason     Age                        From     Message
  ----     ------     ----                       ----     -------
  Warning  Unhealthy  3m12s (x37338 over 3d19h)  kubelet  Readiness probe failed: dial tcp 10.4.31.214:5672: connect: connection refused

[ccmcbeck]

We eventually found the logs in the rabbit-server-0 container but it didn't shed any light as to why it stopped everything. We also found the kubectl get events (there were 37338 of them) so I think that may be the best way to monitor health for now. We created a Datadog monitor.

[Zerpet]

I'd recommend to start a new discussion with you issue, instead of comenting on an answered discussion from 2 years ago :-)