Statuses of CRD Resources Created is Misrepresented

[davido912]

We're looking into integrating RMQ cluster operator + topology operator. We noticed that when deploying resources and reconciling fails, their status isn't reflected correctly in the CRD. This leaves us with validating deployments only via users checking out logs which is sub-optimal.

Example: creating a federation CRD with a secret reference that does not exist, still results in a success status.

What is the correct way to tackle this really?

Thanks!

Example CRD info:

Name:         rabbitmq-test-cluster-shovel-name
Namespace:    default
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: rabbitmq-shovel
              meta.helm.sh/release-namespace: default
API Version:  rabbitmq.com/v1beta1
Kind:         Shovel
Metadata:
  Creation Timestamp:  2023-06-06T16:00:39Z
  Finalizers:
    deletion.finalizers.shovels.rabbitmq.com
  Generation:  1
  Managed Fields:
    API Version:  rabbitmq.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:labels:
          .:
          f:app.kubernetes.io/managed-by:
      f:spec:
        .:
        f:destQueue:
        f:name:
        f:rabbitmqClusterReference:
          .:
          f:name:
        f:srcQueue:
        f:uriSecret:
        f:vhost:
    Manager:      helm
    Operation:    Update
    Time:         2023-06-06T16:00:39Z
    API Version:  rabbitmq.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"deletion.finalizers.shovels.rabbitmq.com":
    Manager:      manager
    Operation:    Update
    Time:         2023-06-06T16:00:39Z
    API Version:  rabbitmq.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:observedGeneration:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2023-06-06T16:00:39Z
  Resource Version:  281988962
  UID:               53eef8a8-1215-4ea1-ae66-f7b454bb5f6e
Spec:
  Dest Queue:  dst-shovel
  Name:        shovel-name
  Rabbitmq Cluster Reference:
    Name:     rabbitmq-test-cluster
  Src Queue:  source-shovel
  Uri Secret:
    Name:  test-shovel
  Vhost:   /
Status:
  Conditions:
    Last Transition Time:  2023-06-06T16:00:39Z
    Reason:                SuccessfulCreateOrUpdate
    Status:                True
    Type:                  Ready
  Observed Generation:     1
Events:                    <none>

[ChunyiLyu]

Hi @davido912, could you tell us a bit more about how you encountered this issue, e.g. version of operators you've used, manifest of the federation CRD you created, logs from messaging topology operator at the time of creation.

I did a simple test with creating a federation that specifies a spec.uriSecret.name that does not exist. I saw an error in status condition and cannot reproduce that you've described:

Status:
  Conditions:
    Last Transition Time:  2023-06-09T08:27:38Z
    Message:               failed to parse federation uri secret; secret name: federation-uri, error: Secret "federation-uri" not found
    Reason:                FailedCreateOrUpdate
    Status:                False
    Type:                  Ready

[michaelklishin]

Thank you for this important detail. This is a good hypothesis for Team RabbitMQ to investigate.

[ChunyiLyu]

@davido912 In this case the operator will log errors (as you've reported) and will not set the CRD status.condition or event. I see an empty status.condition instead of a successfulCreateorUpdate condition being true, like so:

Spec:
  Ack Mode:  on-confirm
  Name:      origin
  Rabbitmq Cluster Reference:
    Connection Secret:
      Name:  secret-that-doesnt-exist
  Uri Secret:
    Name:  federation-uri
  Vhost:   downstream
Events:    <none>

On the other hand, I could just update the operator so it publishes event and status.condition in this case. Would that be useful for your use case?

[davido912]

Hey @ChunyiLyu. That'd be great so if something goes wrong our downstream users are aware of this and we have a way of reporting it rather than getting tickets opened. :)

[ChunyiLyu]

@davido912 PR created: rabbitmq/messaging-topology-operator#633
You can test out this change by using image: rabbitmqoperator/messaging-topology-operator-dev:chunyi-test

FYI we just cut a topology operator release this week, so the next release might take a while.

[davido912]

Hey @ChunyiLyu , didn't get to this yet but will get back asap today or tomorrow latest. Sorry!