Very very confuse hint: "Failed to fetch secret from all regions" --- even secrets can be fetch.

[johnzheng1975]

Describe the bug
I am trying csi on eks, fail the secrets from secret manager.
ALL is right, except I wrote one key wrong.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: infra-demo-secrets-aws
  namespace: myns
spec:
  parameters:
    objects: |
      - objectName: arn:aws:secretsmanager:us-west-2:xxxxx
        objectAlias: dev-ai-infra
        jmesPath:
        - path: DB_AI_HIVE_URL  **# should be db_ai_hive_url**
          objectAlias: DB_AI_HIVE_URL

The error message showed when I describe pod

Warning  FailedMount  3m39s (x710 over 23h)  kubelet  MountVolume.SetUp failed for volume "secrets-store-test" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod xxx-dev/sklearn-harbor-container2-predictor-67b49f889d-rvqmb, err: rpc error: code = Unknown desc = Failed to fetch secret from all regions: arn:aws:secretsmanager:us-west-2:xxx:secret:dev-ai-infra-pmUT8Z

It make me thinking whether I give rights to this secrets or not.
Since cross cluster rights assign is very complex, I cost almost one day to debug -- whether I give enough rights to role, service account...., although root reason is just one secret key name is wrong. -- secrets can be fetch/list/view!

So, this error message is very very confuse, it should say, secrets can be get, but key DB_AI_HIVE_URL does not exists!

To Reproduce

Steps to reproduce the behavior:

Do you also notice this bug when using a different secrets store provider (Vault/Azure/GCP...)? Yes/No

If yes, the issue is likely with the k8s Secrets Store CSI driver, not the AWS provider. Open an issue in that repo.

Expected behavior

Environment:
OS, Go version, etc.

Additional context
Add any other context about the problem here.

I really think this should be improve, since configure rights is complex here!
You should provide useful info to user. Thanks.

[ductnn]

You can try this: #176 (comment)
It's work for me

[johnzheng1975]

I know how to resolve this, as I mentioned, I found root reason and resolved it.
What I am asking this: improve the error message.

[purusang]

I was having the same issue as @johnzheng1975 , for me it was a lot misleading as I was doing cluster migration so I could have so many reasons to doubt my own configurations. But the issue ended up to be as naive as missing a key in aws secret manager. The error message could have been more clear. They closed a similar issue 2 yrs back and still people are getting misguided by the error logs. Please work on this issue.