Updated QPS and Burst capacity for k8s client

[abheda-crest]

The PR focuses on updating QPS and Burst capacity for k8s client to handle a higher frequency of SA token retrieval. These configurations are now set via environment variables in deployment config, providing flexibility for adjusting values as per requirement. Additionally, the load test has been updated to reflect these changes.

Related to: #198 & #383

Example (deploy/provider-gcp-plugin.yaml):

          env:
            - name: TARGET_DIR
              value: "/etc/kubernetes/secrets-store-csi-providers"
            - name: K8S_CLIENT_QPS
              value: "20.0"
            - name: K8S_CLIENT_BURST
              value: "50"

Tested in the GKE environment across various configurations demonstrated successful scaling under moderate loads.
For ~100 secrets, the GKE environment remained stable across all tested pod and node configurations, supporting ~300 pods without issues.

[dargudear-google]

should we take this from the deployment config ?
same for other

[abheda-crest]

Accepting QPS and Burst values from the deployment configuration seems like a good approach, as it offers more flexibility for updates when necessary. There are two options I’m considering for retrieving these values from the config:

• Passing QPS and Burst as environment variables in the DaemonSet at deploy/provider-gcp-plugin.yaml.
• Specifying QPS and Burst as parameters within the SecretProviderClass for the GCP provider.

Please let me know which approach you think is preferable.

[dargudear-google]

deploy/provider-gcp-plugin.yaml seems to be a good option. Also I think default behaviour should be the existing values (default values).
@amitmodak WDYT?

[abheda-crest]

Updated to configure QPS and Burst from values provided in environment variable in deployment config.