Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VC-36950: It is now possible to exclude labels and annotations #614

Merged
merged 6 commits into from
Nov 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions deploy/charts/venafi-kubernetes-agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,22 @@ Control Plane.
> ```yaml
> helm.sh/release.v1
> ```
#### **config.excludeAnnotationKeysRegex** ~ `array`
> Default value:
> ```yaml
> []
> ```

You can configure Venafi Kubernetes Agent to exclude some annotations or labels from being pushed to the Venafi Control Plane. All Kubernetes objects are affected. The objects are still pushed, but the specified annotations and labels are removed before being sent to the Venafi Control Plane.

Dots is the only character that needs to be escaped in the regex. Use either double quotes with escaped single quotes or unquoted strings for the regex to avoid YAML parsing issues with `\.`.

Example: excludeAnnotationKeysRegex: ['^kapp\.k14s\.io/original.*']
#### **config.excludeLabelKeysRegex** ~ `array`
> Default value:
> ```yaml
> []
> ```
#### **config.configmap.name** ~ `unknown`
> Default value:
> ```yaml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ data:
cluster_description: {{ .Values.config.clusterDescription | quote }}
server: {{ .Values.config.server | quote }}
period: {{ .Values.config.period | quote }}
exclude-annotation-keys-regex:
{{ .Values.config.excludeAnnotationKeysRegex | toYaml | nindent 6 }}
exclude-label-keys-regex:
{{ .Values.config.excludeLabelKeysRegex | toYaml | nindent 6 }}
venafi-cloud:
uploader_id: "no"
upload_path: "/v1/tlspk/upload/clusterdata"
Expand Down
17 changes: 17 additions & 0 deletions deploy/charts/venafi-kubernetes-agent/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,12 @@
"configmap": {
"$ref": "#/$defs/helm-values.config.configmap"
},
"excludeAnnotationKeysRegex": {
"$ref": "#/$defs/helm-values.config.excludeAnnotationKeysRegex"
},
"excludeLabelKeysRegex": {
"$ref": "#/$defs/helm-values.config.excludeLabelKeysRegex"
},
"ignoredSecretTypes": {
"$ref": "#/$defs/helm-values.config.ignoredSecretTypes"
},
Expand Down Expand Up @@ -206,6 +212,17 @@
},
"helm-values.config.configmap.key": {},
"helm-values.config.configmap.name": {},
"helm-values.config.excludeAnnotationKeysRegex": {
"default": [],
"description": "You can configure Venafi Kubernetes Agent to exclude some annotations or labels from being pushed to the Venafi Control Plane. All Kubernetes objects are affected. The objects are still pushed, but the specified annotations and labels are removed before being sent to the Venafi Control Plane.\n\nDots is the only character that needs to be escaped in the regex. Use either double quotes with escaped single quotes or unquoted strings for the regex to avoid YAML parsing issues with `\\.`.\n\nExample: excludeAnnotationKeysRegex: ['^kapp\\.k14s\\.io/original.*']",
"items": {},
"type": "array"
},
"helm-values.config.excludeLabelKeysRegex": {
"default": [],
"items": {},
"type": "array"
},
"helm-values.config.ignoredSecretTypes": {
"items": {
"$ref": "#/$defs/helm-values.config.ignoredSecretTypes[0]"
Expand Down
29 changes: 21 additions & 8 deletions deploy/charts/venafi-kubernetes-agent/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ podSecurityContext: {}
securityContext:
capabilities:
drop:
- ALL
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true

Expand Down Expand Up @@ -230,13 +230,26 @@ config:
# * https://kubernetes.io/docs/concepts/configuration/secret/#secret-types
# * https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/#list-of-supported-fields
ignoredSecretTypes:
- kubernetes.io/service-account-token
- kubernetes.io/dockercfg
- kubernetes.io/dockerconfigjson
- kubernetes.io/basic-auth
- kubernetes.io/ssh-auth
- bootstrap.kubernetes.io/token
- helm.sh/release.v1
- kubernetes.io/service-account-token
- kubernetes.io/dockercfg
- kubernetes.io/dockerconfigjson
- kubernetes.io/basic-auth
- kubernetes.io/ssh-auth
- bootstrap.kubernetes.io/token
- helm.sh/release.v1

# You can configure Venafi Kubernetes Agent to exclude some annotations or
# labels from being pushed to the Venafi Control Plane. All Kubernetes objects
# are affected. The objects are still pushed, but the specified annotations
# and labels are removed before being sent to the Venafi Control Plane.
#
# Dots is the only character that needs to be escaped in the regex. Use either
# double quotes with escaped single quotes or unquoted strings for the regex
# to avoid YAML parsing issues with `\.`.
#
# Example: excludeAnnotationKeysRegex: ['^kapp\.k14s\.io/original.*']
excludeAnnotationKeysRegex: []
excludeLabelKeysRegex: []

# Specify ConfigMap details to load config from an existing resource.
# This should be blank by default unless you have you own config.
Expand Down
32 changes: 31 additions & 1 deletion pkg/agent/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"io"
"net/url"
"os"
"regexp"
"time"

"github.com/go-logr/logr"
Expand Down Expand Up @@ -54,6 +55,12 @@ type Config struct {
InputPath string `yaml:"input-path"`
// For testing purposes.
OutputPath string `yaml:"output-path"`

// Skips annotation keys that match the given set of regular expressions.
// Example: ".*someprivateannotation.*".
ExcludeAnnotationKeysRegex []string `yaml:"exclude-annotation-keys-regex"`
// Skips label keys that match the given set of regular expressions.
ExcludeLabelKeysRegex []string `yaml:"exclude-label-keys-regex"`
}

type Endpoint struct {
Expand Down Expand Up @@ -339,7 +346,9 @@ type CombinedConfig struct {
VenConnNS string

// VenafiCloudKeypair and VenafiCloudVenafiConnection modes only.
DisableCompression bool
DisableCompression bool
ExcludeAnnotationKeysRegex []*regexp.Regexp
ExcludeLabelKeysRegex []*regexp.Regexp

// Only used for testing purposes.
OutputPath string
Expand Down Expand Up @@ -585,6 +594,27 @@ func ValidateAndCombineConfig(log logr.Logger, cfg Config, flags AgentCmdFlags)
res.DisableCompression = flags.DisableCompression
}

// Validation of the config fields exclude_annotation_keys_regex and
// exclude_label_keys_regex.
{
for i, regex := range cfg.ExcludeAnnotationKeysRegex {
r, err := regexp.Compile(regex)
if err != nil {
errs = multierror.Append(errs, fmt.Errorf("invalid exclude_annotation_keys_regex[%d]: %w", i, err))
continue
}
res.ExcludeAnnotationKeysRegex = append(res.ExcludeAnnotationKeysRegex, r)
}
for i, regex := range cfg.ExcludeLabelKeysRegex {
r, err := regexp.Compile(regex)
if err != nil {
errs = multierror.Append(errs, fmt.Errorf("invalid exclude_label_keys_regex[%d]: %w", i, err))
continue
}
res.ExcludeLabelKeysRegex = append(res.ExcludeLabelKeysRegex, r)
}
}

if errs != nil {
return CombinedConfig{}, nil, errs
}
Expand Down
7 changes: 7 additions & 0 deletions pkg/agent/run.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ import (
"github.com/jetstack/preflight/api"
"github.com/jetstack/preflight/pkg/client"
"github.com/jetstack/preflight/pkg/datagatherer"
"github.com/jetstack/preflight/pkg/datagatherer/k8s"
"github.com/jetstack/preflight/pkg/kubeconfig"
"github.com/jetstack/preflight/pkg/logs"
"github.com/jetstack/preflight/pkg/version"
Expand Down Expand Up @@ -176,6 +177,12 @@ func Run(cmd *cobra.Command, args []string) (returnErr error) {
return fmt.Errorf("failed to instantiate %q data gatherer %q: %v", kind, dgConfig.Name, err)
}

dynDg, isDynamicGatherer := newDg.(*k8s.DataGathererDynamic)
if isDynamicGatherer {
dynDg.ExcludeAnnotKeys = config.ExcludeAnnotationKeysRegex
dynDg.ExcludeLabelKeys = config.ExcludeLabelKeysRegex
}
Comment on lines +180 to +184
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is super hacky... I tried finding a cleaner way of "injecting" the ExcludeAnnotKeys and ExcludeLabelKeys, but gave up.


log.V(logs.Debug).Info("Starting DataGatherer", "name", dgConfig.Name)

// start the data gatherers and wait for the cache sync
Expand Down
87 changes: 85 additions & 2 deletions pkg/datagatherer/k8s/dynamic.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package k8s
import (
"context"
"fmt"
"regexp"
"strings"
"time"

Expand Down Expand Up @@ -260,6 +261,9 @@ type DataGathererDynamic struct {
// informer watches the events around the targeted resource and updates the cache
informer k8scache.SharedIndexInformer
registration k8scache.ResourceEventHandlerRegistration

ExcludeAnnotKeys []*regexp.Regexp
ExcludeLabelKeys []*regexp.Regexp
}

// Run starts the dynamic data gatherer's informers for resource collection.
Expand Down Expand Up @@ -338,7 +342,7 @@ func (g *DataGathererDynamic) Fetch() (interface{}, int, error) {
}

// Redact Secret data
err := redactList(items)
err := redactList(items, g.ExcludeAnnotKeys, g.ExcludeLabelKeys)
if err != nil {
return nil, -1, errors.WithStack(err)
}
Expand All @@ -349,7 +353,7 @@ func (g *DataGathererDynamic) Fetch() (interface{}, int, error) {
return list, len(items), nil
}

func redactList(list []*api.GatheredResource) error {
func redactList(list []*api.GatheredResource, excludeAnnotKeys, excludeLabelKeys []*regexp.Regexp) error {
for i := range list {
if item, ok := list[i].Resource.(*unstructured.Unstructured); ok {
// Determine the kind of items in case this is a generic 'mixed' list.
Expand All @@ -374,6 +378,10 @@ func redactList(list []*api.GatheredResource) error {

// remove managedFields from all resources
Redact(RedactFields, resource)

RemoveUnstructuredKeys(excludeAnnotKeys, resource, "metadata", "annotations")
RemoveUnstructuredKeys(excludeLabelKeys, resource, "metadata", "labels")

continue
}

Expand All @@ -386,6 +394,9 @@ func redactList(list []*api.GatheredResource) error {
item.GetObjectMeta().SetManagedFields(nil)
delete(item.GetObjectMeta().GetAnnotations(), "kubectl.kubernetes.io/last-applied-configuration")

RemoveTypedKeys(excludeAnnotKeys, item.GetObjectMeta().GetAnnotations())
RemoveTypedKeys(excludeLabelKeys, item.GetObjectMeta().GetLabels())

resource := item.(runtime.Object)
gvks, _, err := scheme.Scheme.ObjectKinds(resource)
if err != nil {
Expand All @@ -411,6 +422,78 @@ func redactList(list []*api.GatheredResource) error {
return nil
}

// Meant for typed clientset objects.
func RemoveTypedKeys(excludeAnnotKeys []*regexp.Regexp, m map[string]string) {
for key := range m {
for _, excludeAnnotKey := range excludeAnnotKeys {
if excludeAnnotKey.MatchString(key) {
delete(m, key)
}
}
}
}

// Meant for unstructured clientset objects. Removes the keys from the field
// given as input. For example, let's say we have the following object:
//
// {
// "metadata": {
// "annotations": {
// "key1": "value1",
// "key2": "value2"
// }
// }
// }
//
// Then, the following call:
//
// RemoveUnstructuredKeys("^key1$", obj, "metadata", "annotations")
//
// Will result in:
//
// {
// "metadata": {
// "annotations": {"key2": "value2"}
// }
// }
//
// If the given path doesn't exist or leads to a non-map object, nothing
// happens. The leaf object must either be a map[string]interface{} (that's
// what's returned by the unstructured clientset) or a map[string]string (that's
// what's returned by the typed clientset).
func RemoveUnstructuredKeys(excludeKeys []*regexp.Regexp, obj *unstructured.Unstructured, path ...string) {
annotsRaw, ok, err := unstructured.NestedFieldNoCopy(obj.Object, path...)
if err != nil {
return
}
if !ok {
return
}

// The field may be nil since yaml.Unmarshal's omitempty might not be set on
// on this struct field.
if annotsRaw == nil {
return
}

// The only possible type in an unstructured.Unstructured object is
// map[string]interface{}. That's because the yaml.Unmarshal func is used
// with an empty map[string]interface{} object, which means all nested
// objects will be unmarshalled to a map[string]interface{}.
annots, ok := annotsRaw.(map[string]interface{})
if !ok {
return
}

for key := range annots {
for _, excludeAnnotKey := range excludeKeys {
if excludeAnnotKey.MatchString(key) {
delete(annots, key)
}
}
}
}

// generateExcludedNamespacesFieldSelector creates a field selector string from
// a list of namespaces to exclude.
func generateExcludedNamespacesFieldSelector(excludeNamespaces []string) fields.Selector {
Expand Down
Loading