From fc4b6c7ace031c2eae47880abfbba035dac78b99 Mon Sep 17 00:00:00 2001 From: Michalis Katsoulis Date: Mon, 11 Nov 2024 14:50:04 +0200 Subject: [PATCH 01/11] Update kubernetes provider documentation --- .../composable/providers/kubernetes/README.md | 100 ++++++++++++------ 1 file changed, 67 insertions(+), 33 deletions(-) diff --git a/internal/pkg/composable/providers/kubernetes/README.md b/internal/pkg/composable/providers/kubernetes/README.md index 8fe7492c87a..d1d75f8642f 100644 --- a/internal/pkg/composable/providers/kubernetes/README.md +++ b/internal/pkg/composable/providers/kubernetes/README.md @@ -1,20 +1,32 @@ -## Autodiscover provider in Elastic Agent +## Autodiscover provider in Elastic Agent -https://www.elastic.co/guide/en/fleet/8.0/dynamic-input-configuration.html#dynamic-providers + -Currently Kubernetes dynamic provider can only be configured in [standalone](https://github.com/elastic/beats/blob/03bf16907bea9768427f8305a5c345368b55d834/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L24) agent. -In fleet managed agent it is enabled by default with default values. +https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html -Template based autodiscover of Kubernetes resources is only supported in standalone mode as of now. -It is not part of the Kubernetes Integration yet. + -Hints based autodiscover is not supported yet. +Kubernetes provider can be configured both in **standalone** and **fleet managed** elastic agent. -### Template based autodiscover +a. In **standalone** elastic agent, user needs to edit the [elastic-agent-standalone-kubernetes.yaml](https://github.com/elastic/elastic-agent/blob/f994f5bfdf68db27902a4175c3b655b4d611cf7c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L28) +b. In **fleet managed** elastic agent, kubernetes provider is enabled by default with default values. User needs to follow the steps described in the [documentation](https://www.elastic.co/guide/en/fleet/current/advanced-kubernetes-managed-by-fleet.html) to configure it. + +Condition based autodiscover is supported both in **fleet managed** elastic agent and in **standalone** elastic agent(see [doc](https://www.elastic.co/guide/en/fleet/current/conditions-based-autodiscover.html)). + +Hints based autodiscover is only supported in **standalone** elastic agent(see [doc](https://www.elastic.co/guide/en/fleet/current/hints-annotations-autodiscovery.html)). + + + +### Conditions based autodiscover + + Example: -As an example we will use again redis module. -In agent.yml(configmap) an extra input block needs to be added. + +As an example we will use the redis module. + +To automatically identify a Redis Pod and monitor it with the Redis integration, add the following input configuration inside the [Elastic Agent Standalone manifest](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml): + ``` # Add extra input blocks here, based on conditions # so as to automatically identify targeted Pods and start monitoring them @@ -43,28 +55,50 @@ In agent.yml(configmap) an extra input block needs to be added. condition: ${kubernetes.pod.labels.app} == 'redis' ``` -What makes this input block dynamic are the variables hosts and condition. -`${kubernetes.pod.ip}` and `${kubernetes.pod.labels.app}` + + +What makes this input block dynamic are the variables under hosts block and the condition. + +`${kubernetes.pod.ip}` and `${kubernetes.pod.labels.app} == 'redis'` + + #### High level description -The Kubernetes dynamic provider watches for Kubernetes resources and generates mappings from them (similar to events in beats provider). The mappings include those variables([list of variables](https://www.elastic.co/guide/en/fleet/03bf16907bea9768427f8305a5c345368b55d834/dynamic-input-configuration.html#kubernetes-provider)) for each k8s resource with unique value for each one of them. -Agent composable controller which controls all the providers receives these mappings and tries to match them with the input blogs of the configurations. -This means that for every mapping that the condition matches (kubernetes.pod.labels.app equals to redis), a -new input will be created in which the condition will be removed(not needed anymore) and the `kubernetes.pod.ip` variable will be substituted from the value in the same mapping. -The updated complete inputs blog will be then forwarded to agent to spawn/update metricbeat and filebeat instances. - -##### Internals - -Step-by-step walkthrough -1. Elastic agent running in local mode initiates a new [composable controller](https://github.com/elastic/beats/blob/03bf16907bea9768427f8305a5c345368b55d834/x-pack/elastic-agent/pkg/agent/application/local_mode.go#L112). -2. The controller consists of all contextProviders and [dynamicProviders](https://github.com/elastic/beats/blob/03bf16907bea9768427f8305a5c345368b55d834/x-pack/elastic-agent/pkg/composable/controller.go#L73). -3. Agent initiates a new [emitter](https://github.com/elastic/beats/blob/03bf16907bea9768427f8305a5c345368b55d834/x-pack/elastic-agent/pkg/agent/application/local_mode.go#L118) which [starts](https://github.com/elastic/beats/blob/03bf16907bea9768427f8305a5c345368b55d834/x-pack/elastic-agent/pkg/agent/application/pipeline/emitter/emitter.go#L27) all the dynamicProviders of the [controller](https://github.com/elastic/beats/blob/03bf16907bea9768427f8305a5c345368b55d834/x-pack/elastic-agent/pkg/composable/controller.go#L122). -4. Kubernetes Dynamic provider depending on the [resource type](https://github.com/elastic/beats/blob/03bf16907bea9768427f8305a5c345368b55d834/x-pack/elastic-agent/pkg/composable/providers/kubernetes/kubernetes.go#L56) (default is pod) initiates a [watcher](https://github.com/elastic/beats/blob/3c77c9a92a2e90b85f525293cb4c2cfc5bc996b1/x-pack/elastic-agent/pkg/composable/providers/kubernetes/pod.go#L69) for - the specific resource the same way as in metrcbeat/filebeat kubernetes provider. -5. Under the hood a dedicated watcher starts for pods, nodes and namespaces as well as a metadata generator. -6. The difference is that the watchers instead of publishing events, they create [data](https://github.com/elastic/beats/blob/3c77c9a92a2e90b85f525293cb4c2cfc5bc996b1/x-pack/elastic-agent/pkg/composable/providers/kubernetes/pod.go#L134) from the objects read from the queue. These [data](https://github.com/elastic/beats/blob/3c77c9a92a2e90b85f525293cb4c2cfc5bc996b1/x-pack/elastic-agent/pkg/composable/providers/kubernetes/pod.go#L244) consist of mappings, processors and a priority . -7. A [mapping](https://github.com/elastic/beats/blob/3c77c9a92a2e90b85f525293cb4c2cfc5bc996b1/x-pack/elastic-agent/pkg/composable/providers/kubernetes/pod.go#L217) includes all those variables retrieved from the kubernetes resource metadata while the [processors](https://github.com/elastic/beats/blob/3c77c9a92a2e90b85f525293cb4c2cfc5bc996b1/x-pack/elastic-agent/pkg/composable/providers/kubernetes/pod.go#L236) indicate the addition of extra fields. -8. Composable controller [collects](https://github.com/elastic/beats/blob/3c77c9a92a2e90b85f525293cb4c2cfc5bc996b1/x-pack/elastic-agent/pkg/composable/controller.go#L244) the created data and [compares](https://github.com/elastic/beats/blob/3c77c9a92a2e90b85f525293cb4c2cfc5bc996b1/x-pack/elastic-agent/pkg/composable/controller.go#L263) their mappings and processors against the existing ones. If there is a change, it updates the dynamicProviderState and notifies a worker thread through a [signal](https://github.com/elastic/beats/blob/3c77c9a92a2e90b85f525293cb4c2cfc5bc996b1/x-pack/elastic-agent/pkg/composable/controller.go#L272). -9. When the worker gets [notified](https://github.com/elastic/beats/blob/3c77c9a92a2e90b85f525293cb4c2cfc5bc996b1/x-pack/elastic-agent/pkg/composable/controller.go#L141) for a change it creates new [variables](https://github.com/elastic/beats/blob/3c77c9a92a2e90b85f525293cb4c2cfc5bc996b1/x-pack/elastic-agent/pkg/composable/controller.go#L170) from the mappings and processors. -10. It then updates the [emitter](https://github.com/elastic/beats/blob/3c77c9a92a2e90b85f525293cb4c2cfc5bc996b1/x-pack/elastic-agent/pkg/agent/application/pipeline/emitter/controller.go#L111) with them. -11. The emitter controller will update the ast that is then used by the agent to generate the final inputs and spawn new programs to deploy the changes([code](https://github.com/elastic/beats/blob/3c77c9a92a2e90b85f525293cb4c2cfc5bc996b1/x-pack/elastic-agent/pkg/agent/application/pipeline/emitter/controller.go#L151)). + +The Kubernetes provider watches for Kubernetes resources and generates mappings from them (similar to events in beats provider). The mappings include those variables([list of variables](https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html#_provider_for_pod_resources)) for each k8s resource with unique value for each one of them. + +Agent composable controller which controls all the providers receives these mappings and tries to match them with the input blogs of the configurations. + +This means that for every mapping that the condition matches (`kubernetes.pod.labels.app == redis`), a new input will be created in which the condition will be removed(not needed anymore) and the `kubernetes.pod.ip` variable will be substituted from the value in the same mapping. + +The updated complete inputs block will be then forwarded to agent to spawn/update metricbeat and filebeat instances. + + +### Hints based autodiscover + +Standalone elastic agent supports autodiscover based on hints collected from the [Kubernetes Provider](https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html). The hints mechanism looks for hints in kubernetes pod annotations that have the prefix `co.elastic.hints`. As soon as the container starts, elastic agent checks it for hints and launches the proper configuration for the container. Hints tell elastic agent how to monitor the container by using the proper integration. +The full list of supported hints can be found [here](https://www.elastic.co/guide/en/fleet/current/hints-annotations-autodiscovery.html#_required_hints). + +Example: + +As an example we will use again redis module. +Add the following annotations to a redis pod. Elastic agent will then initiate a new input with redis module to properly monitor the redis pod. +``` +apiVersion: v1 +kind: Pod +metadata: + name: redis + annotations: + co.elastic.hints/package: redis + co.elastic.hints/data_streams: info + co.elastic.hints/info.period: 5m +``` + +In order to enable hints based autodiscover, user needs to uncomment the following lines in the [Elastic Agent Standalone manifest](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml) +1. Add the [init container]((https://github.com/elastic/elastic-agent/blob/c01636e7383a9b2af9a588e0fcf1a4cae7d0d65c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L697-L709)) that downloads the templates of various packages. +2. Add the [volumeMount](https://github.com/elastic/elastic-agent/blob/c01636e7383a9b2af9a588e0fcf1a4cae7d0d65c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L783-L785) and [volume](https://github.com/elastic/elastic-agent/blob/c01636e7383a9b2af9a588e0fcf1a4cae7d0d65c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L824-L826). + +The init container will start before the elastic agent pod and will donwload all the templates of packages [supported](https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone/templates.d). +Elastic agent will then collect from the pods running(through the watchers mechanism) all the hints annotations and will try to match them with the correct package. +In the redis example, it will use the [redis template](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone/templates.d/redis.yml) and substitute the template variables with the values specified in the annotations. Default values will be used for variables not provided. +A new input will be then created for redis and reds data will start being collected. \ No newline at end of file From 9f41ae1baa7f122feba59b645bf0c01303774069 Mon Sep 17 00:00:00 2001 From: Michael Katsoulis Date: Mon, 18 Nov 2024 16:48:54 +0200 Subject: [PATCH 02/11] Update internal/pkg/composable/providers/kubernetes/README.md Co-authored-by: Panos Koutsovasilis --- internal/pkg/composable/providers/kubernetes/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/pkg/composable/providers/kubernetes/README.md b/internal/pkg/composable/providers/kubernetes/README.md index d1d75f8642f..442248c1b6e 100644 --- a/internal/pkg/composable/providers/kubernetes/README.md +++ b/internal/pkg/composable/providers/kubernetes/README.md @@ -101,4 +101,4 @@ In order to enable hints based autodiscover, user needs to uncomment the followi The init container will start before the elastic agent pod and will donwload all the templates of packages [supported](https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone/templates.d). Elastic agent will then collect from the pods running(through the watchers mechanism) all the hints annotations and will try to match them with the correct package. In the redis example, it will use the [redis template](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone/templates.d/redis.yml) and substitute the template variables with the values specified in the annotations. Default values will be used for variables not provided. -A new input will be then created for redis and reds data will start being collected. \ No newline at end of file +A new input will be then created for redis and redis-specific data will start being collected. \ No newline at end of file From ef5d6c0b51b682211b667ee4fcd30a2f88bc69cf Mon Sep 17 00:00:00 2001 From: Michael Katsoulis Date: Mon, 18 Nov 2024 16:49:08 +0200 Subject: [PATCH 03/11] Update internal/pkg/composable/providers/kubernetes/README.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Mikołaj Świątek --- internal/pkg/composable/providers/kubernetes/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/pkg/composable/providers/kubernetes/README.md b/internal/pkg/composable/providers/kubernetes/README.md index 442248c1b6e..8c8386e372d 100644 --- a/internal/pkg/composable/providers/kubernetes/README.md +++ b/internal/pkg/composable/providers/kubernetes/README.md @@ -76,7 +76,7 @@ The updated complete inputs block will be then forwarded to agent to spawn/updat ### Hints based autodiscover -Standalone elastic agent supports autodiscover based on hints collected from the [Kubernetes Provider](https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html). The hints mechanism looks for hints in kubernetes pod annotations that have the prefix `co.elastic.hints`. As soon as the container starts, elastic agent checks it for hints and launches the proper configuration for the container. Hints tell elastic agent how to monitor the container by using the proper integration. +Standalone elastic agent supports autodiscover based on hints collected from the [Kubernetes Provider](https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html). The hints mechanism looks for hints in kubernetes pod annotations that have the prefix `co.elastic.hints`. As soon as the Pod is ready, elastic agent checks it for hints and launches the proper configuration for the container. Hints tell elastic agent how to monitor the container by using the proper integration. The full list of supported hints can be found [here](https://www.elastic.co/guide/en/fleet/current/hints-annotations-autodiscovery.html#_required_hints). Example: From 5e8dbb8ec9be82908e78dae7496fe35270df85fa Mon Sep 17 00:00:00 2001 From: Michael Katsoulis Date: Mon, 18 Nov 2024 16:49:20 +0200 Subject: [PATCH 04/11] Update internal/pkg/composable/providers/kubernetes/README.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Mikołaj Świątek --- internal/pkg/composable/providers/kubernetes/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/pkg/composable/providers/kubernetes/README.md b/internal/pkg/composable/providers/kubernetes/README.md index 8c8386e372d..5d38d47f2f0 100644 --- a/internal/pkg/composable/providers/kubernetes/README.md +++ b/internal/pkg/composable/providers/kubernetes/README.md @@ -67,7 +67,7 @@ What makes this input block dynamic are the variables under hosts block and the The Kubernetes provider watches for Kubernetes resources and generates mappings from them (similar to events in beats provider). The mappings include those variables([list of variables](https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html#_provider_for_pod_resources)) for each k8s resource with unique value for each one of them. -Agent composable controller which controls all the providers receives these mappings and tries to match them with the input blogs of the configurations. +Agent composable controller which controls all the providers receives these mappings and tries to match them with the input blocks of the configurations. This means that for every mapping that the condition matches (`kubernetes.pod.labels.app == redis`), a new input will be created in which the condition will be removed(not needed anymore) and the `kubernetes.pod.ip` variable will be substituted from the value in the same mapping. From f7aec7063ae373d3b023c72aa2f7b173c38abb6b Mon Sep 17 00:00:00 2001 From: Michalis Katsoulis Date: Tue, 19 Nov 2024 10:27:23 +0200 Subject: [PATCH 05/11] Review comments updates --- .../pkg/composable/providers/kubernetes/README.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/internal/pkg/composable/providers/kubernetes/README.md b/internal/pkg/composable/providers/kubernetes/README.md index 5d38d47f2f0..375107f9cbc 100644 --- a/internal/pkg/composable/providers/kubernetes/README.md +++ b/internal/pkg/composable/providers/kubernetes/README.md @@ -8,7 +8,7 @@ https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html Kubernetes provider can be configured both in **standalone** and **fleet managed** elastic agent. -a. In **standalone** elastic agent, user needs to edit the [elastic-agent-standalone-kubernetes.yaml](https://github.com/elastic/elastic-agent/blob/f994f5bfdf68db27902a4175c3b655b4d611cf7c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L28) +a. In **standalone** elastic agent, the kubernetes provider is configured inside the [elastic-agent-standalone-kubernetes.yaml](https://github.com/elastic/elastic-agent/blob/f994f5bfdf68db27902a4175c3b655b4d611cf7c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L28). User can update the parameters based on their needs. b. In **fleet managed** elastic agent, kubernetes provider is enabled by default with default values. User needs to follow the steps described in the [documentation](https://www.elastic.co/guide/en/fleet/current/advanced-kubernetes-managed-by-fleet.html) to configure it. Condition based autodiscover is supported both in **fleet managed** elastic agent and in **standalone** elastic agent(see [doc](https://www.elastic.co/guide/en/fleet/current/conditions-based-autodiscover.html)). @@ -19,9 +19,6 @@ Hints based autodiscover is only supported in **standalone** elastic agent(see [ ### Conditions based autodiscover - - -Example: As an example we will use the redis module. @@ -67,7 +64,7 @@ What makes this input block dynamic are the variables under hosts block and the The Kubernetes provider watches for Kubernetes resources and generates mappings from them (similar to events in beats provider). The mappings include those variables([list of variables](https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html#_provider_for_pod_resources)) for each k8s resource with unique value for each one of them. -Agent composable controller which controls all the providers receives these mappings and tries to match them with the input blocks of the configurations. +Agent [composable controller](https://github.com/elastic/elastic-agent/blob/f994f5bfdf68db27902a4175c3b655b4d611cf7c/internal/pkg/composable/controller.go#L117) which controls all the providers [receives](https://github.com/elastic/elastic-agent/blob/f994f5bfdf68db27902a4175c3b655b4d611cf7c/internal/pkg/composable/controller.go#L371) these mappings and tries to match them with the input blocks of the configurations. This means that for every mapping that the condition matches (`kubernetes.pod.labels.app == redis`), a new input will be created in which the condition will be removed(not needed anymore) and the `kubernetes.pod.ip` variable will be substituted from the value in the same mapping. @@ -95,8 +92,12 @@ metadata: ``` In order to enable hints based autodiscover, user needs to uncomment the following lines in the [Elastic Agent Standalone manifest](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml) -1. Add the [init container]((https://github.com/elastic/elastic-agent/blob/c01636e7383a9b2af9a588e0fcf1a4cae7d0d65c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L697-L709)) that downloads the templates of various packages. -2. Add the [volumeMount](https://github.com/elastic/elastic-agent/blob/c01636e7383a9b2af9a588e0fcf1a4cae7d0d65c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L783-L785) and [volume](https://github.com/elastic/elastic-agent/blob/c01636e7383a9b2af9a588e0fcf1a4cae7d0d65c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L824-L826). +1. [Enable hints](https://github.com/elastic/elastic-agent/blob/4e983446837ff551a9f058fdc575a193d9afcbab/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L32) feature in kubernetes provider configuration. +2. Add the [init container]((https://github.com/elastic/elastic-agent/blob/c01636e7383a9b2af9a588e0fcf1a4cae7d0d65c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L697-L709)) that downloads the templates of various packages. +3. Add the [volumeMount](https://github.com/elastic/elastic-agent/blob/c01636e7383a9b2af9a588e0fcf1a4cae7d0d65c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L783-L785) and [volume](https://github.com/elastic/elastic-agent/blob/c01636e7383a9b2af9a588e0fcf1a4cae7d0d65c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L824-L826). + +**Note:** If hints are enabled, the [`default_container_logs`](https://github.com/elastic/elastic-agent/blob/4e983446837ff551a9f058fdc575a193d9afcbab/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L33C8-L33C42) option is set to `True` by default. This means that logs from all discovered pods will be collected automatically, without requiring any annotations. To disable this behavior, set the option to `False`. This will stop the automatic log collection from all discovered pods. In this case, you must explicitly annotate your pods with the annotation `co.elastic.hints/package: "container_logs"` to collect their logs. + The init container will start before the elastic agent pod and will donwload all the templates of packages [supported](https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone/templates.d). Elastic agent will then collect from the pods running(through the watchers mechanism) all the hints annotations and will try to match them with the correct package. From 9152fa97f8d0eaea2914d7d4039285ccd921ffa3 Mon Sep 17 00:00:00 2001 From: Michalis Katsoulis Date: Tue, 19 Nov 2024 10:28:57 +0200 Subject: [PATCH 06/11] Add empty line --- internal/pkg/composable/providers/kubernetes/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/internal/pkg/composable/providers/kubernetes/README.md b/internal/pkg/composable/providers/kubernetes/README.md index 375107f9cbc..e51583a22fb 100644 --- a/internal/pkg/composable/providers/kubernetes/README.md +++ b/internal/pkg/composable/providers/kubernetes/README.md @@ -9,6 +9,7 @@ https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html Kubernetes provider can be configured both in **standalone** and **fleet managed** elastic agent. a. In **standalone** elastic agent, the kubernetes provider is configured inside the [elastic-agent-standalone-kubernetes.yaml](https://github.com/elastic/elastic-agent/blob/f994f5bfdf68db27902a4175c3b655b4d611cf7c/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml#L28). User can update the parameters based on their needs. + b. In **fleet managed** elastic agent, kubernetes provider is enabled by default with default values. User needs to follow the steps described in the [documentation](https://www.elastic.co/guide/en/fleet/current/advanced-kubernetes-managed-by-fleet.html) to configure it. Condition based autodiscover is supported both in **fleet managed** elastic agent and in **standalone** elastic agent(see [doc](https://www.elastic.co/guide/en/fleet/current/conditions-based-autodiscover.html)). From 9d1fe9e0ba433a4c98c7e1c9346477ca210ca9e3 Mon Sep 17 00:00:00 2001 From: Michalis Katsoulis Date: Tue, 19 Nov 2024 10:35:49 +0200 Subject: [PATCH 07/11] Create changelog fragment --- ...1732005233-kubernetes-provider-readme.yaml | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 changelog/fragments/1732005233-kubernetes-provider-readme.yaml diff --git a/changelog/fragments/1732005233-kubernetes-provider-readme.yaml b/changelog/fragments/1732005233-kubernetes-provider-readme.yaml new file mode 100644 index 00000000000..bd17950b6b1 --- /dev/null +++ b/changelog/fragments/1732005233-kubernetes-provider-readme.yaml @@ -0,0 +1,32 @@ +# Kind can be one of: +# - breaking-change: a change to previously-documented behavior +# - deprecation: functionality that is being removed in a later release +# - bug-fix: fixes a problem in a previous version +# - enhancement: extends functionality but does not break or fix existing behavior +# - feature: new functionality +# - known-issue: problems that we are aware of in a given version +# - security: impacts on the security of a product or a user’s deployment. +# - upgrade: important information for someone upgrading from a prior version +# - other: does not fit into any of the other categories +kind: feature + +# Change summary; a 80ish characters long description of the change. +summary: Update kubernetes composable provider README + +# Long description; in case the summary is not enough to describe the change +# this field accommodate a description without length limits. +# NOTE: This field will be rendered only for breaking-change and known-issue kinds at the moment. +#description: + +# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc. +component: elastic-agent + +# PR URL; optional; the PR number that added the changeset. +# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added. +# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number. +# Please provide it if you are adding a fragment for a different PR. +#pr: https://github.com/owner/repo/1234 + +# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of). +# If not present is automatically filled by the tooling with the issue linked to the PR number. +#issue: https://github.com/owner/repo/1234 From d9551d9692e1866e06488b4f11f9929d1e45f33b Mon Sep 17 00:00:00 2001 From: Michael Katsoulis Date: Wed, 20 Nov 2024 09:42:17 +0200 Subject: [PATCH 08/11] Update internal/pkg/composable/providers/kubernetes/README.md Co-authored-by: Andrzej Stencel --- internal/pkg/composable/providers/kubernetes/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/pkg/composable/providers/kubernetes/README.md b/internal/pkg/composable/providers/kubernetes/README.md index e51583a22fb..e2836b51a53 100644 --- a/internal/pkg/composable/providers/kubernetes/README.md +++ b/internal/pkg/composable/providers/kubernetes/README.md @@ -101,6 +101,6 @@ In order to enable hints based autodiscover, user needs to uncomment the followi The init container will start before the elastic agent pod and will donwload all the templates of packages [supported](https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone/templates.d). -Elastic agent will then collect from the pods running(through the watchers mechanism) all the hints annotations and will try to match them with the correct package. +Elastic agent will then collect from the pods running (through the watchers mechanism) all the hints annotations and will try to match them with the correct package. In the redis example, it will use the [redis template](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone/templates.d/redis.yml) and substitute the template variables with the values specified in the annotations. Default values will be used for variables not provided. A new input will be then created for redis and redis-specific data will start being collected. \ No newline at end of file From 7bd72526df2af793607802a545dcb384c6b71ddf Mon Sep 17 00:00:00 2001 From: Michael Katsoulis Date: Wed, 20 Nov 2024 09:42:49 +0200 Subject: [PATCH 09/11] Update internal/pkg/composable/providers/kubernetes/README.md Co-authored-by: Andrzej Stencel --- internal/pkg/composable/providers/kubernetes/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/pkg/composable/providers/kubernetes/README.md b/internal/pkg/composable/providers/kubernetes/README.md index e2836b51a53..d784d2000f3 100644 --- a/internal/pkg/composable/providers/kubernetes/README.md +++ b/internal/pkg/composable/providers/kubernetes/README.md @@ -12,7 +12,7 @@ a. In **standalone** elastic agent, the kubernetes provider is configured insid b. In **fleet managed** elastic agent, kubernetes provider is enabled by default with default values. User needs to follow the steps described in the [documentation](https://www.elastic.co/guide/en/fleet/current/advanced-kubernetes-managed-by-fleet.html) to configure it. -Condition based autodiscover is supported both in **fleet managed** elastic agent and in **standalone** elastic agent(see [doc](https://www.elastic.co/guide/en/fleet/current/conditions-based-autodiscover.html)). +Condition based autodiscover is supported both in **fleet managed** elastic agent and in **standalone** elastic agent (see [doc](https://www.elastic.co/guide/en/fleet/current/conditions-based-autodiscover.html)). Hints based autodiscover is only supported in **standalone** elastic agent(see [doc](https://www.elastic.co/guide/en/fleet/current/hints-annotations-autodiscovery.html)). From 8b055ff73d4467c02e2ad4039fa4939055a57f40 Mon Sep 17 00:00:00 2001 From: Michael Katsoulis Date: Wed, 20 Nov 2024 09:43:07 +0200 Subject: [PATCH 10/11] Update internal/pkg/composable/providers/kubernetes/README.md Co-authored-by: Andrzej Stencel --- internal/pkg/composable/providers/kubernetes/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/pkg/composable/providers/kubernetes/README.md b/internal/pkg/composable/providers/kubernetes/README.md index d784d2000f3..6641a745776 100644 --- a/internal/pkg/composable/providers/kubernetes/README.md +++ b/internal/pkg/composable/providers/kubernetes/README.md @@ -63,7 +63,7 @@ What makes this input block dynamic are the variables under hosts block and the #### High level description -The Kubernetes provider watches for Kubernetes resources and generates mappings from them (similar to events in beats provider). The mappings include those variables([list of variables](https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html#_provider_for_pod_resources)) for each k8s resource with unique value for each one of them. +The Kubernetes provider watches for Kubernetes resources and generates mappings from them (similar to events in beats provider). The mappings include those variables ([list of variables](https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html#_provider_for_pod_resources)) for each k8s resource with unique value for each one of them. Agent [composable controller](https://github.com/elastic/elastic-agent/blob/f994f5bfdf68db27902a4175c3b655b4d611cf7c/internal/pkg/composable/controller.go#L117) which controls all the providers [receives](https://github.com/elastic/elastic-agent/blob/f994f5bfdf68db27902a4175c3b655b4d611cf7c/internal/pkg/composable/controller.go#L371) these mappings and tries to match them with the input blocks of the configurations. From d4257b5279ed6d56bf9e65bafc39ae7ab7f4b2c4 Mon Sep 17 00:00:00 2001 From: Michael Katsoulis Date: Wed, 20 Nov 2024 09:43:23 +0200 Subject: [PATCH 11/11] Update internal/pkg/composable/providers/kubernetes/README.md Co-authored-by: Andrzej Stencel --- internal/pkg/composable/providers/kubernetes/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/pkg/composable/providers/kubernetes/README.md b/internal/pkg/composable/providers/kubernetes/README.md index 6641a745776..49421f8bec4 100644 --- a/internal/pkg/composable/providers/kubernetes/README.md +++ b/internal/pkg/composable/providers/kubernetes/README.md @@ -14,7 +14,7 @@ b. In **fleet managed** elastic agent, kubernetes provider is enabled by default Condition based autodiscover is supported both in **fleet managed** elastic agent and in **standalone** elastic agent (see [doc](https://www.elastic.co/guide/en/fleet/current/conditions-based-autodiscover.html)). -Hints based autodiscover is only supported in **standalone** elastic agent(see [doc](https://www.elastic.co/guide/en/fleet/current/hints-annotations-autodiscovery.html)). +Hints based autodiscover is only supported in **standalone** elastic agent (see [doc](https://www.elastic.co/guide/en/fleet/current/hints-annotations-autodiscovery.html)).