diff --git a/charts/logzio-logs-collector/Chart.yaml b/charts/logzio-logs-collector/Chart.yaml new file mode 100644 index 00000000..acc007b0 --- /dev/null +++ b/charts/logzio-logs-collector/Chart.yaml @@ -0,0 +1,10 @@ +apiVersion: v2 +name: logzio-logs-collector +version: 1.0.0 +description: kubernetes logs collection agent for logz.io based on opentelemetry collector +type: application +home: https://logz.io/ +maintainers: + - name: yotam loewenbach + email: yotam.loewenbach@logz.io +appVersion: 0.80.0 diff --git a/charts/logzio-logs-collector/README.md b/charts/logzio-logs-collector/README.md new file mode 100644 index 00000000..127d6f59 --- /dev/null +++ b/charts/logzio-logs-collector/README.md @@ -0,0 +1,139 @@ +# logzio-logs-collector + +**In development** + +kubernetes logs collection agent for logz.io based on opentelemetry collector + +## Prerequisites + +- Kubernetes 1.24+ +- Helm 3.9+ + +Below is the extended README.md including the full configuration table based on the provided `values.yaml`. + +* * * + +Logz.io Logs Collector for Kubernetes +===================================== + +The `logzio-logs-collector` Helm chart deploys a Kubernetes logs collection agent designed to forward logs from your Kubernetes clusters to Logz.io. This solution leverages the OpenTelemetry Collector, offering a robust and flexible way to manage log data, ensuring that your logging infrastructure scales with your application needs. + +Features +-------- + +* **Easy Integration with Logz.io**: Pre-configured to send logs to Logz.io, simplifying setup and integration. +* **Secure Secret Management**: Option to automatically manage secrets for seamless and secure authentication with Logz.io. + +Getting Started +--------------- + +### Add Logz.io Helm Repository + +Before installing the chart, add the Logz.io Helm repository: + +``` +helm repo add logzio-helm https://logzio.github.io/logzio-helm +helm repo update +``` + +### Installation + +1. **Create the Logz.io Secret** + + If not managing secrets externally, ensure the Logz.io secret with your shipping token and other relevant information is created. + +2. **Install the Chart** + + Install `logzio-logs-collector` from the Logz.io Helm repository, with logzio authentication values: + + ``` + helm install logzio-logs-collector -n monitoring \ + --set enabled=true \ + --set secrets.logzioLogsToken=<> \ + --set secrets.logzioRegion=<> \ + --set secrets.env_id=<> \ + --set secrets.logType=<> \ + logzio-helm/logzio-logs-collector + ``` + + Replace `logzio-logs-collector` with your release name. + +### Uninstalling the Chart + +To uninstall/delete the `logzio-logs-collector` deployment: + +shellCopy code + +`helm delete -n monitoring logzio-logs-collector` + +Configuration +------------- + +The following table lists the configurable parameters of the `logzio-logs-collector` chart and their default values. + +| Key | Description | Default Value | +|--------------------------|----------------------------------------------------------------------------------|----------------------------------------| +| enabled | Toggle for enabling the Helm chart deployment. | `false` | +| nameOverride | Override the default name for the deployment. | `""` | +| fullnameOverride | Set a full name override for the deployment. | `""` | +| mode | Deployment mode (currently supports only "daemonset"). | `"daemonset"` | +| namespaceOverride | Override the namespace into which the resources will be deployed. | `""` | +| fargateLogRouter.enabled | Boolean to decide if to configure Fargate log router (EKS Fargate environments). | `false` | +| secrets.enabled | Toggle for creating and managing the Logz.io secret by this chart. | `true` | +| secrets.name | The name of the secret for Logz.io log collector. | `"logzio-log-collector-secrets"` | +| secrets.env_id | Environment identifier attribute added to all logs. | `"my_env"` | +| secrets.logType | Default log type field. | `"k8s"` | +| secrets.logzioLogsToken | Secret with your Logz.io logs shipping token. | `"token"` | +| secrets.LogzioRegion | Secret with your Logz.io region. | `"us"` | +| secrets.customEndpoint | Secret with your custom endpoint, overrides Logz.io region listener address. | `""` | +| configMap.create | Specifies whether a configMap should be created. | `true` | +| config | Base collector configuration, supports templating. | Complex structure (see `values.yaml`) | +| image.repository | Docker image repository. | `"otel/opentelemetry-collector-contrib"` | +| image.pullPolicy | Image pull policy. | `"IfNotPresent"` | +| image.tag | Overrides the image tag. | `""` | +| image.digest | Pull images by digest. | `""` | +| imagePullSecrets | Specifies image pull secrets. | `[]` | +| command.name | OpenTelemetry Collector executable. | `"otelcol-contrib"` | +| command.extraArgs | Additional arguments for the command. | `[]` | +| serviceAccount.create | Specifies whether a service account should be created. | `true` | +| serviceAccount.name | The name of the service account to use. | `""` | +| clusterRole.create | Specifies whether a clusterRole should be created. | `true` | +| clusterRole.name | The name of the clusterRole to use. | `""` | +| podSecurityContext | Security context policies for the pod. | `{}` | +| securityContext | Security context policies for the container. | `{}` | +| nodeSelector | Node labels for pod assignment. | `{}` | +| tolerations | Tolerations for pod assignment. | `[]` | +| affinity | Affinity rules for pod assignment. | Complex structure (see `values.yaml`) | +| priorityClassName | Scheduler priority class name. | `""` | +| extraEnvs | Extra environment variables to set in the pods. | `[]` | +| ports | Defines ports configurations. | Complex structure (see `values.yaml`) | +| resources | CPU/memory resource requests/limits. | `limits.cpu:250m` `limits.cpu:512Mi` | +| podAnnotations | Annotations to add to the pod. | `{}` | +| podLabels | Labels to add to the pod. | `{}` | +| hostNetwork | Use the host's network namespace. | `false` | +| dnsPolicy | Pod DNS policy. | `""` | +| livenessProbe | Liveness probe configuration. | (see `values.yaml`) | +| readinessProbe | Readiness probe configuration. | (see `values.yaml`) | +| service.enabled | Enable the creation of a Service. | `true` | +| ingress.enabled | Enable ingress resource creation. | `false` | +| podMonitor.enabled | Enable the creation of a PodMonitor. | `false` | +| networkPolicy.enabled | Enable NetworkPolicy creation. | `false` | +| useGOMEMLIMIT | Set GOMEMLIMIT env var to a percentage of resources.limits.memory. | `false` | + +### Configure customization options + +You can use the following options to update the Helm chart parameters: + +* Specify parameters using the `--set key=value[,key=value]` argument to `helm install` + +* Edit the `values.yaml` + +* Overide default values with your own `my_values.yaml` and apply it in the `helm install` command. + +Multi line logs configuration +----------------------------- +The collector supports by default various log formats (including multiline logs) such as `CRI-O` `CRI-Containerd` `Docker` formats. You can configure the chart to parse custom multiline logs pattern according to your needs, please read [Customizing Multiline Log Handling](./examples/multiline.md) guide for more details. + +## Change log +* 1.0.0 + - kubernetes logs collection agent for logz.io based on opentelemetry collector \ No newline at end of file diff --git a/charts/logzio-logs-collector/examples/multiline.md b/charts/logzio-logs-collector/examples/multiline.md new file mode 100644 index 00000000..c2009dcc --- /dev/null +++ b/charts/logzio-logs-collector/examples/multiline.md @@ -0,0 +1,211 @@ +Customizing Multiline Log Handling +================================== + +Handling multiline logs efficiently in the OpenTelemetry Collector deployed through `logzio-logs-collector` Helm chart requires understanding of the `filelog` receiver, particularly the `recombine` operator. This operator is crucial for parsing and recombining logs that span multiple lines into a single entry, such as stack traces or multiline application logs. + +Key Configuration Options of the `recombine` Operator +----------------------------------------------------- + +To tailor the `filelog` receiver for multiline logs, focus on these essential `recombine` operator configurations: + +* **`combine_field`**: Specifies which field of the log entry should be combined into a single entry. Typically, this is the message body of the log (`body` or `body.message`). + +* **`is_first_entry`** and **`is_last_entry`**: Logical expressions that evaluate to `true` if the log entry being processed is the beginning or the end of a multiline series, respectively. You need to specify at least one of these based on whether you can reliably identify the start or the end of a multiline log entry. + +* **`combine_with`**: Defines the delimiter used to concatenate the log entries. For most logs, `"\n"` (newline) is a suitable delimiter, preserving the original log's structure. + +* **`source_identifier`**: Helps differentiate between log sources when combining entries, ensuring logs from different sources are not mistakenly merged. + + +Creating Custom Formats for Multiline Logs +------------------------------------------ + +To configure custom formats, you must understand your logs' structure to accurately use `is_first_entry` or `is_last_entry` expressions. Regular expressions (regex) are powerful tools in matching specific log patterns, allowing you to identify the start or end of a multiline log entry effectively. + +Custom multiline `recombine` operators should be added before `move from attributes.log to body`: +```yaml + # Update body field after finishing all parsing + - from: attributes.log + to: body + type: move +``` +Here is an example `custom-values.yaml` that shows how where to add custom multiline `recombine` operators: +```yaml +secrets: + enabled: true + logzioLogsToken: "<>" + +config: + receivers: + filelog: + operators: + - id: get-format + routes: + - expr: body matches "^\\{" + output: parser-docker + - expr: body matches "^[^ Z]+ " + output: parser-crio + - expr: body matches "^[^ Z]+Z" + output: parser-containerd + type: router + - id: parser-crio + regex: ^(?P