Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft: data streams scenarios #1623

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
+253 −4
Draft

Draft: data streams scenarios #1623

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
257 changes: 253 additions & 4 deletions docs/en/ingest-management/data-streams.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,11 +294,260 @@ Refer to the breaking change in the 8.12.0 Release Notes for more detail and wor
See <<data-streams-pipeline-tutorial>> to get started.

[[data-streams-ilm-tutorial]]
== Tutorial: Customize data retention policies
== Tutorials: Customize data retention policies

This tutorial explains how to apply a custom {ilm-init} policy to an integration's data stream.
These tutorials explain how to apply a custom {ilm-init} policy to an integration's data stream.

**Scenario:** You have {agent}s collecting system metrics with the System integration in two environments--one with the namespace `development`, and one with `production`.
[discrete]
[[data-streams-general-info]]
== Before you begin

For certain features you'll need to use a slightly different procedure to manage the index lifecycle:

* APM: For verions 8.15 and later, refer to {observability-guide}/apm-ilm-how-to.html[Index lifecycle management].
* Synthetic monitoring: Refer to {observability-guide}/synthetics-manage-retention.html[Manage data retention].
* Universal Profiling: Refer to {observability-guide}/profiling-index-lifecycle-management.html[Universal Profiling index life cycle management].

[discrete]
[[data-streams-scenarios]]
== Identify your scenario

How you apply an ILM policy depends on your use case. Choose a scenario for the detailed steps.

* **<<data-streams-scenario1,Scenario 1>>**: You want to apply an ILM policy to all data streams generated from Fleet integrations across all namespaces.

* **<<data-streams-scenario2,Scenario 2>>**: You want to apply an ILM policy to one or more specific integrations across all namespaces.

* **<<data-streams-scenario3,Scenario 3>>**: You want apply an ILM policy for a data stream for a specific namespace in an integration


[[data-streams-scenario1]]
== Scenario 1: Apply an ILM policy to all data streams generated from Fleet integrations across all namespaces

++++
<titleabbrev>Scenario 1: All data streams; all namespaces</titleabbrev>
++++

NOTE: This tutorial uses a `global@custom` component template which is available in versions 8.12 and later. For versions 8.4 and later, and earlier than 8.12, you instead need to use the `<integration prefix>@custom component template` and add the ILM policy to that template. This needs to be done on every newly added integration.

**DRAFT CONTENT: ALL OF THE FOLLOWING IS COPIED VERBATIM FROM {observability-guide}/apm-ilm-how-to.html#apm-data-streams-custom-policy[Configure a custom index lifecycle policy]**


Mappings and settings for data streams can be customized through the creation of `*@custom` component templates,
which are referenced by the index templates created by the {es} apm-data plugin.
The easiest way to configure a custom index lifecycle policy per data stream is to edit this template.

This tutorial explains how to apply a custom index lifecycle policy to the `traces-apm` data stream.

[discrete]
[id="data-streams-scenario1-step1"]
== Step 1: View data streams

The **Data Streams** view in {kib} shows you data streams,
index templates, and lifecycle policies:

. To open **Index Management**, find **Stack Management** in the main menu or use the {kibana-ref}/introduction.html#kibana-navigation-search[global search field].
. Select **Data Streams**.
. Search for `traces-apm` to see all data streams associated with APM trace data.
. In this example, I only have one data stream because I'm only using the `default` namespace.
You may have more if your setup includes multiple namespaces.
+
[role="screenshot"]
image::images/data-stream-overview.png[Data streams info]

[discrete]
[id="data-streams-scenario1-step2"]
== Step 2: Create an index lifecycle policy

. To open **Lifecycle Policies**, find **Stack Management** in the main menu or use the {kibana-ref}/introduction.html#kibana-navigation-search[global search field].
. Click **Create policy**.

Name your new policy; For this tutorial, I've chosen `custom-traces-apm-policy`.
Customize the policy to your liking, and when you're done, click **Save policy**.

[discrete]
[id="data-streams-scenario1-step3"]
== Step 3: Apply the index lifecycle policy

To apply your new index lifecycle policy to the `traces-apm-*` data stream,
edit the `<data-stream-name>@custom` component template.

. Click on the **Component Template** tab and search for `traces-apm`.
. Select the `traces-apm@custom` template and click **Manage** → **Edit**.
. Under **Index settings**, set the {ilm-init} policy name created in the previous step:
+
[source,json]
----
{
"lifecycle": {
"name": "custom-traces-apm-policy",
"prefer_ilm": true
}
}
----
. Continue to **Review** and ensure your request looks similar to the image below.
If it does, click **Create component template**.
+
[role="screenshot"]
image::images/create-component-template.png[Create component template]

[discrete]
[id="data-streams-scenario1-step4"]
== Step 4: Roll over the data stream (optional)

To confirm that the data stream is now using the new index template and {ilm-init} policy,
you can either repeat Step 1, or navigate to **{dev-tools-app}** and run the following:

[source,bash]
----
GET /_data_stream/traces-apm-default <1>
----
<1> The name of the data stream we've been hacking on appended with your <namespace>

The result should include the following:

[source,json]
----
{
"data_streams" : [
{
...
"template" : "traces-apm-default", <1>
"ilm_policy" : "custom-traces-apm-policy", <2>
...
}
]
}
----
<1> The name of the custom index template created in step three
<2> The name of the {ilm-init} policy applied to the new component template in step two

New {ilm-init} policies only take effect when new indices are created,
so you either must wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB),
or force a rollover using the {ref}/indices-rollover-index.html[{es} rollover API]:

[source,bash]
----
POST /traces-apm-default/_rollover/
----

[[data-streams-scenario2]]
== Scenario 2: Apply an ILM policy to one or more specific integrations across all namespaces

++++
<titleabbrev>Scenario 2: Specific integrations; all namespaces</titleabbrev>
++++

NOTE: This tutorial uses a `<integration prefix>@custom` component template which is available in versions 8.4 and later.

**DRAFT CONTENT: ALL OF THE FOLLOWING IS COPIED VERBATIM FROM {observability-guide}/apm-ilm-how-to.html#apm-data-streams-custom-policy[Configure a custom index lifecycle policy]**

Mappings and settings for data streams can be customized through the creation of `*@custom` component templates,
which are referenced by the index templates created by the {es} apm-data plugin.
The easiest way to configure a custom index lifecycle policy per data stream is to edit this template.

This tutorial explains how to apply a custom index lifecycle policy to the `traces-apm` data stream.

[discrete]
[id="data-streams-scenario2-step1"]
== Step 1: View data streams

The **Data Streams** view in {kib} shows you data streams,
index templates, and lifecycle policies:

. To open **Index Management**, find **Stack Management** in the main menu or use the {kibana-ref}/introduction.html#kibana-navigation-search[global search field].
. Select **Data Streams**.
. Search for `traces-apm` to see all data streams associated with APM trace data.
. In this example, I only have one data stream because I'm only using the `default` namespace.
You may have more if your setup includes multiple namespaces.
+
[role="screenshot"]
image::images/data-stream-overview.png[Data streams info]

[discrete]
[id="data-streams-scenario2-step2"]
== Step 2: Create an index lifecycle policy

. To open **Lifecycle Policies**, find **Stack Management** in the main menu or use the {kibana-ref}/introduction.html#kibana-navigation-search[global search field].
. Click **Create policy**.

Name your new policy; For this tutorial, I've chosen `custom-traces-apm-policy`.
Customize the policy to your liking, and when you're done, click **Save policy**.

[discrete]
[id="data-streams-scenario2-step3"]
== Step 3: Apply the index lifecycle policy

To apply your new index lifecycle policy to the `traces-apm-*` data stream,
edit the `<data-stream-name>@custom` component template.

. Click on the **Component Template** tab and search for `traces-apm`.
. Select the `traces-apm@custom` template and click **Manage** → **Edit**.
. Under **Index settings**, set the {ilm-init} policy name created in the previous step:
+
[source,json]
----
{
"lifecycle": {
"name": "custom-traces-apm-policy",
"prefer_ilm": true
}
}
----
. Continue to **Review** and ensure your request looks similar to the image below.
If it does, click **Create component template**.
+
[role="screenshot"]
image::images/create-component-template.png[Create component template]

[discrete]
[id="data-streams-scenario2-step4"]
== Step 4: Roll over the data stream (optional)

To confirm that the data stream is now using the new index template and {ilm-init} policy,
you can either repeat Step 1, or navigate to **{dev-tools-app}** and run the following:

[source,bash]
----
GET /_data_stream/traces-apm-default <1>
----
<1> The name of the data stream we've been hacking on appended with your <namespace>

The result should include the following:

[source,json]
----
{
"data_streams" : [
{
...
"template" : "traces-apm-default", <1>
"ilm_policy" : "custom-traces-apm-policy", <2>
...
}
]
}
----
<1> The name of the custom index template created in step three
<2> The name of the {ilm-init} policy applied to the new component template in step two

New {ilm-init} policies only take effect when new indices are created,
so you either must wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB),
or force a rollover using the {ref}/indices-rollover-index.html[{es} rollover API]:

[source,bash]
----
POST /traces-apm-default/_rollover/
----

[[data-streams-scenario3]]
== Scenario 3: Apply an ILM policy with integrations using multiple namespaces

++++
<titleabbrev>Scenario 3: Specific integrations; specific namespaces</titleabbrev>
++++

In this scenario, you have {agent}s collecting system metrics with the System integration in two environments--one with the namespace `development`, and one with `production`.

**Goal:** Customize the {ilm-init} policy for the `system.network` data stream in the `production` namespace.
Specifically, apply the built-in `90-days-default` {ilm-init} policy so that data is deleted after 90 days.
Expand Down Expand Up @@ -393,7 +642,7 @@ image::images/create-index-template.png[Create index template]
== Step 4: Roll over the data stream (optional)

To confirm that the data stream is now using the new index template and {ilm-init} policy,
you can either repeat <<data-streams-ilm-one,step one>>, or navigate to **{dev-tools-app}** and run the following:
you can either repeat Step 1, or navigate to **{dev-tools-app}** and run the following:

[source,bash]
----
Expand Down
Binary file added docs/en/ingest-management/images/data-stream-overview.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading