From 12e23a4c888f1050e479f170bc81aa33d5ddc9bd Mon Sep 17 00:00:00 2001 From: Krystle Salazar Date: Wed, 15 Jan 2025 15:07:55 -0400 Subject: [PATCH] Update instructions for Elasticsearch node replacements --- .../meta/maintenance/elasticsearch_cluster.md | 21 +++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/documentation/meta/maintenance/elasticsearch_cluster.md b/documentation/meta/maintenance/elasticsearch_cluster.md index f3e7b96d2b9..b6901c45edc 100644 --- a/documentation/meta/maintenance/elasticsearch_cluster.md +++ b/documentation/meta/maintenance/elasticsearch_cluster.md @@ -354,6 +354,9 @@ In the steps below, `` refers to a subfolder in the `environments/` directory, and can be one of "dev" or "prod". This is different from `` which is part of the ES module name and can be one of "staging" or "production". + +Consider pausing the `_elasticsearch_cluster_healthcheck` to avoid +getting redundant alerts during this process. ``` 1. Find the instance in the AWS management console using the ID from the @@ -381,9 +384,13 @@ or "production". with this change in production. Apply this change. The plan should include: - creation of a new Elasticsearch data node - - creation of new alarms and changes to existing alarms + - creation of new alarms and changes to existing alarms (for production) - addition of the new instance's IP to the Route53 record + ```bash + just tf apply -target='module.staging-elasticsearch-8-8-2' + ``` + 4. Wait for a new instance to be provisioned by Terraform. Record the public IPv4 DNS and Terraform index of the new instance, they will @@ -394,7 +401,7 @@ or "production". pass the public IPv4 DNS step from step 3 to the `-l`/`--limit` flag. ```bash - just ansible/playbook elasticsearch/sync_config.yml -e apply=true -l ` + just ansible/playbook elasticsearch/sync_config.yml -e apply=true -l ``` ```{note} @@ -413,7 +420,7 @@ or "production". `http://localhost:9220`. ```{tip} - Use the Elasticvue extension or app because the web interface cannot connect + Use the Elasticvue browser extension or app because the web interface cannot connect to Elasticsearch due to CORS protection. ``` @@ -422,7 +429,9 @@ or "production". ```json { - "transient.cluster.routing.allocation.exclude.name": "" + "transient": { + "cluster.routing.allocation.exclude.name": "" + } } ``` @@ -458,6 +467,10 @@ or "production". - subtraction of the retired instance's IP from the Route53 record - destruction of extra alarms and changes to existing alarms + ```bash + just tf apply -target='module.staging-elasticsearch-8-8-2' + ``` + 14. If the deletion of the individual alarms fails due to them being part of a composite alarm, go into the AWS management console and remove the alarms that are supposed to be deleted from any composite alarms that they are a