SB 5.1 - OCI - Deploy/Kubernetes/Monitoring failing

[ddevadat]

Env details
cloud provider: OCI
Deploy/Kubernetes/Monitoring
branch_or_tag: release-5.0.0_RC2, tag: all
https://github.com/project-sunbird/sunbird-devops.git

`
TASK [sunbird-monitoring : tempating variables] ********************************
changed: [localhost] => (item=prometheus-operator) => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"ansible_loop_var": "item",
"changed": true,
"checksum": "a04a393cae51e53cb00b41ae9f8bc1298de2d353",
"deprecations": [
{
"msg": "Distribution Ubuntu 18.04 on host localhost should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.8/reference_appendices/interpreter_discovery.html for more information",
"version": "2.12"
}
],
"dest": "/tmp/prometheus-operator.yaml",
"gid": 116,
"group": "jenkins",
"item": "prometheus-operator",
"md5sum": "f3e8ed695db2178f31560ffb9226c189",
"mode": "0644",
"owner": "jenkins",
"size": 32045,
"src": "/var/lib/jenkins/.ansible/tmp/ansible-tmp-1672149642.8538342-25718-272097166978864/source",
"state": "file",
"uid": 112
}
changed: [localhost] => (item=cassandra-jmx-exporter) => {
"ansible_loop_var": "item",
"changed": true,
"checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
"dest": "/tmp/cassandra-jmx-exporter.yaml",
"gid": 116,
"group": "jenkins",
"item": "cassandra-jmx-exporter",
"md5sum": "d41d8cd98f00b204e9800998ecf8427e",
"mode": "0644",
"owner": "jenkins",
"size": 0,
"src": "/var/lib/jenkins/.ansible/tmp/ansible-tmp-1672149643.2313921-25718-31654675673229/source",
"state": "file",
"uid": 112
}
changed: [localhost] => (item=elasticsearch-exporter) => {
"ansible_loop_var": "item",
"changed": true,
"checksum": "9d16095952cab19d81c83fbe113b7417691ac400",
"dest": "/tmp/elasticsearch-exporter.yaml",
"gid": 116,
"group": "jenkins",
"item": "elasticsearch-exporter",
"md5sum": "3a3256cc4be6dce28deb01c18d47bdb4",
"mode": "0644",
"owner": "jenkins",
"size": 269,
"src": "/var/lib/jenkins/.ansible/tmp/ansible-tmp-1672149643.371412-25718-275893959836627/source",
"state": "file",
"uid": 112
}
changed: [localhost] => (item=logger) => {
"ansible_loop_var": "item",
"changed": true,
"checksum": "d6987f9ffda5b6d4188b7ba0f11c4fa8931944e5",
"dest": "/tmp/logger.yaml",
"gid": 116,
"group": "jenkins",
"item": "logger",
"md5sum": "adcbb56f571d949b6d1fd7e25c940a56",
"mode": "0644",
"owner": "jenkins",
"size": 269,
"src": "/var/lib/jenkins/.ansible/tmp/ansible-tmp-1672149643.511086-25718-44592605577193/source",
"state": "file",
"uid": 112
}
changed: [localhost] => (item=blackbox-exporter) => {
"ansible_loop_var": "item",
"changed": true,
"checksum": "2398ae5dab5544fd62d875690e4cd283fcf8a847",
"dest": "/tmp/blackbox-exporter.yaml",
"gid": 116,
"group": "jenkins",
"item": "blackbox-exporter",
"md5sum": "914559d5818d0fadc1f7990e9a93a7e5",
"mode": "0644",
"owner": "jenkins",
"size": 72,
"src": "/var/lib/jenkins/.ansible/tmp/ansible-tmp-1672149643.6490002-25718-218379055288015/source",
"state": "file",
"uid": 112
}
failed: [localhost] (item=additional-scrape-configs) => {
"ansible_loop_var": "item",
"changed": false,
"item": "additional-scrape-configs"
}

MSG:

AnsibleUndefinedVariable: 'dict object' has no attribute 'graylog'

`

Could you please share how to define the group graylog in the ansible inventory host file.?

[beepdot]

You can add a dummy host to get around this error as below

[graylog-1]
0.0.0.0

[graylog:children]
graylog-1

[ddevadat]

After adding this it has gone past the previous failure..but now getting error at

`
TASK [sunbird-monitoring : Creating sunbird monitoring stack] ******************
failed: [localhost] (item=prometheus-operator) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m prometheus-operator /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/prometheus-operator --namespace monitoring -f /tmp/prometheus-operator.yaml",
"delta": "0:00:00.768314",
"end": "2022-12-28 11:33:39.480042",
"item": "prometheus-operator",
"rc": 1,
"start": "2022-12-28 11:33:38.711728"
}

STDOUT:

Release "prometheus-operator" does not exist. Installing it now.

STDERR:

Error: failed to install CRD crds/crd-alertmanager.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"

MSG:

non-zero return code

failed: [localhost] (item=cassandra-jmx-exporter) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m cassandra-jmx-exporter /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/cassandra-jmx-exporter --namespace monitoring -f /tmp/cassandra-jmx-exporter.yaml",
"delta": "0:00:00.925786",
"end": "2022-12-28 11:33:40.482206",
"item": "cassandra-jmx-exporter",
"rc": 1,
"start": "2022-12-28 11:33:39.556420"
}

STDOUT:

Release "cassandra-jmx-exporter" does not exist. Installing it now.

STDERR:

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

MSG:

non-zero return code

failed: [localhost] (item=elasticsearch-exporter) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m elasticsearch-exporter /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/elasticsearch-exporter --namespace monitoring -f /tmp/elasticsearch-exporter.yaml",
"delta": "0:00:00.874363",
"end": "2022-12-28 11:33:41.433735",
"item": "elasticsearch-exporter",
"rc": 1,
"start": "2022-12-28 11:33:40.559372"
}

STDOUT:

Release "elasticsearch-exporter" does not exist. Installing it now.

STDERR:

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

MSG:

non-zero return code

failed: [localhost] (item=logger) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m logger /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/logger --namespace monitoring -f /tmp/logger.yaml",
"delta": "0:00:00.990615",
"end": "2022-12-28 11:33:42.501837",
"item": "logger",
"rc": 1,
"start": "2022-12-28 11:33:41.511222"
}

STDOUT:

Release "logger" does not exist. Installing it now.

STDERR:

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

MSG:

non-zero return code

changed: [localhost] => (item=blackbox-exporter) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m blackbox-exporter /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/blackbox-exporter --namespace monitoring -f /tmp/blackbox-exporter.yaml",
"delta": "0:00:02.908480",
"end": "2022-12-28 11:33:45.487116",
"item": "blackbox-exporter",
"rc": 0,
"start": "2022-12-28 11:33:42.578636"
}

STDOUT:

Release "blackbox-exporter" has been upgraded. Happy Helming!
NAME: blackbox-exporter
LAST DEPLOYED: Wed Dec 28 11:33:43 2022
NAMESPACE: monitoring
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
See https://github.com/prometheus/blackbox_exporter/ for how to configure Prometheus and the Blackbox Exporter.

failed: [localhost] (item=additional-scrape-configs) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m additional-scrape-configs /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/additional-scrape-configs --namespace monitoring -f /tmp/additional-scrape-configs.yaml",
"delta": "0:00:00.870153",
"end": "2022-12-28 11:33:46.433888",
"item": "additional-scrape-configs",
"rc": 1,
"start": "2022-12-28 11:33:45.563735"
}

STDOUT:

Release "additional-scrape-configs" does not exist. Installing it now.

STDERR:

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

MSG:

non-zero return code

failed: [localhost] (item=alertrules) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m alertrules /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/alertrules --namespace monitoring -f /tmp/alertrules.yaml",
"delta": "0:00:00.919373",
"end": "2022-12-28 11:33:47.430401",
"item": "alertrules",
"rc": 1,
"start": "2022-12-28 11:33:46.511028"
}

STDOUT:

Release "alertrules" does not exist. Installing it now.

STDERR:

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"

MSG:

non-zero return code

failed: [localhost] (item=kafka-topic-exporter) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m kafka-topic-exporter /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/kafka-topic-exporter --namespace monitoring -f /tmp/kafka-topic-exporter.yaml",
"delta": "0:00:00.920674",
"end": "2022-12-28 11:33:48.428149",
"item": "kafka-topic-exporter",
"rc": 1,
"start": "2022-12-28 11:33:47.507475"
}

STDOUT:

Release "kafka-topic-exporter" does not exist. Installing it now.

STDERR:

Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]

MSG:

non-zero return code

failed: [localhost] (item=prometheus-redis-exporter) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m prometheus-redis-exporter /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/prometheus-redis-exporter --namespace monitoring -f /tmp/prometheus-redis-exporter.yaml",
"delta": "0:00:00.979701",
"end": "2022-12-28 11:33:49.487569",
"item": "prometheus-redis-exporter",
"rc": 1,
"start": "2022-12-28 11:33:48.507868"
}

STDOUT:

Release "prometheus-redis-exporter" does not exist. Installing it now.

STDERR:

Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Role" in version "rbac.authorization.k8s.io/v1beta1", unable to recognize "": no matches for kind "RoleBinding" in version "rbac.authorization.k8s.io/v1beta1"]

MSG:

non-zero return code

failed: [localhost] (item=processing-kafka-exporter) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m processing-kafka-exporter /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/processing-kafka-exporter --namespace monitoring -f /tmp/processing-kafka-exporter.yaml",
"delta": "0:00:00.903988",
"end": "2022-12-28 11:33:50.467715",
"item": "processing-kafka-exporter",
"rc": 1,
"start": "2022-12-28 11:33:49.563727"
}

STDOUT:

Release "processing-kafka-exporter" does not exist. Installing it now.

STDERR:

Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]

MSG:

non-zero return code

failed: [localhost] (item=json-path-exporter) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m json-path-exporter /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/json-path-exporter --namespace monitoring -f /tmp/json-path-exporter.yaml",
"delta": "0:00:00.896022",
"end": "2022-12-28 11:33:51.441526",
"item": "json-path-exporter",
"rc": 1,
"start": "2022-12-28 11:33:50.545504"
}

STDOUT:

Release "json-path-exporter" does not exist. Installing it now.

STDERR:

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

MSG:

non-zero return code

changed: [localhost] => (item=oauth2-proxy) => {
"ansible_loop_var": "item",
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m oauth2-proxy /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/oauth2-proxy --namespace monitoring -f /tmp/oauth2-proxy.yaml",
"delta": "0:00:02.960558",
"end": "2022-12-28 11:33:54.482778",
"item": "oauth2-proxy",
"rc": 0,
"start": "2022-12-28 11:33:51.522220"
}

STDOUT:

Release "oauth2-proxy" has been upgraded. Happy Helming!
NAME: oauth2-proxy
LAST DEPLOYED: Wed Dec 28 11:33:52 2022
NAMESPACE: monitoring
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
To verify that oauth2-proxy has started, run:

kubectl --namespace=monitoring get pods -l "app=oauth2-proxy"

PLAY RECAP *********************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

`

[beepdot]

The current helm chart for monitoring doesn't work on Kubernetes cluster version > 1.21. If you are running a higher version of Kubernetes, you will need to make changes to some of the chart files to ensure the API is supported. Here is the error you need to look at

 no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"

[ddevadat]

does any other deployment job depends on Deploy/K8S/Monitoring?
May be we can park it for now and proceed with the rest of the jobs

[beepdot]

The flink jobs from KP and DP have service monitor enabled during deployment. You will need to disable service monitor flag before deploying them.
https://github.com/project-sunbird/sunbird-learning-platform/blob/release-5.1.0/kubernetes/helm_charts/datapipeline_jobs/values.j2#L8-L9

[gandham-santhosh]

@ddevadat can you please mark this thread if it is resolved?

[ddevadat]

@gandham-santhosh does it mean we cant enabled service monitoring as of today?
I have disabled it and deployed the jobs which works fine.

[gandham-santhosh]

@ddevadat we made changes to release-5.1.0 branch. Can you please use this branch to deploy the monitoring and let us know?
@santhosh-tg

[Krishnaj20]

@ddevadat - let us know if your query is now resolved?

@chitranshu-keshav

[ddevadat]

i am working on this. Will update soon

[ddevadat]

i tried with release-5.1.0 and job failed with the below error

failed: [localhost] (item=alertrules) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "helm upgrade --install --atomic --timeout 10m alertrules /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Monitoring/kubernetes/helm_charts/monitoring/alertrules --namespace monitoring -f /tmp/alertrules.yaml",
    "delta": "0:00:11.024822",
    "end": "2023-02-15 16:08:12.947847",
    "item": "alertrules",
    "rc": 1,
    "start": "2023-02-15 16:08:01.923025"
}

STDOUT:

Release "alertrules" does not exist. Installing it now.


STDERR:

Error: release alertrules failed, and has been uninstalled due to atomic being set: Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "[https://sunbird-monitoring-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s](https://sunbird-monitoring-operator.monitoring.svc/admission-prometheusrules/mutate?timeout=10s)": dial tcp 10.0.66.6:8443: i/o timeout


MSG:

non-zero return code

The status of all pods

jenkins@jenkins:~$ k get pods
NAME                                                              READY   STATUS             RESTARTS        AGE
alertmanager-sunbird-monitoring-alertmanager-0                    1/2     CrashLoopBackOff   5 (92s ago)     4m48s
blackbox-exporter-prometheus-blackbox-exporter-55787d76d6-447sc   2/2     Running            0               49d
cassandra-jmx-exporter-56c47b968f-2nnhq                           1/1     Running            0               4m13s
elasticsearch-exporter-dc67c4f7f-x6m4c                            1/1     Running            0               3m58s
json-path-exporter-6c4f5ff5f9-z5xls                               1/1     Running            0               2m14s
kafka-topic-exporter-68d997678-ctnt9                              1/1     Running            0               2m53s
logger-elasticsearch-exporter-697f885dd8-wzd72                    1/1     Running            0               3m35s
oauth2-proxy-6bbf96c86b-wnxld                                     1/1     Running            0               49d
processing-kafka-exporter-8c679dc66-gqqhn                         1/1     Running            0               2m27s
prometheus-operator-grafana-7cff8bc954-p6c56                      2/2     Running            0               5m3s
prometheus-operator-kube-state-metrics-7684b99876-k8rxn           1/1     Running            0               5m3s
prometheus-operator-prometheus-node-exporter-662fb                1/1     Running            0               5m3s
prometheus-operator-prometheus-node-exporter-b4gcj                1/1     Running            0               5m3s
prometheus-operator-prometheus-node-exporter-b9frs                1/1     Running            0               5m3s
prometheus-operator-prometheus-node-exporter-j7v5f                1/1     Running            0               5m3s
prometheus-sunbird-monitoring-prometheus-0                        4/4     Running            1 (2m49s ago)   3m9s
redis-exporter-7bc65999d9-qcrhp                                   1/1     Running            0               2m40s
sunbird-monitoring-operator-546694c984-l2hz4                      2/2     Running            0               5m3s

[santhosh-tg]

@ddevadat Please check if sunbird-monitoring-admission is created
kubectl get mutatingwebhookconfigurations -n monitoring

If yes, try to re-deploy once.

[ddevadat]

@ddevadat Please check if sunbird-monitoring-admission is created kubectl get mutatingwebhookconfigurations -n monitoring

If yes, try to re-deploy once.
mutatingwebhookconfigurations was present
redeployed k8s/monitoring job and it was successful.

Does it mean we have to deploy twice. What could be the issue here

[santhosh-tg]

I'm not able to replicate the issue, from the error (https://sunbird-monitoring-operator.monitoring.svc/admission-prometheusrules/mutate?timeout=10s)": dial tcp 10.0.66.6:8443: i/o timeout it looks like the URL didn't respond within 10 seconds.
If the issue persists we can consider increasing the timeout.

[ddevadat]

@gandham-santhosh the problem here is that when we have two k8s cluster ( core and flink)
We are deploying k8s/monitoring in the core cluster.
But when we enable service_monitor_enabled: true for flinkjobs, its still failing with errors like

Error: UPGRADE FAILED: unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

[ddevadat]

how will the flink cluster know about the crd ServiceMonitor

[santhosh-tg]

@ddevadat Please run Deploy/DataPipeline/Monitoring
so the monitoring stack is deployed in separate Flink cluster as well.

Also, I hope the Deploy/DataPipeline/BootstrapMinimal job is already ran. If not, please run that first before that.

[ddevadat]

the branch instruction is not given in the documentation for deploy/dp/monitoring
is it also release-5.1.0

[ddevadat]

tried deploying from release-5.1.0 but it giving the below kind of errors

failed: [localhost] (item=prometheus-operator) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "helm upgrade --install prometheus-operator /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/Monitoring/kubernetes/helm_charts/monitoring/prometheus-operator --namespace monitoring -f /tmp/dp_prometheus-operator.yaml",
    "delta": "0:00:01.717653",
    "end": "2023-03-20 13:57:58.421282",
    "item": "prometheus-operator",
    "rc": 1,
    "start": "2023-03-20 13:57:56.703629"
}

STDOUT:

Release "prometheus-operator" does not exist. Installing it now.


STDERR:

Error: failed to install CRD crds/crd-alertmanager.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"


MSG:

non-zero return code

failed: [localhost] (item=alertrules) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "helm upgrade --install alertrules /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/Monitoring/kubernetes/helm_charts/monitoring/alertrules --namespace monitoring -f /tmp/dp_alertrules.yaml",
    "delta": "0:00:02.617174",
    "end": "2023-03-20 13:58:01.105534",
    "item": "alertrules",
    "rc": 1,
    "start": "2023-03-20 13:57:58.488360"
}

STDOUT:

Release "alertrules" does not exist. Installing it now.


STDERR:

Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"


MSG:

non-zero return code

failed: [localhost] (item=prometheus-adapter) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "helm upgrade --install prometheus-adapter /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/Monitoring/kubernetes/helm_charts/monitoring/prometheus-adapter --namespace monitoring -f /tmp/dp_prometheus-adapter.yaml",
    "delta": "0:00:01.567653",
    "end": "2023-03-20 13:58:02.738354",
    "item": "prometheus-adapter",
    "rc": 1,
    "start": "2023-03-20 13:58:01.170701"
}

STDOUT:

Release "prometheus-adapter" does not exist. Installing it now.
WARNING: This chart is deprecated


STDERR:

coalesce.go:199: warning: destination for custom is a table. Ignoring non-table value []
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "APIService" in version "apiregistration.k8s.io/v1beta1"

[santhosh-tg]

Okay, this issue was fixed in sunbird-devops repo. We also need to make the updates in DP repo as well https://github.com/Sunbird-Obsrv/sunbird-data-pipeline. We will update you here once that is done.

[ddevadat]

@santhosh-tg i tried running this from
https://github.com/santhosh-tg/sunbird-data-pipeline/tree/release-5.1.0-monitoring-fix

But getting the below error

failed: [localhost] (item=alertrules) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "helm upgrade --install alertrules /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/Monitoring/kubernetes/helm_charts/monitoring/alertrules --namespace monitoring -f /tmp/dp_alertrules.yaml",
    "delta": "0:00:03.384555",
    "end": "2023-03-28 05:57:31.382842",
    "item": "alertrules",
    "rc": 1,
    "start": "2023-03-28 05:57:27.998287"
}

STDOUT:

Release "alertrules" does not exist. Installing it now.


STDERR:

Error: Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "[https://sunbird-monitoring-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s](https://sunbird-monitoring-operator.monitoring.svc/admission-prometheusrules/mutate?timeout=10s)": EOF


MSG:

non-zero return code

failed: [localhost] (item=prometheus-adapter) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "helm upgrade --install prometheus-adapter /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/Monitoring/kubernetes/helm_charts/monitoring/prometheus-adapter --namespace monitoring -f /tmp/dp_prometheus-adapter.yaml",
    "delta": "0:00:01.442445",
    "end": "2023-03-28 05:57:32.947676",
    "item": "prometheus-adapter",
    "rc": 1,
    "start": "2023-03-28 05:57:31.505231"
}

STDOUT:

Release "prometheus-adapter" does not exist. Installing it now.
WARNING: This chart is deprecated


STDERR:

coalesce.go:199: warning: destination for custom is a table. Ignoring non-table value []
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "APIService" in version "apiregistration.k8s.io/v1beta1"


MSG:

non-zero return code

failed: [localhost] (item=processing-kafka-exporter) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "helm upgrade --install processing-kafka-exporter /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/Monitoring/kubernetes/helm_charts/monitoring/processing-kafka-exporter --namespace monitoring -f /tmp/dp_processing-kafka-exporter.yaml",
    "delta": "0:00:02.433066",
    "end": "2023-03-28 05:57:35.496886",
    "item": "processing-kafka-exporter",
    "rc": 1,
    "start": "2023-03-28 05:57:33.063820"
}

STDOUT:

Release "processing-kafka-exporter" does not exist. Installing it now.


STDERR:

Error: Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "[https://sunbird-monitoring-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s](https://sunbird-monitoring-operator.monitoring.svc/admission-prometheusrules/mutate?timeout=10s)": EOF


MSG:

non-zero return code

[santhosh-tg]

@ddevadat this looks like the similar error we had with webhook last time ? Did you try to re-deploy once ?

[ddevadat]

@santhosh-tg

Still errored

ailed: [localhost] (item=alertrules) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "helm upgrade --install alertrules /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/Monitoring/kubernetes/helm_charts/monitoring/alertrules --namespace monitoring -f /tmp/dp_alertrules.yaml",
    "delta": "0:00:01.082031",
    "end": "2023-03-28 10:59:45.660543",
    "item": "alertrules",
    "rc": 1,
    "start": "2023-03-28 10:59:44.578512"
}

STDERR:

Error: UPGRADE FAILED: "alertrules" has no deployed releases


MSG:

non-zero return code

failed: [localhost] (item=prometheus-adapter) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "helm upgrade --install prometheus-adapter /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/Monitoring/kubernetes/helm_charts/monitoring/prometheus-adapter --namespace monitoring -f /tmp/dp_prometheus-adapter.yaml",
    "delta": "0:00:01.381986",
    "end": "2023-03-28 10:59:47.160134",
    "item": "prometheus-adapter",
    "rc": 1,
    "start": "2023-03-28 10:59:45.778148"
}

STDOUT:

Release "prometheus-adapter" does not exist. Installing it now.
WARNING: This chart is deprecated


STDERR:

coalesce.go:199: warning: destination for custom is a table. Ignoring non-table value []
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "APIService" in version "apiregistration.k8s.io/v1beta1"


MSG:

non-zero return code

failed: [localhost] (item=processing-kafka-exporter) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "helm upgrade --install processing-kafka-exporter /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/Monitoring/kubernetes/helm_charts/monitoring/processing-kafka-exporter --namespace monitoring -f /tmp/dp_processing-kafka-exporter.yaml",
    "delta": "0:00:01.300820",
    "end": "2023-03-28 10:59:48.574916",
    "item": "processing-kafka-exporter",
    "rc": 1,
    "start": "2023-03-28 10:59:47.274096"
}

STDERR:

Error: UPGRADE FAILED: "processing-kafka-exporter" has no deployed releases

[santhosh-tg]

@ddevadat I pushed some changes. Can you redeploy and check pls ?

[ddevadat]

its failing for the below

failed: [localhost] (item=alertrules) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "helm upgrade --install alertrules /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/Monitoring/kubernetes/helm_charts/monitoring/alertrules --namespace monitoring -f /tmp/dp_alertrules.yaml",
    "delta": "0:00:00.998505",
    "end": "2023-03-28 17:23:06.042012",
    "item": "alertrules",
    "rc": 1,
    "start": "2023-03-28 17:23:05.043507"
}

STDERR:

Error: UPGRADE FAILED: "alertrules" has no deployed releases


MSG:

non-zero return code

failed: [localhost] (item=processing-kafka-exporter) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": "helm upgrade --install processing-kafka-exporter /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/Monitoring/kubernetes/helm_charts/monitoring/processing-kafka-exporter --namespace monitoring -f /tmp/dp_processing-kafka-exporter.yaml",
    "delta": "0:00:00.973923",
    "end": "2023-03-28 17:23:09.866371",
    "item": "processing-kafka-exporter",
    "rc": 1,
    "start": "2023-03-28 17:23:08.892448"
}

STDERR:

Error: UPGRADE FAILED: "processing-kafka-exporter" has no deployed releases


MSG:

non-zero return code

[ddevadat]

here are the list of pods

ddevadat@ddevadat-mac .ansible % k get pods
NAME                                                      READY   STATUS    RESTARTS        AGE
alertmanager-sunbird-monitoring-alertmanager-0            2/2     Running   0               11h
processing-kafka-exporter-7dd89bb79c-cw7w4                1/1     Running   0               11h
prometheus-adapter-7cff6645d4-tcwtg                       1/1     Running   0               2m39s
prometheus-operator-grafana-6bb59b8ccb-glnl9              2/2     Running   0               11h
prometheus-operator-kube-state-metrics-7684b99876-56jf8   1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-4qxw9        1/1     Running   1 (142m ago)    11h
prometheus-operator-prometheus-node-exporter-8fwcq        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-8sghj        1/1     Running   1 (8h ago)      11h
prometheus-operator-prometheus-node-exporter-8tntb        1/1     Running   1 (3h33m ago)   11h
prometheus-operator-prometheus-node-exporter-95cxh        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-99spt        1/1     Running   3 (3h23m ago)   11h
prometheus-operator-prometheus-node-exporter-9flg8        1/1     Running   5 (4h10m ago)   11h
prometheus-operator-prometheus-node-exporter-9pk4b        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-d7dfq        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-dkmz5        1/1     Running   2 (174m ago)    11h
prometheus-operator-prometheus-node-exporter-dqwld        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-gklv7        1/1     Running   3 (123m ago)    11h
prometheus-operator-prometheus-node-exporter-gxx6l        1/1     Running   3 (141m ago)    11h
prometheus-operator-prometheus-node-exporter-hdsst        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-m9lnn        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-mmhfw        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-n6fdh        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-pjgxj        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-q9cdr        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-tpxw2        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-tr6mj        1/1     Running   2 (141m ago)    11h
prometheus-operator-prometheus-node-exporter-vfs7l        1/1     Running   1 (5h39m ago)   11h
prometheus-operator-prometheus-node-exporter-wr47m        1/1     Running   2 (10h ago)     11h
prometheus-operator-prometheus-node-exporter-x2m67        1/1     Running   0               11h
prometheus-operator-prometheus-node-exporter-zzf8x        1/1     Running   3 (3h28m ago)   11h
prometheus-sunbird-monitoring-prometheus-0                3/3     Running   1 (11h ago)     11h
sunbird-monitoring-operator-69f6c9574b-zbzzs              2/2     Running   0               11h

[ddevadat]

ddevadat@ddevadat-mac .ansible % kubectl get mutatingwebhookconfigurations -n monitoring
NAME                           WEBHOOKS   AGE
sunbird-monitoring-admission   1          11h

[ddevadat]

ddevadat@ddevadat-mac .ansible % helm list
NAME                     	NAMESPACE 	REVISION	UPDATED                                  	STATUS  	CHART                          	APP VERSION
alertrules               	monitoring	1       	2023-03-28 05:57:29.200376003 +0530 +0530	failed  	alertrules-0.1.0               	1.16.0
processing-kafka-exporter	monitoring	1       	2023-03-28 05:57:34.272826279 +0530 +0530	failed  	processing-kafka-exporter-1.0.0	1.0
prometheus-adapter       	monitoring	2       	2023-03-28 17:23:07.543965279 +0530 +0530	deployed	prometheus-adapter-2.5.1       	v0.7.0
prometheus-operator      	monitoring	4       	2023-03-28 17:22:41.742930941 +0530 +0530	deployed	prometheus-operator-8.13.12    	0.38.1

[santhosh-tg]

did you happen to cancel the deploy job while the helm install was running ? in any case, can you try to uninstall alertrules and processing-kafka-exporter and deploy again ?

helm uninstall alertrules

[ddevadat]

uninstall helped..job is successfull now. I still run from https://github.com/santhosh-tg/sunbird-data-pipeline/tree/release-5.1.0-monitoring-fix

Also do we need to run the Deploy/DP/Logging ?

[santhosh-tg]

Also do we need to run the Deploy/DP/Logging ?

Yes, please.

uninstall helped..job is successfull now. I still run from https://github.com/santhosh-tg/sunbird-data-pipeline/tree/release-5.1.0-monitoring-fix

@anandp504 @manjudr @gandham-santhosh Please help to merge this PR

[ddevadat]

@santhosh-tg i couldnt run deploy/DP/logging with the same branch as above

ERROR: /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/Logging@script/1b841a5466287e309870246dc8c755b55212aa98524a71d742c91d65ee6633f3/kubernetes/pipelines/logging/Jenkinsfile not found
Finished: FAILURE

[santhosh-tg]

@ddevadat Logging is deployed from sunbird-devops repo. So, please use the same branch that you used for Kubernetes/Logging