Deploy> DataPipeline > KafkaSetup The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'ingestion-cluster-zookeeper'

[daveanvit]

Jenkins: Ubuntu 18.04 LTS, 4vCPU, 16GB RAM, 250 GB HDD
Github Release Tag: release-4.6.0_RC3
Jenkins Logs:

Started by user [admin](http://34.138.204.52:8080/user/admin)
Checking out git https://github.com/project-sunbird/sunbird-data-pipeline.git into /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup@script/a80ca29e77e6368749e3841598b5da252dc9038cc0296719aea4f05fefffba20 to read pipelines/deploy/kafka/Jenkinsfile
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup@script/a80ca29e77e6368749e3841598b5da252dc9038cc0296719aea4f05fefffba20/.git # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/project-sunbird/sunbird-data-pipeline.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-data-pipeline.git
 > git --version # timeout=10
 > git --version # 'git version 2.37.3'
 > git fetch --tags --force --progress --depth=1 -- https://github.com/project-sunbird/sunbird-data-pipeline.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse origin/release-4.6.0_RC3^{commit} # timeout=10
 > git rev-parse release-4.6.0_RC3^{commit} # timeout=10
Checking out Revision 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac (release-4.6.0_RC3)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac # timeout=10
Commit message: "Merge pull request #1511 from kumarks1122/release-4.6.0-content-rating-job-fixes"
 > git rev-list --no-walk 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac # timeout=10
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
 > git --version # timeout=10
 > git --version # 'git version 2.37.3'
 > git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup@libs/deploy-conf/.git # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
 > git --version # timeout=10
 > git --version # 'git version 2.37.3'
 > git fetch --no-tags --force --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/*:refs/remotes/origin/* refs/heads/shared-lib # timeout=10
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
 > git rev-list --no-walk 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] node
Running on [Jenkins](http://34.138.204.52:8080/computer/(built-in)/) in /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] echo
Found .git folder. Clearing it..
[Pipeline] sh
+ git clean -fxd
Removing private@tmp/
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/.git # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/project-sunbird/sunbird-data-pipeline.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-data-pipeline.git
 > git --version # timeout=10
 > git --version # 'git version 2.37.3'
 > git fetch --tags --force --progress --depth=1 -- https://github.com/project-sunbird/sunbird-data-pipeline.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse origin/release-4.6.0_RC3^{commit} # timeout=10
 > git rev-parse release-4.6.0_RC3^{commit} # timeout=10
Checking out Revision 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac (release-4.6.0_RC3)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6c0b35f8f75cfb1b1cf7e96f8ffcdfb781c82bac # timeout=10
Commit message: "Merge pull request #1511 from kumarks1122/release-4.6.0-content-rating-job-fixes"
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (deploy)
[Pipeline] sh
+ echo Deploy/dev/DataPipeline/KafkaSetup
[Pipeline] sh
+ echo Deploy/dev/DataPipeline/KafkaSetup
[Pipeline] sh
+ echo Deploy/dev/DataPipeline/KafkaSetup
[Pipeline] sh
+ pwd
[Pipeline] echo
{currentWs=/var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup, env=dev, module=DataPipeline, jobName=KafkaSetup, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/ansible/dp_kafka_setup.yml, ansibleExtraArgs=--vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{currentWs=/var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup, env=dev, module=DataPipeline, jobName=KafkaSetup, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/ansible/dp_kafka_setup.yml, ansibleExtraArgs=--vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is enabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-cred
 > git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/private/.git # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/daveanvit/searce-diksha-sunbird.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/daveanvit/searce-diksha-sunbird.git
 > git --version # timeout=10
 > git --version # 'git version 2.37.3'
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --force --progress --depth=1 -- https://github.com/daveanvit/searce-diksha-sunbird.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse origin/main^{commit} # timeout=10
Checking out Revision d3866aa62da9b6d757885e293293295526a8d1d3 (origin/main)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d3866aa62da9b6d757885e293293295526a8d1d3 # timeout=10
Commit message: "Update common.yml"
 > git rev-list --no-walk 833d61fc75e22d945dbcbe47a2cc058a001d6833 # timeout=10
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh
+ rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/private/ansible/inventory/dev/DataPipeline/common.yml /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/private/ansible/inventory/dev/DataPipeline/hosts /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/private/ansible/inventory/dev/DataPipeline/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/ansible/inventory/env/
+ [ -f /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/ansible/inventory/env/kubernetes.yaml ]
+ ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/ansible/dp_kafka_setup.yml --vault-password-file /var/lib/jenkins/secrets/vault-pass
[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details

[WARNING]:  * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/DataPipelin
e/KafkaSetup/ansible/inventory/env/hosts with yaml plugin: Syntax Error while
loading YAML.   did not find expected <document start>  The error appears to be
in '/var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/ansible/inven
tory/env/hosts': line 3, column 1, but may be elsewhere in the file depending
on the exact syntax problem.  The offending line appears to be:  # Replace the
value deployer to your server login user name, for example ubuntu
ansible_ssh_user=deployer ^ here

[WARNING]:  * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/DataPipelin
e/KafkaSetup/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/work
space/Deploy/dev/DataPipeline/KafkaSetup/ansible/inventory/env/hosts:10:
Section [keycloak:children] includes undefined group: keycloak-1

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/dev/DataPipeline/K
afkaSetup/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'


PLAY [processing-cluster-kafka] ************************************************

TASK [Gathering Facts] *********************************************************
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
ok: [192.170.0.5]

TASK [Registering kafka_id] ****************************************************
ok: [192.170.0.5]

TASK [Print kafka_id] **********************************************************
ok: [192.170.0.5] => {
    "kafka_id": "1"
}

TASK [setup-kafka : create topics] *********************************************
fatal: [192.170.0.5]: FAILED! => {}

MSG:

The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'ingestion-cluster-zookeeper'

The error appears to be in '/var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/ansible/roles/setup-kafka/tasks/main.yml': line 1, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: create topics
  ^ here


...ignoring

TASK [setup-kafka : override retention time] ***********************************
fatal: [192.170.0.5]: FAILED! => {}

MSG:

The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'ingestion-cluster-zookeeper'

The error appears to be in '/var/lib/jenkins/workspace/Deploy/dev/DataPipeline/KafkaSetup/ansible/roles/setup-kafka/tasks/main.yml': line 9, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: override retention time
  ^ here



PLAY RECAP *********************************************************************
192.170.0.5                : ok=4    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=1   

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh
+ git rev-parse --short HEAD
[Pipeline] sh
+ cut -d / -f1
+ rev
+ git name-rev --name-only HEAD
+ rev
[Pipeline] sh
+ echo Deploy/dev/DataPipeline/KafkaSetup
[Pipeline] echo
Could not find env specific Slack channel. Check for global slack channel..
[Pipeline] echo
Could not find slack environment variable. Skipping slack notification..
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh
+ echo Deploy/dev/DataPipeline/KafkaSetup
[Pipeline] echo
Could not find env specific email group. Check for global email group..
[Pipeline] echo
Could not find global email group variable. Skipping email notification..
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[beepdot]

Please define this host entry in your private repo - https://github.com/project-sunbird/sunbird-devops/blob/master/private_repo/ansible/inventory/dev/DataPipeline/hosts#L81-L82

[daveanvit]

worked thanks @keshavprasadms