-
Notifications
You must be signed in to change notification settings - Fork 151
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
test: Makefile target, CI and documentation for alert unit tests (#1355)
* Adding makefile and scripts for prometheus unit-tests * Changes from PR review * Update prometheus-params * Change extract-alerts to associative array + githubCI fix * Update to github CI * Update check_alert_tests script to use ALERT_SEVERITY * Update make test-alert command and remove debug echos * Update name for Checkout step github workflow * chore: remove need for extract_alerts.sh Make excels at tracking generated output files from specified input files (it might be the only thing it's good at). By having a convention of naming the generated rule files and their corresponding unit test files based on the file names in the ConfigMap, then we don't need a separate script to track the difference in what the files are called inside and outside the ConfigMap, which would need to be updated each time a new unit test file is added. With this change, as long as you name your unit test file in the appropriate way, and there's a corresponding entry in the ConfigMap, then the rule file should get extracted in the expected way. * chore: fix yq in Makefile (no need to be so PHONY) * Update to Makefile - var assigment * Reverting YQ target change --------- Co-authored-by: Gerard Ryan <git@grdryn.xyz>
- Loading branch information
Showing
17 changed files
with
148 additions
and
43 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| name: Run prometheus unit tests | ||
| on: | ||
| pull_request: | ||
| paths: | ||
| - 'config/monitoring/prometheus/**' | ||
| - 'tests/prometheus_unit_tests/**' | ||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
| - name: Setup Go | ||
| uses: actions/setup-go@v4 | ||
| with: | ||
| go-version-file: go.mod | ||
| - name: Install Promtool | ||
| run: | | ||
| sudo apt-get update && sudo apt-get install -y prometheus | ||
| - name: Run prometheus-unit-tests | ||
| run : make test-alerts |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -62,4 +62,3 @@ local.mk | |
|
|
||
| # Ignore temporary files created by the Makefile | ||
| *.mktmp.* | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # Ignore temporary alert yaml files created by the Makefile | ||
| *.rules.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| #!/bin/bash | ||
|
|
||
| PROMETHEUS_CONFIG_YAML=$1 | ||
| UNIT_TEST_DIR=$2 | ||
| ALERT_SEVERITY=$3 | ||
|
|
||
| # Collect all alerts from the configuration file | ||
| while IFS= read -r ALERT; do | ||
| ALL_ALERTS+=("$ALERT") | ||
| done < <(yq -N e '.data[] | ||
| | from_yaml | ||
| | .groups[].rules[] | ||
| | select(.alert != "DeadManSnitch" and .labels.severity == "'${ALERT_SEVERITY}'") | ||
| | .alert' "${PROMETHEUS_CONFIG_YAML}") | ||
|
|
||
| # Collect all alerts from the unit test files | ||
| while IFS= read -r ALERT; do | ||
| PROMETHEUS_UNIT_TEST_CHECK+=("$ALERT") | ||
| done < <( | ||
| for alert in "$UNIT_TEST_DIR"/*.yaml; do | ||
| yq -N eval-all '.tests[] | ||
| | .alert_rule_test[] | ||
| | .exp_alerts[] | ||
| | .exp_labels | ||
| | select(.severity == "'${ALERT_SEVERITY}'") | ||
| | .alertname' "$alert" | ||
| done | ||
| ) | ||
|
|
||
| # Sorting the PROMETHEUS_UNIT_TEST_CHECK array for comparison | ||
| PROMETHEUS_UNIT_TEST_CHECK_SORTED=($(echo "${PROMETHEUS_UNIT_TEST_CHECK[@]}" | sort | uniq)) | ||
|
|
||
| # Finding items in ALL_ALERTS not in PROMETHEUS_UNIT_TEST_CHECK_SORTED | ||
| ALERTS_WITHOUT_UNIT_TESTS=() | ||
| for ALERT in "${ALL_ALERTS[@]}"; do | ||
| if [[ ! " ${PROMETHEUS_UNIT_TEST_CHECK_SORTED[@]} " =~ " ${ALERT} " ]]; then | ||
| ALERTS_WITHOUT_UNIT_TESTS+=("$ALERT") | ||
| fi | ||
| done | ||
|
|
||
| # Printing the alerts without unit tests | ||
| echo "Alerts without unit tests:" | ||
| for ALERT in "${ALERTS_WITHOUT_UNIT_TESTS[@]}"; do | ||
| echo "$ALERT" | ||
| done |
Oops, something went wrong.