fix(cu): add pending evals metric for alerts #1105
Conversation
servers/cu/src/bootstrap.js
Outdated
| labelNames: ['type'] | ||
| }) | ||
|
|
||
| pendingEvaluationCounter.set({ type: 'primary' }, primaryWorkQueue.size) |
There was a problem hiding this comment.
This will set the metric amount when this file is run at startup, then never again. We will need to move the sets to the location that the work queues are added to (evaluateWith, maybe?) and increment it by 1 on addition and decrement by 1 on completion.
set does not currently exist on the grafana counters. Only inc is implemented (see effects/metrics.js, counterWith). This can be implemented with inc / dec or feel free to implement set in metrics.js - up to you.
There was a problem hiding this comment.
Got it, will adjust this accordingly.
There was a problem hiding this comment.
@jfrain99 Seems there is a way to listen to events on the PQueue and accordingly inc, dec the pendingEvaluationsCounter given those events take place. I've added this accordingly. Not so familiar with JS, so let me know if I need to change further.
Added these for grafana alerting purposes.