server should monitor and report events

[ssfrr]

we have a graphite instance running to track basic system stats (hard drive space CPU usage, etc.), and a statsd server so it's easy to pump new data in. I'd like to start keeping track of the rate new data is coming in, error rates, etc.

[kkleidal]

This seems like a good task for a "topics" network flow: whenever a request is made/data is posted, we could call a method which writes the event details to a websockets (or ZMQ) stream with a tag. Then, we could have an analyzer microservice monitoring the stream and logging as necessary.

[ssfrr]

I think it's even easier than that. We already have the graphite instance up and receiving data, so all we need to do us use a statsd client from python to pump metrics in.

Statsd works with a tagging system sort of like what you're describing so you can label your metrics, but we can just instrument the code so that we send the event signal when the things we're tracking happen. It sends over UDP so it doesn't really add overhead even if the receiving server goes down.