Telegraf unable to scrape localhost:8080/metrics/ endpoint with error 406 Not Acceptable

[danlsgiga]

What went wrong?

What happened:
In Telegraf, this line defines a constant value for the Accept header value used by Telegraf's HTTP client. If I do a curl -H "Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited;q=0.7,text/plain;version=0.0.4;q=0.3" http://localhost:8080/metrics/ I get a 406 Not Acceptable error as well from the Oncall HTTP Server, which is the same error I'm seeing in Telegraf's logs as below:

telegraf[10438]: 2024-06-11T20:14:00Z E! [inputs.prometheus] Error in plugin: "http://localhost:8080/metrics/" returned HTTP status "406 Not Acceptable"

And this is the corresponding log from the Oncall HTTP server:

2024-06-11 22:05:10 source=engine:app google_trace_id=none logger=root inbound latency=0.000662 status=406 method=GET path=/metrics/ user_agent=Telegraf/1.31.0 Go/1.22.4 content-length=0 slow=0
2024-06-11 22:05:10 source=engine:app google_trace_id=none logger=django.request Not Acceptable: /metrics/

If I do a curl http://localhost:8080/metrics/ the request succeeds and I can see the oncall related metrics in the response.

What did you expect to happen:
Telegraf to be able to scrape the http://localhost:8080/metrics/ endpoint successfully.

How do we reproduce it?

1. Run the official docker-compose.yml stack with the FEATURE_PROMETHEUS_EXPORTER_ENABLED set to true
2. Curl the http://localhost:8080/metrics/ endpoint using the command in the bug description
3. Verify the HTTP response is 406 Not Acceptable

Grafana OnCall Version

Tested with v1.4.4 and v1.7.0

Product Area

Other

Grafana OnCall Platform?

Docker

User's Browser?

No response

Anything else to add?

No response

[tyrken]

Surprised you're getting a 406 error with v1.7.0 - I'm seeing a 500 error with the same error in the engine's logs as shown in #4455

[joeyorlando]

Hi there @danlsgiga, this is to be expected. The oncall backend is a REST API (application/json content type header).

From the logs you shared, it seems like Telegraf is asking for protobuf, which our backend does not support. I'm not familiar with Telegraf, but maybe there is a way to configure it to work with JSON?

[danlsgiga]

Hey @joeyorlando, thanks for taking a look at this.

The header defined by Telegraf, also includes text/plain, so I think this should be accepted by the /metrics/ endpoint, which responds with a prometheus formatted output that is not json formatted.

$ curl -I -H "Accept: text/plain" http://localhost:8080/metrics/
HTTP/1.1 406 Not Acceptable
Content-Type: application/json
Allow: GET, HEAD, OPTIONS
X-Frame-Options: DENY
Content-Length: 57
Vary: Origin
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Cross-Origin-Opener-Policy: same-origin

$ curl -I -H "Accept: text/plain,application/json" http://localhost:8080/metrics/
HTTP/1.1 200 OK
Content-Type: text/plain; version=0.0.4; charset=utf-8
Allow: GET, HEAD, OPTIONS
X-Frame-Options: DENY
Content-Length: 24027
Vary: Origin
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Cross-Origin-Opener-Policy: same-origin

Note that the successful response comes with Content-Type: text/plain; version=0.0.4; charset=utf-8, so, text/plain should be supported as Accepted.

If localhost:8080/metrics/ is a prometheus compatible endpoint, I think its fair for it to support text/plain. Could you please reconsider supporting this?

[joeyorlando]

which responds with a prometheus formatted output that is not json formatted.

you are right, my apologies. I'll go ahead and re-open this 👍

[danlsgiga]

I found a workaround for now.

By setting http_headers = {"Accept" = "text/plain,application/json"} on telegraf's [[inputs.prometheus]] config I'm able to overcome this issue.