ActiveMQ Scaler - getting error "invalid character '<' looking for beginning of value"

[frachel]

I set up an activemq scaler. I looked at the scaler code and using the URL it constructs I can successfully curl or use a web browser to get a valid response. But in K8s when I describe the scaledobject I see this:

Warning KEDAScalerFailed 16s (x7 over 3m17s) keda-operator error inspecting ActiveMQ queue size: invalid character '<' looking for beginning of value

Any idea where that's coming from/what's causing it?

[frachel]

As I put in the issue created from this discussion, the scaler DOES work with an older ActiveMQ version (5.10.x).. so at some point something broke.

[zroubalik]

As mention on slack, it is probably some incompatibility introduced in the activemq, would be great if you are able to figure out what is the problem.

[frachel]

I was hoping I'd be able to recreate it quickly using just a plain Go program doing mostly the same things the scaler code does, but that was succesful. The only difference was I didn't use the http.NewRequestWithContext, just NewRequest.

Question, probably unrelated.. I see the scaler reference "defaultCorsHeader".. which for the AMQ scaler isn't defined vs another scaler I found it in (artemis) which has it defined as defaultCorsHeader = "http://%s". Not sure if thats a problem/bug or not?

[zroubalik]

That's strange, I don't understand why the context should make any difference? 🤔

wrt defaultCorsHeader good catch, you are right, this should be a separate property.

[frachel]

I don't know if the context "matters" or not, I was just saying that I wrote a quick command-line Go program to mimic what the scaler does as much as possible, but wasn't sure what the values for the context stuff would be so I used "http.NewRequest" rather than "http.NewRequestWithContext".

I'll see if I can get a real test working with actual Keda, K8s and AMQ 5.17.3 (unless that's somethig you can do, since I have to learn how to build it, modify it, test it, learn some "go", etc).

I'm just looking to spit out the resp.Body that comes back from the REST call to see what it contains.. Since it's finding a "<" I assume its some HTML response that most likely contains some textual error information as to why the call didn't return the JSON Queue data requested.

[zroubalik]

Unit tests are usually just for testing of metadata parsing and what not.

There is e2e test running against ActiveMQ instance in the cluster: https://github.com/kedacore/keda/blob/main/tests/scalers/activemq/activemq_test.go

https://github.com/kedacore/keda/tree/main/tests

[frachel]

I was able to get it so I can build/deploy my own version and start messing around. I'm printing out the contents of the resp.Body now.. this is what the operator is outputting:

*********************
{"request":{"mbean":"org.apache.activemq:brokerName=localhost,destinationName=ALIP.REQUESTQUEUE,destinationType=Queue,type=Broker","attribute":"QueueSize","type":"read"},"value":1,"timestamp":1677618022,"status":200}
*********************
*********************
{"request":{"mbean":"org.apache.activemq:brokerName=localhost,destinationName=ALIP.REQUESTQUEUE,destinationType=Queue,type=Broker","attribute":"QueueSize","type":"read"},"value":1,"timestamp":1677618022,"status":200}
*********************
2023-02-28T21:00:22Z    ERROR   scale_handler   error getting metric for scaler {"scaledObject.Namespace": "innovdev1", "scaledObject.Name": "activemq-scaledobject", "scaler": "activeMQScaler", "error": "error inspecting ActiveMQ queue size: EOF"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics
        /workspace/pkg/scaling/scale_handler.go:485
github.com/kedacore/keda/v2/pkg/metricsservice.(*GrpcServer).GetMetrics
        /workspace/pkg/metricsservice/server.go:48
github.com/kedacore/keda/v2/pkg/metricsservice/api._MetricsService_GetMetrics_Handler
        /workspace/pkg/metricsservice/api/metrics_grpc.pb.go:79
google.golang.org/grpc.(*Server).processUnaryRPC
        /workspace/vendor/google.golang.org/grpc/server.go:1340
google.golang.org/grpc.(*Server).handleStream
        /workspace/vendor/google.golang.org/grpc/server.go:1713

Not sure why it's printing twice? I just added these 4 lines (one line above and below mine for context)

	defer resp.Body.Close()

	// Dump the contents of Body here to see why its failing.
	body, err := ioutil.ReadAll(resp.Body)

	println("*********************")
	println(string(body))
	println("*********************")

	if err := json.NewDecoder(resp.Body).Decode(&monitoringInfo); err != nil {

Since this is a different error than the "<" character one, I want to make sure I'm doing/deploying things correctly before going much further.. in case the EOF error has something to do with my dev/deploy process..

[zroubalik]

Do you see the out twice constantly? This might be inital validation, when it tries to get both status (to decide on the activity - ie scaling 0<->1) and also metrics for HPA (for scaling 1<->N).

[frachel]

Yeah it shows two times consistently throughout the log. I didn't see it one time at all, always pairs of two.

I'm going to do a few simple tests today to make sure I have all my scenarios straight. I'll set up a VM and go through different versions of activemq trying to find exactly what version it stopped working in, then check the changelog to see if they modified anything that seems relevant.

[zroubalik]

@melisatanrverdi FYI ^^

[frachel]

So this now looks more be me being an idiot and having a typo in the YAML. I have it successfuly working with AMQ 5.17.3. I am doing some "long running" tests right now because I did see the "<" error in the operator log "in the middle" of a working test.. but I've isolated myself in case someone else at work was touching anything.

I had the encoded user and password backwards in the YAML, so my guess is the "<" character was from an "unauthorized" HTML response from AMQ.

I'll continue my tests throughout the day, and if it all works I'll close this.

[frachel]

This can be closed. All stemmed from the AMQ Scaler sample YAML having:

data:
  activemq-password: ACTIVEMQ_PASSWORD
  activemq-username: ACTIVEMQ_USERNAME

and me blindly pasting in username first since username always comes first :-)

During my hour-long test I did at one point see the operator spit out the "<" invalid character, but then it started working on the next metric request. There was a timeout error, so something must have happened with our AMQ server to cause a blip but the operator/scaler recovered just fine.