Not detecting termination on Rancher v1.5.x

[vkruoso]

I'm using the catalog entry to have hosts deactivated when a spot is marked to be terminated. The script seems to not be able to detect the situation. Right now I'm using rancher reaper, but that solution may cause some downtime because the host removal happens only when the AWS host is in the "Terminated" status.

If you can point the way to get more information about the process I would be happy to help.

[chrisurwin]

Hey, the way it works is it looks at the aws metadata to see if it has been marked for termination, it then temporarily increases the node code of the ASG it's running in by 1 so that a new instance is initiated. It will then evacuate itself and remove itself from the Rancher server before it is terminated. What appears not to be working?

[vkruoso]

Thanks for your answer @chrisurwin. Ii looks like I probably misunderstood the goal of the tool. Looking at the code I don't see any reference to ASGs. My understanding was that if the termination is detected, it would simply evacuate the host on rancher before the instance is terminated. To my use case I really don't need it to increase any ASGs count because we are using Spot/Ec2 Fleets.

Besides that, when an instance is marked to terminate I did not see any log messages like, so Im assuming it was not able to detect it. Maybe the meta-data is not indicating a termination mark because the termination is handled by the spot fleet (image below)?

[chrisurwin]

Yeah, this was written to work in conjunction with the autospotting catalog item that I published. It would be pretty simple to update the code to deal with your scenario.

[vkruoso]

Can you point the direction? I might be able to get a PR to resolve this.

[chrisurwin]

Sorry, looked through the code and this version doesn't have the ASG stuff in so it basically checks for termination time and if it exists then it will evacuate the host. the termination check is in the agent.go code and just looks for a termination time which AWS populate once the instance is set to terminate. shouldn't matter that it's a fleet as this is just looking for the metadata flag per host and I don't think they handle them differently. It may be that AWS have changed the termination notification, I've not looked at this code in a while. If they haven't it should work.

[vkruoso]

As per their docs looks like it still the same endpoint to get the termination information.

Just checked that it works with the spot fleet as expected. Next will try to simulate in rancher.