Setting disable_runner_autoupdate parameter has no effect

[cmckeen]

Hello,

We have been running into an issue recently with our ephemeral runners. The behavior we were seeing was that a runner being spun up with an old version of the Github runner agent would update as soon as it was launched, and if this was in response to a scale-up lambda run, a race condition would occur that would prevent the triggering workflow from ever running since the agent update would take place first.

We thought to resolve this problem by using the disable_runner_autoupdate parameter as described here:

disable_runner_autoupdate: "Disable the auto update of the github runner agent. Be aware there is a grace period of 30 days, see also the [GitHub article](https://github.blog/changelog/2022-02-01-github-actions-self-hosted-runners-can-now-disable-automatic-updates/)"

Reference

This however does not seem to be working - we have seen the same behavior recently when our runners were configured to use an image with v2.318.0 of the Github runner agent, and the launched runners upgraded to v2.319.0 after launch.

I confirmed that the environment variable DISABLE_RUNNER_AUTOUPDATE is set to true on the scale-up lambda function, and in the opt/actions-runner/_diag/ directory on the launched runner, there were two log files - one each corresponding with v2.318.0 and v2.319.0, which I would not expect to see if this were functioning as intended. The scale-up Lambda function seems to try to add the parameter to the config here, but I haven't been able to tell from debug logs what the final configuration looks like that is used for the runner.

Below is our abbreviated configuration:

module "multi-runner" {
  source              = "philips-labs/github-runner/aws//modules/multi-runner"
  version             = "5.12.2"
  multi_runner_config = {
    "linux-x64" = {
      matcherConfig = {
        labelMatchers = [["self-hosted", "aws", "linux", "x64", "prod"]]
        exactMatch    = true
      }
      runner_config = {
        delay_webhook_event             = 0
        runner_os                       = "linux"
        runner_architecture             = "x64"
        instance_types                  = ["t3.2xlarge"]
        disable_runner_autoupdate       = true
        enable_ephemeral_runners        = true
        enable_organization_runners     = true
        enable_ssm_on_runners           = true
        enable_userdata                 = true
        instance_target_capacity_type   = "on-demand"
        ...
       }
    }
}

[winwinashwin]

@cmckeen Could you please point me to the module version you are using?

I haven't been able to tell from debug logs what the final configuration looks like that is used for the runner

The final command line options passed to the runner binary is stored in SSM parameter store. The parameter name will be similar to .../tokens/<ec2-instance-id>. Could you please check if the --disableupdate flag is present in the parameter value for any of your ephemeral instances?

By looking at the codepath, seems like the config options are only generated for non-ephemeral runners. The branching is happening here. The --disableupdate config option is only set in the codepath of function createRegistrationTokenConfig which is not called for ephemeral runners with jitconfig enabled.