Build hangs for hours in ProducerConsumerExecutor start() method

[chris-laplante]

We are using TeamCity 2021.2.2 with version 2.9.1 of teamcity-artifactory-plugin. On most of our nightly builds, our build hangs for anywhere between 1 and 15 hours. We initially reported the problem to JetBrains here: https://youtrack.jetbrains.com/issue/TW-73001. They analyzed the thread dumps we gave them and determined it was this plugin causing the hang.

Here is the relevant part of the thread-dump:

"TC: 06:23:31 Processing 19 listeners for runnerFinished; 03:00:13 Build {id=22785, buildTypeId='Our_Build_Id', buildTypeInternalId='bt75'}" #8125 prio=5 os_prio=0 cpu=801.05ms elapsed=47748.40s tid=0x00007f4034008000 nid=0x4f5e in Object.wait()  [0x00007f3ff67f1000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(java.base@11.0.13/Native Method)
        - waiting on <no object reference available>
        at java.lang.Thread.join(java.base@11.0.13/Thread.java:1300)
        - waiting to re-lock in wait() <0x000000070236d2b0> (a java.lang.Thread)
        at java.lang.Thread.join(java.base@11.0.13/Thread.java:1375)
        at org.jfrog.build.extractor.producerConsumer.ProducerConsumerExecutor.start(ProducerConsumerExecutor.java:73)
        at org.jfrog.build.extractor.clientConfiguration.util.spec.SpecsHelper.uploadArtifactsBySpec(SpecsHelper.java:111)
        at org.jfrog.build.extractor.clientConfiguration.util.spec.SpecsHelper.uploadArtifactsBySpec(SpecsHelper.java:62)
        at org.jfrog.teamcity.agent.GenericBuildInfoExtractor.appendRunnerSpecificDetails(GenericBuildInfoExtractor.java:66)
        at org.jfrog.teamcity.agent.BaseBuildInfoExtractor.extract(BaseBuildInfoExtractor.java:84)
        at org.jfrog.teamcity.agent.listener.AgentListenerBuildInfoHelper.extractBuildInfo(AgentListenerBuildInfoHelper.java:220)
        at org.jfrog.teamcity.agent.listener.AgentListenerBuildInfoHelper.runnerFinished(AgentListenerBuildInfoHelper.java:150)
        at org.jfrog.teamcity.agent.ArtifactoryAgentListener.runnerFinished(ArtifactoryAgentListener.java:133)
        at jdk.internal.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.13/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.13/Method.java:566)
        at jetbrains.buildServer.util.EventDispatcher$3.run(EventDispatcher.java:138)
        at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
        at jetbrains.buildServer.util.EventDispatcher.dispatch(EventDispatcher.java:132)
        at jetbrains.buildServer.util.EventDispatcher$2.invoke(EventDispatcher.java:82)
        at com.sun.proxy.$Proxy9.runnerFinished(Unknown Source)
        at jetbrains.buildServer.agent.impl.buildStages.runnerStages.finish.FireRunnerFinishedFStage.doFinishStage(FireRunnerFinishedFStage.java:31)
        at jetbrains.buildServer.agent.impl.buildStages.RunnerFinishStagesExecutor.runFinishStage(RunnerFinishStagesExecutor.java:32)
        at jetbrains.buildServer.agent.impl.buildStages.RunnerFinishStagesExecutor.doBuildFinishStages(RunnerFinishStagesExecutor.java:23)
        at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.RunnerContextExecutor.safeCallFinishStages(RunnerContextExecutor.java:65)
        at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.RunnerContextExecutor.callRunnerStages(RunnerContextExecutor.java:56)
        at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.StepExecutor.processNextStep(StepExecutor.java:25)
        at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.executeRunnerStep(ForEachBuildRunnerStage.java:139)
        at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.runStep(ForEachBuildRunnerStage.java:124)
        at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.executeBuildRunners(ForEachBuildRunnerStage.java:84)
        at jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage.doBuildStage(ForEachBuildRunnerStage.java:45)
        at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:33)
        at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:24)
        at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:76)
        at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:35)
        at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor.doStages(BuildStagesExecutor.java:24)
        at jetbrains.buildServer.agent.impl.BuildRunActionImpl.doStages(BuildRunActionImpl.java:81)
        at jetbrains.buildServer.agent.impl.BuildRunActionImpl.runBuild(BuildRunActionImpl.java:57)
        at jetbrains.buildServer.agent.impl.BuildAgentImpl.doActualBuild(BuildAgentImpl.java:332)
        at jetbrains.buildServer.agent.impl.BuildAgentImpl.access$200(BuildAgentImpl.java:58)
        at jetbrains.buildServer.agent.impl.BuildAgentImpl$1.run(BuildAgentImpl.java:287)
        at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)

It appears the ProducerConsumerExecutor.start method is waiting for some threads to join.

In our experience, eventually the method does return. But like I said, sometimes it takes 15 hours.