Rate Exceeded Error

[DanDubinsky]

We've been using the Code Deploy Jenkins plug in for about a year with no trouble. Then about 2 weeks ago we started getting errors like this about every other build:

Failed CodeDeploy post-build step; exception follows.
Rate exceeded (Service: AmazonCodeDeploy; Status Code: 400; Error Code: ThrottlingException;

The deploys actually succeeds, but Jenkins reports it as a failure. Amazon support says the problem is with the GetDeployment API, the polling that the plug in does to see if the deployment is successful. They say our Jenkins server is sending 41 of those per minute with higher bursts. They say the bursts are causing the error. We have the polling interval set to 5 seconds.

Anybody have any ideas? Seems to me the problem must be on the Amazon side since we haven't changed anything on the Jenkins server in months. Amazon is insisting that the problem is with Jenkins.

[rohkat-aws]

@DanDubinsky can you give more details? Like When it started happening exactly and do you see any errors logged in the agent?

[DanDubinsky]

First error was May 7, 2018 around 4PM Eastern Time. We've been getting the error about every other build since then. The error in the Jenkins console was:

Rate exceeded (Service: AmazonCodeDeploy; Status Code: 400; Error Code: ThrottlingException; Request ID: d494fea1-52c1-11e8-9335-bdce7ca12b64)
com.amazonaws.services.codedeploy.model.AmazonCodeDeployException: Rate exceeded (Service: AmazonCodeDeploy; Status Code: 400; Error Code: ThrottlingException; Request ID: d494fea1-52c1-11e8-9335-bdce7ca12b64)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1545)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1183)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)
at com.amazonaws.services.codedeploy.AmazonCodeDeployClient.doInvoke(AmazonCodeDeployClient.java:2517)
at com.amazonaws.services.codedeploy.AmazonCodeDeployClient.invoke(AmazonCodeDeployClient.java:2493)
at com.amazonaws.services.codedeploy.AmazonCodeDeployClient.getDeployment(AmazonCodeDeployClient.java:1495)
at com.amazonaws.codedeploy.AWSCodeDeployPublisher.waitForDeployment(AWSCodeDeployPublisher.java:459)
at com.amazonaws.codedeploy.AWSCodeDeployPublisher.perform(AWSCodeDeployPublisher.java:239)
at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:32)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:720)
at hudson.model.Build$BuildExecution.post2(Build.java:185)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:665)
at hudson.model.Run.execute(Run.java:1745)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:404)

[annamataws]

Hi,

We have been working on improving efficiency of our internal calls. Is this still affecting you? If so, can you please add information about how your Jenkins box is set up?

Thank you