Hubot overloading Slack API calls while running on PCF

[brehma100]

We have a Hubot app on our PCF and it appears once a few commands are made via Slack it will work fine the first few times, but then something happens and it starts spamming the Slack API to the point of where Slack disconnects the bot.

If I run this locally, the issue has not presented itself, so it appears to be a PCF/Hubot issue.

I have attached copies of the main script (called dsli.txt) , the function that seems to break it the most (help.txt), and log (dsli.log). I have changed the .coffee extensions to .txt for purposes of attaching them to this post. I also changed IP and site address for security reasons.

dsli.txt
help.txt
dsli.log

Row 1 shows a GET 404 0 143 error, followed be several more 404 type errors. I cannot figure out where they are originating from, especially the "OPTIONS", but if what I read is correct, I don't need to worry about the "OPTIONS" 404. Row 24 seems to be a known issue and does not have a fix. Please let me know if I'm wrong on that. Row 31 does not make sense to me either and does seem to prelude the issue, so is that the cause? But that doesn't make sense to me that it would be the cause for spamming the API. Finally, row 53 is the start of the "Rate limited, will retry..." issue.

So far the "fix" is to restart the app, but then it only works for a handful of commands before it goes haywire again.

Other services or general info that might be useful to know:
Connected to Redis
App AutoScaler
T-Logger
Nodejs Buildpack

Any help you can offer is appreciated, thank you.

[joeyguerra]

I'm wondering if PCF is killing and re deploying the instance?

[brehma100]

I'm wondering if PCF is killing and re deploying the instance?

To the best of my knowledge, the tails do not indicate anything like that in the log file.

[nguyeabcd55544]

We have a Hubot app on our PCF and it appears once a few commands are made via Slack it will work fine the first few times, but then something happens and it starts spamming the Slack API to the point of where Slack disconnects the bot.

If I run this locally, the issue has not presented itself, so it appears to be a PCF/Hubot issue.

I have attached copies of the main script (called dsli.txt) , the function that seems to break it the most (help.txt), and log (dsli.log). I have changed the .coffee extensions to .txt for purposes of attaching them to this post. I also changed IP and site address for security reasons.

dsli.txt
help.txt
dsli.log

Row 1 shows a GET 404 0 143 error, followed be several more 404 type errors. I cannot figure out where they are originating from, especially the "OPTIONS", but if what I read is correct, I don't need to worry about the "OPTIONS" 404. Row 24 seems to be a known issue and does not have a fix. Please let me know if I'm wrong on that. Row 31 does not make sense to me either and does seem to prelude the issue, so is that the cause? But that doesn't make sense to me that it would be the cause for spamming the API. Finally, row 53 is the start of the "Rate limited, will retry..." issue.

So far the "fix" is to restart the app, but then it only works for a handful of commands before it goes haywire again.

Other services or general info that might be useful to know:
Connected to Redis
App AutoScaler
T-Logger
Nodejs Buildpack

Any help you can offer is appreciated, thank you.

[joeyguerra]

I believe Cloud Foundry makes those initial requests to the App instance as part of its lifecycle. Since the app is a NodeJS app and not a Spring Boot app, it doesn't handle those URIs.

I think the warning in line 31 is because the code is doing exports.help = (robot) -> instead of exports = (robot) ->. But I haven't tried to prove that with running code. I suggest you try that change first and run it to see.

I think I ran into the retry error when I inadvertently started multiple instances of Hubot. My guess is that App AutoScaler is the actual cause of the problem. Try not using that and just start a single (1) instance of Hubot.