[BUGS] Current implementation of concurrency is causing errors

[pradhanhitesh]

The current implementation of concurrency using Promise is causing errors such as Error checking robots.txt for {URL}: fetch failed which is likely due to multiple concurrent requests and causing Skipping {URL}: Not allowed by robots.txt Attaching a screenshot of the error. Upon manual inspection of failed cases, I have found that robots.txt does exist.

Upon doing some investigations, I believe that limiting the number of requests in the checkRobotsAndEnqueue function using const limit = pLimit(10); have yielded better results and less number of failed cases. @in-c0 I will be adding PR soon.