Can I set a time delay on the crawler/spider between making requests to a website?

Yes, you can set a delay on the spider/crawler to avoid hammering a website with continuous requests for pages/data. The delay can be set in milliseconds under the Collection > Settings tab. This value is set on a per collection basis. 

Have more questions? Submit a request

Comments