Yes, you can set a delay on the spider/crawler to avoid hammering a website with continuous requests for pages/data. The delay can be set in milliseconds under the Collection > Settings tab in the field named Spider Delay. This value is set on a per collection basis.
To learn more on HTTP Collection Settings read: HTTP Collection Settings in SearchBlox
Comments
0 comments
Please sign in to leave a comment.