Yes, SearchBlox comes with an HTTP/HTTPS crawler that browses systematically. The web crawler, also known as a web spider, is an automatic indexer. Crawlers can validate hyperlinks and HTML code.
Related discussions:
Can I restrict the crawler from indexing certain folder/url paths?
How can I see a detailed spider/crawler activity?
Can I customize the settings for the SearchBlox crawler?
Can I set a time delay on the crawler/spider between making requests to a website?
Comments