Does SearchBlox come with a HTTP/HTTPS crawler/spider?

Yes, SearchBlox comes with a HTTP/HTTPS crawler that browses systematically. The web crawler, also known as a web spider, is an automatic indexer. Crawlers can validate hyperlinks and HTML code.

Related discussions:

Can I restrict the crawler from indexing certain folder/url paths?

How can I see a detailed spider/crawler activity?

 

Can I customize the settings for the SearchBlox crawler?

Can I set a time delay on the crawler/spider between making requests to a website?

 

Have more questions? Submit a request

Comments