Can SearchBlox spider through a proxy server?

Yes, SearchBlox supports spidering through a proxy server. A web spider (aka crawler), is simply a web crawler that systematically browses through indexing. You can set the proxy URL, username, and password under Collections > Settings in the Admin menu. Also, you can set a different web proxy for each HTTP collection.

Related Topics:

Can I restrict the crawler from indexing certain folder/URL paths?

How can I see detailed spider/crawler activity?

Can I customize the settings for the SearchBlox crawler?

Can I set a time delay on the crawler/spider between making requests to a website?

To learn more about SearchBlox please visit our developer document site

Have more questions? Submit a request