Yes, the following preferences can be customized for the crawler. Log in to Searchblox Admin. Parameters can be found under the Collections > Settings Admin tab when you use the HTTP-based collection.
The following parameters can be set in HTTP Collection settings:
- Description source selection
- Lastmodified date
- Max document age
- Max document size
- User Agent Name
- Spider Delay (in milli secs)
- Referrer
- Spider Depth
- Follow Robots
- Follow Sitemaps
- Follow Redirects
- HTTP Basic Authentication information (user/password)
- Form Authentication (Form URL, Name/Value, Form Action)
- HTTP Proxy Server Settings
- Boosting
- Removal of Duplicates
- Stemming
- Spelling Suggestions
- Enable Logging
To learn more on HTTP Collections read: HTTP Collection in SearchBlox
Related Topics:
Can I restrict the crawler from indexing certain folder/url paths?
Can the SearchBlox crawler access documents over HTTPS?
How can I see detailed spider/crawler activity?
Can I set a time delay on the crawler/spider between making requests to a website?
Comments
0 comments
Please sign in to leave a comment.