What is the syntax used by SearchBlox for Allow and Disallow paths?

SearchBlox uses the standard regular expression syntax for the Allow and Disallow paths. 

To allow a domain for crawling and to keep the crawler within the same domain, add the domain in the Allow Paths.

Examples:

 Allow Paths

cnn.com

or 

searchblox.com

 

To prevent the crawler from crawling sections of the website or sub-domains, add the path to the Disallow Paths

Examples:

Disallow Paths

/videos/

or 

/print-only/

 

 

Have more questions? Submit a request

Comments