Can I restrict the crawler from indexing certain folder/URL paths?

Yes, SearchBlox's collection filters let you add disallow paths under Collections > Paths > Disallow Paths.

For example, /cgi-bin/.* would be a pattern to disallow the collection from having any documents which have /cgi-bin/ in the URL/directory path. The Disallow Path uses regular expressions, but you can create complex patterns by understanding the regular expression engine.

Have more questions? Submit a request