Can I restrict the crawler from indexing certain folder/URL paths?

Yes, SearchBlox's collection filters let you add disallow paths under Collections > Paths > Disallow Paths.

For example, /cgi-bin/.* would be a pattern to disallow the collection from having any documents which have /cgi-bin/ in the URL/directory path. The Disallow Path uses regular expressions, but you can create complex patterns by understanding the regular expression engine.



To learn more about SearchBlox please visit our developer document site https://developer.searchblox.com/docs

Have more questions? Submit a request

Comments