Can I restrict the crawler from indexing certain folder/URL paths?

Blake Cohen

August 10, 2020 09:23
Updated

Yes, SearchBlox's collection filters let you add disallow paths under Collections > Paths > Disallow Paths.

For example, /cgi-bin/ would be a pattern to disallow the collection from having any documents which have /cgi-bin/ in the URL/directory path. The Disallow Path uses regular expressions, but you can create complex patterns by understanding the regular expression engine.

To learn more on HTTP Collection settings read: HTTP Collection Settings in SearchBlox

To learn more on File Collection settings read: File Collection Settings in SearchBlox