What is the robots.txt file?

Did you know that there's a special file called "robots.txt" located in the root domain ( https://www.searchblox.com/robots.txt )? This file tells web crawlers and other robots which files they're allowed to index or download. It's known as The Robots Exclusion Standard and it's used to prevent these robots from accessing parts of a website that are meant to be private. Search engines will automatically look for this file in the root domain and use it to determine which files to index. If the file isn't present, the search engines will attempt to index everything on the website.

To learn more on Robots.txt file impact in Web Collection, read: Using Robots.txt in SearchBlox