The right robots.txt settings for allowing SharePoint to crawl your site
If you want you want to allow SharePoint 2010 or 2013 to crawl your web site add the following to your robots.txt file.
User-agent: MS Search 6.0 Robot
Disallow:
Even though the crawler sends Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot) as the user agent string, this is not what you should check against. Logical…. nah, but it is what it is.
Total cost to figure this out: 6h
Reference: The SharePoint Server crawler ignored directives in Robots.txt
