Robots.txt Template for Community Server
I was getting annoyed by MSNBot trying to index /reader/default.aspx. So, I decided to deploy a Robots.txt file. In case anyone wants a template for a Community Server Robots.txt file, here it is. I don't know what the Downloads folder is used for (other than what I found at
this thread
), so I've included it anyway.
# Robots.txt file for http://domain
#User-agent: *Disallow: /aspnet_client/Disallow: /ControlPanel/Disallow: /Downloads/Disallow: /Emoticons/Disallow: /FreeTextBox3/Disallow: /Ink/Disallow: /Installer/Disallow: /Languages/Disallow: /Messages/Disallow: /reader/Disallow: /Themes/Disallow: /User/Disallow: /Utility/