The Robots Exclusion Protocol (REP)

I’m currently spending some time in the BA lounge at Tacoma airport doing some follow-ups on various SMX notes. I stumbled upon a note regarding robots.txt and the Robots Exclusion Protocol (REP) which I quickly want to share with all my fellow readers.

During the Bot Herding sessions (really great one, thanks Rand) Priyank Garg from Yahoo! mentioned that they finally came up with an official documentation covering everything related to the REP. All major search engines (especially Google, Yahoo! and MSN) support this de facto standard which was already introduced in the early 90’s. They did quite some “paperwork” bringing this up so don’t miss to check it out: Robots Exclusion Protocol for Yahoo!, Google and Microsoft. Same documents are also available on Google and MSN.