The Best robots.txt File

A robots.txt file is a way for a site administrator to give instructions to web crawlers (robots) that are crawling the site.

They are not usually anywhere near this funny.

Editorial Director, God King of BuzzFeed
Contact Jack Shepherd at
Got a confidential tip? Submit it here.
Now Buzzing