Tech

The Best robots.txt File

A robots.txt file is a way for a site administrator to give instructions to web crawlers (robots) that are crawling the site. They are not usually anywhere near this funny.

I am BuzzFeed's Editorial Director and Beastmaster. I live in Brooklyn with my cat and my regrets.
Contact Jack Shepherd at jack@buzzfeed.com
 
 
More News
Now Buzzing