I keep a simple robots.txt and I rarely change it, mostly because I’m no pro SEO expert, I know a fair bit about it, I can often get sites to rank high but I don’t spend my days tweaking sites for search engines, I spend my time tweaking for the end user.

SEOBook on the other hand is a pro SEO kinda guy and he does spend all his time optimizing for search engines, that his business, but with one change he’s managed to cut himself out of $10,000 in profits because of an edit of this robots.txt.

One little wildcard in the file has dumped heaps of his profitable pages into Google’s supplemental index, you may remember JohnChow did something similar a while back, he liked to screw with his robots.txt and he ended up in the supplemental index to, now his whole site has dropped from Google pretty much.

Beware when editing your robots.txt, if you need to edit it, do your research first. The best guide I’ve found for SEO and robots.txt is Dan Thies SEO optimization guide, it’s an in-depth 100 page PDF, skip to page 93 to read about robots.txt. Actually it’s the only good eBook I’ve ever read, it’s not self serving crap like most eBooks people publish to try and get popular, it’s an actual helpful guide.