Understand the rulebook for search engines & control how (and which part) search engines crawl & index your site.
Posts TaggedRobots.txt
LBD #059: Mistakes that make robots.txt a disaster
When was the last time you checked your robots.txt? See if you’re making the mistakes mentioned in this post for effecient crawling & indexing.