robots.txt

[kenie]

What's the importance of robots.txt? How does it work?

[Ceres]

Hi kenie, I think you use a robots.txt file if you want to give instructions to web bots regarding how they crawl and index your website. For example, if you don't want bots to access certain pages on your website or you want to block a particular bot, then you'd create a robots.txt file giving these instructions. Unfortunately, many bots ignore instructions contained in a robots.txt file!

For more info, see: Robots.txt Tutorial.

[Jeff]

What Ceres said!

The most common use is to ensure that the major search engines don't index certain parts of your website, either parts that are confidential or contain duplicate content.

You may also want to use it to prevent Archive.org from archiving your website.