robots.txt

[kenie]

What's the importance of robots.txt? How does it work?

 

[Ceres]

Hi kenie, I think you use a robots.txt file if you want to give instructions to web bots regarding how they crawl and index your website. For example, if you don't want bots to access certain pages on your website or you want to block a particular bot, then you'd create a robots.txt file giving these instructions. Unfortunately, many bots ignore instructions contained in a robots.txt file!

For more info, see: Robots.txt Tutorial.

 

[Jeff]

What Ceres said!

The most common use is to ensure that the major search engines don't index certain parts of your website, either parts that are confidential or contain duplicate content.

You may also want to use it to prevent Archive.org from archiving your website.

 

[Steve Smith]

Robots.txt file is used when you have duplicate content that you don't want to be indexed, spammers use them to scan for email address. It is placed in the main directory.
​

 

[bhavya]

Hi,
Robots.txt is used when several pages of your site need not be crawled by Google.The blog here contains detailed use of robots.txt and also usage of it.
Kindly refer: How To Use Robots.Txt File?

Thanks.