What Ceres said!
The most common use is to ensure that the major search engines don't index certain parts of your website, either parts that are confidential or contain duplicate content.
You may also want to use it to prevent Archive.org from archiving your website.