How to control an "Unknown Robot" from wasting Bandwidth?

[fourwings]

Hi,

I have a website businesslawn.com and i have installed bookmarking script - pligg in that, from past few weeks an unknown robot crawls the site more than 2000 times per day and eats my band width.

I dont know how to block this unknown robot, If any one knows Pls help me.

Thanks

[Ceres]

I don't know too much about this area, but have you tried disallowing the bot via a robots.txt file? For example, to exclude a specific robot, you need to insert the following code:

User-agent: [insert name of bot]
Disallow: /

Read more instructions here

The problem is some bots will ignore instructions contained in a robots.txt file. If that happens, another option is as follows:

If the bad robot operates from a single IP address, you can block its access to your web server through server configuration or with a network firewall.

[pubdomains.in]

Put a statcounter / free counter to first identify if the robot is coming from one IP or is from multiple IP's
If you have already done that, and if suggestion by Ceres doesn't help (some bots are obnoxious and don't know the rules to follow robot.txt) - time to get into .htaccess.
You can give a block / redirect using .htaccess and prevent bot from spidering your site. To achieve that - you would have to identify the IP though - thus need for stat counter / anything that would help to isolate.

[fourwings]

Yes it is true, it doesnt obeys robots.txt - i have to add stats counter to find the ip address.

I have tried with htaccess also, but not clear with that.

Now it is clear - find ip address via stats counter and block or redirect via htaccess

Thanks to ceres and pubdomains.in

[pubdomains.in]

If you identify the IP address, simply add deny directive, i.e. assuming the IP address is 192.168.1.1 than put the following as first line

order allow,deny
deny from 192.168.1.1
allow from all

That should do the trick.

[fourwings]

Thanks,

Now only i have added stats counter to my website to detect ip address which eats my BW

Thanks