How to control an "Unknown Robot" from wasting Bandwidth?

IT.com

fourwings

Banned
Hi,

I have a website businesslawn.com and i have installed bookmarking script - pligg in that, from past few weeks an unknown robot crawls the site more than 2000 times per day and eats my band width.

I dont know how to block this unknown robot, If any one knows Pls help me.

Thanks
 
Last edited by a moderator:
I don't know too much about this area, but have you tried disallowing the bot via a robots.txt file? For example, to exclude a specific robot, you need to insert the following code:

User-agent: [insert name of bot]
Disallow: /

Read more instructions here

The problem is some bots will ignore instructions contained in a robots.txt file. If that happens, another option is as follows:

If the bad robot operates from a single IP address, you can block its access to your web server through server configuration or with a network firewall.
 
Put a statcounter / free counter to first identify if the robot is coming from one IP or is from multiple IP's
If you have already done that, and if suggestion by Ceres doesn't help (some bots are obnoxious and don't know the rules to follow robot.txt) - time to get into .htaccess.
You can give a block / redirect using .htaccess and prevent bot from spidering your site. To achieve that - you would have to identify the IP though - thus need for stat counter / anything that would help to isolate.
 
Yes it is true, it doesnt obeys robots.txt - i have to add stats counter to find the ip address.

I have tried with htaccess also, but not clear with that.

Now it is clear - find ip address via stats counter and block or redirect via htaccess

Thanks to ceres and pubdomains.in
 
If you identify the IP address, simply add deny directive, i.e. assuming the IP address is 192.168.1.1 than put the following as first line
Code:
order allow,deny
deny from 192.168.1.1
allow from all

That should do the trick.
 

whois



Forums dedicated to Indian domain names, including buying, selling, appraising, developing, and monetizing.

About Us

Threads
29,388
Messages
76,792
Members
7,945
Latest member
nilamburfurniture
Top Bottom