| Webspace & Domain Names This is a discussion on, MSN bot within the Freedom2Surf forum; What's the easiest way to specifically stop the MSN bot msnbot64096.search.msn.com from crawling my site? He's used 300MB of bandwidth ... |
 |
 |
|
LinkBack | Thread Tools | |
17-06-2004, 08:20 AM
|
#1 |
|
Tiscali User Member
Join Date: May 2002
Location: Glasgow, UK
Posts: 876
Thanks: 0
Thanked 0 Times in 0 Posts
|
MSN bot
What's the easiest way to specifically stop the MSN bot msnbot64096.search.msn.com from crawling my site?
He's used 300MB of bandwidth just in this month!!! |
|
    
|
|
17-06-2004, 08:27 AM
|
#2 |
|
Tiscali User Member
Join Date: Jan 2004
Location: Leicester
Posts: 621
Thanks: 0
Thanked 0 Times in 0 Posts
|
I assume you have a robots.txt file to define your policy
 If not just Google robots.txt, and the top few entries give reasonable guides 
__________________
Karl Prince Ex-Freedom2Support Knowledge Team Billion 7402VGP, MAX 50 GB 7360kb sync (ex USR9003 - couldn't cope with MAX) |
|
|
|
|
17-06-2004, 08:59 AM
|
#3 |
|
Tiscali User Member
Join Date: May 2002
Location: Glasgow, UK
Posts: 876
Thanks: 0
Thanked 0 Times in 0 Posts
|
That's fair enough, but when I look at my stats, the aforementioned bot doesn't appear in the list of robots/spiders who have crawled my site.
How can I specifically ban this one? I know normally I'd just do User-agent: botname Disallow: / |
|
|
|
|
17-06-2004, 09:11 AM
|
#4 |
|
Tiscali User Member
Join Date: Jan 2004
Location: Leicester
Posts: 621
Thanks: 0
Thanked 0 Times in 0 Posts
|
It should have the agent name "MSNBot", but it could be a development bot with a different agent name
 The MSN Bot page has quite a bit of useful info, also you can email msnbot@microsoft.com if you have a bandwidth issue. I can also be useful to use the HTML Meta Tags for non Bot specific blocking of sub pages, rather than have a very complicated robots.txt
__________________
Karl Prince Ex-Freedom2Support Knowledge Team Billion 7402VGP, MAX 50 GB 7360kb sync (ex USR9003 - couldn't cope with MAX) |
|
|
|
|
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
| Thread Tools | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| DG 834 Help! (newbie) | Sus | Freedom2Surf Broadband | 6 | 02-11-2005 07:40 AM |
| MSN Messenger and F2S ADSL | Stu | Freedom2Surf Broadband | 7 | 06-07-2003 11:46 AM |
| NEW MSN 6 Really does work with NAT | walmsleyd | Freedom2Surf Broadband | 1 | 02-06-2003 03:53 PM |
| MSN Messenger | RevMK | Freedom2Surf Broadband | 14 | 02-05-2003 12:42 AM |
All times are GMT. The time now is 08:21 AM.
