Warzone 2100 Forums

is there a reason for all these unwarranted bots?

Registered users: Baidubot, Bingbot, Googlebot,, Magpiebot, MJ12bot, ,, Yandexbot

i have removed the what appears as humans from the list: ie: myself and safetyoff.

and here again...

Registered users: , Baidubot,, Googlebot, , , Sougoubot

They are all web crawlers , a.k.a spider bots. Pretty harmless. Used by search engines like google to index websites to improve searh results
http://en.wikipedia.org/wiki/Web_crawler

Unless Eggplant's AI bots are taking over the world

Everyone wants to spy on every tiny bit of info on the net I think some of the bots are useful but a good many do not serve a useful purpose to wz2100 It is a drain on the system and annoying when they start posting useless links if allowed

Your the moderator slain the beasts

Berg wrote:Everyone wants to spy on every tiny bit of info on the net I think some of the bots are useful but a good many do not serve a useful purpose to wz2100 It is a drain on the system and annoying when they start posting useless links if allowed

Your the moderator slain the beasts

if i was to do that, the hue and cry would be deafening. we have to put it up for a vote don't you know.

i understand some bots, that i recognize, but most of these are not.

I dont normaly vote when there is a poll about these things cause the bots will find out what your thinking and work up a profile on you...all very under cover and all ...just ban them from the forum then we can have a vote

Berg wrote:I dont normaly vote when there is a poll about these things cause the bots will find out what your thinking and work up a profile on you...all very under cover and all ...just ban them from the forum then we can have a vote

DiiD, diid. so , there it is.

to be honest some are used for good but some are used for evil and I think

Users browsing this forum: Googlebot and 0 guests


that bot is following me

Berg wrote:to be honest some are used for good but some are used for evil and I think

Users browsing this forum: Googlebot and 0 guests


that bot is following me

Have you tried crossing a river to lose it's scent?

They have been there always, they just got names now and don't hide between the guests. Are you more uncomfortable now knowing that they're there? Also they're very sneaky, they don't produce any noticeable server load.

bingbot crashed my company website last week - it doesn't limit the number of open connections it makes to a site. We ended up adding it to robots.txt in the end to block it from our site.

still hasn't answered the first question. why are they here in the first place? i understand one or two are needed... but that does not explain what the rest are doing.

http://www.google.com/feedfetcher.html
http://www.google.com/bot.html
http://www.bing.com/bingbot.htm
http://www.baidu.com/search/spider.html
http://yandex.com/bots
http://help.soso.com/webspider.htm
http://www.sogou.com/docs/help/webmasters.htm#07
http://spinn3r.com/robot
http://www.brandwatch.com/how-it-works/gathering-data/
http://www.majestic12.co.uk/projects/ds ... j12bot.php
http://discoveryengine.com/discobot.html
http://www.entireweb.com/about/search_t ... dy_spider/

They want information, they don't share crawling results (however at least Google sells own results for other companies), except of those who "steal" information from others (quite easy to do...), so they all generate traffic.

And yes, horribly written bots are usually just like DDoS attempt.

They have left forever!


*15 seconds later on General discussion:Nyan Cat - by: Googlebot