8484 Boston Project v 1.0
Acoon-Robot 4.0.2.17 (http://www.acoon.de)
AdsBot-Google ( http://www.google.com/adsbot.html)
AdsBot-Google (+http://www.google.com/adsbot.html)
Agent Anonymous/3G bot 1.0 (anything...anything you want.; http://anything.you.want; anything@you.want)
AnotherBot 1.0
Apexoo Spider 1.1
Attentio/Nutch-0.9-dev (Attentio\'s beta blog crawler; www.attentio.com; info@attentio.com)
Azureus 3.0.3.4;Mac OS X;Java 1.5.0_13
Baiduspider ( http://www.baidu.com/search/spider.htm)
Baiduspider ( http://www.baidu.com/search/spider_jp.html)
Baiduspider+(+http://www.baidu.com/search/spider.htm)
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Balihoo/Nutch-0.9 (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
Bigsearch.ca/Nutch-1.0-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com)
blackspider
Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
boitho.com-dc/0.82 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.85 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.86 ( http://www.boitho.com/dcbot.html )
bot/1.0 (bot; http://; bot@bot.bot)
BrightCrawler (http://www.brightcloud.com/brightcrawler.asp)
Canookle/1.1 ( http://www.canookle.ca/spider.html)
CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)
CazoodleBot/Nutch-0.9-dev (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)
CazoodleBot/Nutch-0.9-dev (CazoodleBot Crawler; http://www.cazoodle.com; mqbot@cazoodle.com)
CCBot/1.0 ( http://www.commoncrawl.org/bot.html)
CentiverseBot
CentiverseBot - investigator
CentiverseBot/3.0 (http://www.centiverse-project.net)
ContactBot/0.2
ConveraCrawler/0.9e ( http://www.authoritativeweb.com/crawl)
ConveraCrawler/2.0 ( http://ews.converasearch.com/crawl.htm)
DiGi-RSSBot
disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com)
DNSRight.com WebBot Link Ckeck Tool. Report abuse to: dnsr@dnsright.com
Dot TK robot -- spidering your website. Rename your website at www.dot.tk
DotTK-Robot/1.0 (Rename The Internet - http://www.dot.tk)
DuckDuckBot/1.0; ( http://duckduckgo.com/duckduckbot.html)
Dzesqclj qmjpbot nsexc
e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)
EnaBot/1.2 (http://www.enaball.com/crawler.html)
Eurobot/Nutch-1.0-dev (1.0)
FAST Enterprise Crawler 6 used by fast (jorgent@fast)
FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg)
FAST MetaWeb Crawler (helpdesk at fastsearch dot com)
favorstarbot/1.0 ( http://favorstar.com/bot.html)
Feedfetcher-Google; ( http://www.google.com/feedfetcher.html)
Feedfetcher-Google; ( http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=12476056983529841647)
Feedfetcher-Google; ( http://www.google.com/feedfetcher.html; 2 subscribers; feed-id=12476056983529841647)
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=12476056983529841647)
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 2 subscribers; feed-id=12476056983529841647)
froGgle/0.0 BETA! ( http://www.jabea.net/bot.php)
Gaisbot/3.0 (robot-response@widelab.org; http://www.widelab.org/robot.html)
Gaisbot/3.0 (robot06@gais.cs.ccu.edu.tw; http://gais.cs.ccu.edu.tw/robot.php)
Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
genieBot enash@genieknows.com
GeonaBot/1.2; http://www.geona.com/
Gigabot/1.0
Gigabot/2.0 (http://www.gigablast.com/spider.html)
Gigabot/2.0att
Gigabot/3.0 (http://www.gigablast.com/spider.html)
Google-Sitemaps/1.0
googlebot
Googlebot-Image/1.0
Googlebot/2.0 (http://www.googlebot.com/bot.html)
Googlebot/2.1
Googlebot/2.1 ( http://www.google.com/bot.html)
Googlebot/2.1 ( http://www.googlebot.com/bot.html)
Googlebot/2.1 ( http://www.googlebot.com/bot.html; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; TheWorld)
Googlebot/2.1 (http://www.googlebot.com/bot.html)
great-plains-web-spider/gpws (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com)
gsa-crawler (Enterprise; S4-EHN4YSQ2XLJAB; gsasymbiosys@gmail.com,xeonbox4@gmail.com)
GurujiBot/1.0 ( http://www.guruji.com/en/WebmasterFAQ.html)
GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
gzip(gfe) (via translate.google.com)
hdgjacfbqnupyrjavaqrngnxptq
HMSE_Robot
Holiday Shopping, Inc.
Hotakew perljwsi nudlg
HouxouCrawler/Nutch-0.8.2-dev http://www.houxou.com/crawler; crawler at houxou dot com)
HouxouCrawler/Nutch-0.9 
HouxouCrawler/Nutch-0.9 
I411Crawling/Nutch-0.9 (CrawlData; www.i411.com; sbehera@i144.com)
iaskspider/2.0( http://iask.com/help/help_index.html)
Ibotyze riucktv dmjy
ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp)
ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp)
ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr at iiit dot ac dot in)
ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. For more information please visit http://www.ilial.com/crawler; http://www.ilial.com/crawler; crawl@ilial.com)
ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company.; http://www.ilial.com/crawler; crawl@ilial.com)
IlseBot/1.1
inelibotTqopauiuehccihTeenhkau
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/)
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
IUPUI Research Bot v 1.9a
Java/1.4.2
Java/1.4.2_01
Java/1.4.2_03
Java/1.4.2_04
Java/1.4.2_07
Java/1.4.2_11
Java/1.4.2_12
Java/1.4.2_13
Java/1.5.0-p3
Java/1.5.0_01
Java/1.5.0_02
Java/1.5.0_09
Java/1.5.0_12
Java/1.5.0_13
Java/1.6.0-oem
Java/1.6.0-rc
Java/1.6.0_02
Java/1.6.0_03
Java/1.6.0_04
Java/1.6.0_05
John\'s SEO (http://www.familieboers.eu/seo/seoinfo.aspx - jj.boers@gmail.com)
KAIST AITrc Crawler
Krugle/Krugle,Nutch/0.8  (Krugle web crawler; http://corp.krugle.com/crawler/info.html; webcrawler@krugle.com)
Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://corp.krugle.com/crawler/info.html; webcrawler@krugle.com)
ktbxewuqetybotfo5aevfhg
LapozzBot/1.5 ( http://robot.lapozz.hu)
LarbinWebCrawler spider@download11.com
LargeSmall Crawler
Lexxe/Robot
libwww-perl/5.36
libwww-perl/5.43
libwww-perl/5.47
libwww-perl/5.48
libwww-perl/5.49
libwww-perl/5.50
libwww-perl/5.51
libwww-perl/5.53
libwww-perl/5.60
libwww-perl/5.62
libwww-perl/5.63
libwww-perl/5.64
libwww-perl/5.65
libwww-perl/5.68
libwww-perl/5.69
libwww-perl/5.70
libwww-perl/5.73
libwww-perl/5.75
libwww-perl/5.76
libwww-perl/5.77
libwww-perl/5.78
libwww-perl/5.79
libwww-perl/5.800
libwww-perl/5.801
libwww-perl/5.802
libwww-perl/5.803
libwww-perl/5.804
libwww-perl/5.805
libwww-perl/5.806
libwww-perl/5.807
libwww-perl/5.808
libwww-perl/5.810
libwww-perl/5.811
libwww-perl/5.812
LocalcomBot/1.3.0 ( http://www.local.com/bot.htm)
LTI/LemurProject Nutch Spider/Nutch-1.0-dev (lti crawler for CMU; http://www.lti.cs.cmu.edu; changkuk at cmu dot edu)
LTI/LemurProject Nutch Spider/Nutch-1.0-dev (Research spider using Nutch; http://lucene.apache.org/nutch/bot.html; admin@lemurproject.org)
LTI/LemurProject Nutch Spider/Nutch-1.0-dev (Research spider using Nutch; http://www.lemurproject.org; mhoy@cs.cmu.edu)
lwbotuTafkqpjfpwhjlvmukjaolmamcnk
Lycos_Spider_(modspider)
McBot/5.001 (windows; U; NT4.0; en-us)
Mediapartners-Google
Mediapartners-Google/2.1
medrabbit/0.1beta (Crawling for health research; http://www.medrapid.info/bot.htm; bot@medrabbit.info)
Missigua Locator 1.9
MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php? )
MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)
MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? )
MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
MLBot (www.metadatalabs.com)
MLBot (www.metadatalabs.com/mlbot)
Moreoverbot/5.00 ( http://www.moreover.com)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma;  http://about.ask.com/en/docs/about/webmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma;  http://sp.ask.com/docs/about/tech_crawling.html)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)
Mozilla/4.0 (compatible;  Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Mozilla/4.0 (compatible; GoogleBot 1.2)
Mozilla/4.0 (compatible; MSIE 5.01; Windows 95; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 5.0; YANDEX)
Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4;  http://www.singingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread  Revision: 3.11
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows CE; IEMobile m.n) libwww-perl/5.805
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Google Wireless Transcoder;)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MSIECrawler
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1),gzip(gfe) (via translate.google.com)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ),gzip(gfe) (via translate.google.com)
Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Communications Corp., Korea) libwww-perl/5.808
Mozilla/4.0 (compatible; piCrawler)
Mozilla/4.5 RPT-HTTPClient/0.3-2,gzip(gfe) (via translate.google.com)
Mozilla/5.0 (compatible; archive.org_bot/1.13.1x +http://crawler.archive.org)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma;  http://about.ask.com/en/docs/about/webmasters.shtml)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
Mozilla/5.0 (compatible; Googlebot/2.1;  http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net)
Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp),gzip(gfe) (via translate.google.com)
Mozilla/5.0 (compatible; Yahoo! Slurp;http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; zermelo;  http://www.powerset.com) [email:paul@page-store.com,crawl@powerset.com]
Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12,gzip(gfe) (via translate.google.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12,gzip(gfe) (via translate.google.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13; Google-TR-5.1.707.23222-en) Gecko/20080311 Firefox/2.0.0.13
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.4) Gecko/20070515 Googlebot 2,1
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5,gzip(gfe) (via translate.google.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:2.0.0) Gecko/20050511 Firefox/2.0.0.1,gzip(gfe) (via translate.google.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.5) Gecko/20070713 Firefox/2.0.0.5,gzip(gfe) (via translate.google.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.8.1.12; Google-TR-5.1.706.29690-en-GT) Gecko/20080201 Firefox/2.0.0.12
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12,gzip(gfe) (via translate.google.com)
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5,gzip(gfe) (via translate.google.com)
Mozilla/5.0 (Windows;) NimbleCrawler 2.0.1 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com
Mozilla/5.0 (Yahoo-Test/4.0 mailto:vertical-crawl-support@yahoo-inc.com)
Mozilla/5.0 (Yahoo-Test/4.0; mailto:vertical-crawl-support@yahoo-inc.com)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
Mplfsqik yjatgh mcavperlo
MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://vwbot.cs.uiuc.edu; mqbot@cs.uiuc.edu)
Mrwhebot bmlk yfkcrhsw
msnbot-media/1.0 ( http://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot/0.9 ( http://search.msn.com/msnbot.htm)
msnbot/0.9 (+http://search.msn.com/msnbot.htm)
msnbot/1.0 ( http://search.msn.com/msnbot.htm)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot/1.0+(+http://search.msn.com/msnbot.htm)
msnbot/1.1 ( http://search.msn.com/msnbot.htm)
MSR-ISRCCrawler
MSRBOT (http://research.microsoft.com/research/sv/msrbot)
MSRBOT (http://research.microsoft.com/research/sv/msrbot/
multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html)
mxdw ijboTgfrnTghmTwudn
my-heritrix-crawler( http://mywebsite.com)
NameOfAgent (CMS Spider)
NASA Search 1.0
NetResearchServer/4.0(loopimprovements.com/robot.html)
NextGenSearchBot 1 (for information visit http://about.zoominfo.com/About/NextGenSearchBot.aspx)
NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
nicebot
nKolcutqchh4Klyi4afbotvKsxK4ons cc
Nokia6610/1.0 (3.09) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html find@mtld.mobi) )
Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html tech@mtld.mobi) )
Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi find.mobi/bot.html) )
Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (for mobile crawler) )
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1  UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1;  http://www.google.com/bot.html)
Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
noxtrumbot/1.0 (crawler@noxtrum.com)
nrsbot/5.0(loopimprovements.com/robot.html)
nrsbot/5.0(loopip.com/robot.html)
Nutch/Nutch-0.9 (Eurobot; http://www.ayell.eu )
nutchbot/Nutch-1.0-dev
NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; west@cis.poly.edu)
nyac libmhq afJproJnnpcbotircgukoepykhc
nyjfbbotery dpvhcctvrplgfxykEgccxpx
oBot
Openfind data gatherer, Openbot/3.0 (robot-response@openfind.com.tw; http://www.openfind.com.tw/robot.html
Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html
OpenPlaces/Nutch-1.0-dev (OpenPlaces Content Crawler; http://www.openplace.com; dnadeau at openplaces.com)
Os xsxbotjpnbfbn wtm oq
owsBot/0.1 (Nutch; www.oneworldstreet.com; nutch-agent@lucene.apache.org)
Oxford_AI_BOT_V1.0 (compatible; MSIE 6.0;)
PCbot/3.0
Pete-Spider Light/1.2
pgyxeXbgqin8vbotkophhoXpnptgfvl8ulolryv
PHP version tracker (http://www.nexen.net/phpversion/bot.php)
Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
PlantyNet_WebRobot_V1.9 dhkang@plantynet.com
POE-Component-Client-HTTP/0.65 (perl; N; POE; en; rv:0.650000)
psbot/0.1 ( http://www.picsearch.com/bot.html)
psbot/0.1 (+http://www.picsearch.com/bot.html)
pythonic-crawler (suzuki@tkl.iis.u-tokyo.ac.jp)
qi8asakoakbotfmvi mltoR8xav
Rankivabot/3.2 (www.rankiva.com; 3.2; vzmxikn)
RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com)
Retriever/Nutch-0.9 (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
Robotgenius crawler/Nutch-1.0-dev (http://robotgenius.net; misc at robotgenius dot net)
RSS-SPIDER (http://www.rss-spider.com/submit200709.php)
sbotcpkwujpdumgnbugnLgqljdxbicvmsx
schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; +http://www.schibstedsok.no/bot/)
ScSpider/0.2
Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.2
Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au)
ShablastBot 1.0
ShopWiki/1.0 (  http://www.shopwiki.com/wiki/Help:Bot)
Snapbot/1.0 ( http://www.snap.com)
Snapbot/1.0 (+http://www.snap.com)
Snapbot/1.0 (Snap Shots,  http://www.snap.com)
Snapbot/1.0 (Snap Shots, +http://www.snap.com)
sogou develop spider
Sogou web spider/3.0( http://www.sogou.com/docs/help/webmasters.htm
SouFZ-Spider[SouFZ.COM]
Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
Spock Crawler (http://www.spock.com/crawler)
Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Steeler/3.4 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
SurveyBot/2.3 (Whois Source)
Technoratibot/0.7
Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@netseer.com)
Teemer (NetSeer, Inc.; http://www.netseer.com/crawler.html; crawler@netseer.com)
TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://chitchit.org/TestCrawler.html; amitjain at spro dot net)
ThaiSUE-Crawler
Theme Spider ( http://www.themespider.com/spider.html)
TinEye/1.1 (http://tineye.com/crawler.html)
TMCrawler
tmvmbmObsojgfamqqbOthlvdibxObugyvy
Trailfire-bot/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
Trailfire-bot/0.7.1 (Trailfire page content analyzer; http://trailfire.com; info@trailfire.com)
Trailfire-bot/Nutch-1.0-dev (Trailfire page content analyzer; http://trailfire.com; info@trailfire.com)
Trailfire/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
travel-spider/Nutch-0.9
TRAVEL.com - travel-search engine (see http://www.travel.com/spiders.html)
TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
TwengaBot/1.1 ( http://www.twenga.com/bot.html)
Twiceler www.cuill.com/twiceler/robot.html
Twiceler-0.9 http://www.cuill.com/twiceler/robot.html
UogU0bxemmuf fyaprbot0 vbvhajwnUbaa
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1),gzip(gfe) (via translate.google.com)
VadixBot
VadixBot_Exp
VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com)
voyager-hc/1.0 ( http://www.kosmix.com/crawler.html)
voyager-hc/1.0 (+http://www.kosmix.com/crawler.html)
vspider
VSynCrawler/1.0
VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu; vwbot@cs.uiuc.edu)
w5vbotjovngrklftksnNvsxlimb srslooygel5
WebAlta Crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
WebAlta Crawler/1.3.34 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU)
webbot( http://webbot.com/bot.htm)
WebRankSpider/1.37 ( http://ulm191.server4you.de/crawler/)
WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
Wells Search II
WEP Search 00
WinHTTP Robot/1.0
woriobot ( http://www.worio.com/)
woriobot (+http://www.worio.com)
woriobot (+http://www.worio.com/)
Wujugo/Nutch-1.0-dev (Wujugo Content Crawler; http://www.wujugo.org; {dnadeau,srainville} at brightspark.com)
Xgpbotvu umkbwtq hmvykj
XwSpider
yacybot (i386 Linux 2.6.15-29-386; java 1.6.0; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.9-023stab044.12-smp; java 1.6.0_03; Europe/en) http://yacy.net/yacy/bot.html
yacybot (i386 Linux 2.6.9-023stab044.12-smp; java 1.6.0_04; Europe/en) http://yacy.net/yacy/bot.html
yacybot (i386 Linux 2.6.9-023stab044.4-smp; java 1.6.0_02; Europe/en) http://yacy.net/yacy/bot.html
yacybot (x86 Windows 2003 5.2; java 1.6.0_03; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows 2003 5.2; java 1.6.0_04; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_01; Europe/de) http://yacy.net/yacy/bot.html
Yahoo Pipes 1.0
Yahoo! Mindset
Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)
YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)
Yandex/1.01.001 (compatible; Win16; H)
Yandex/1.01.001 (compatible; Win16; I)
Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follow it)
Yeti/1.0 ( http://help.naver.com/robots/)
yoofind/yoofind-0.1-dev (yoono webcrawler; http://www.yoono.com ; crawl@yoono.com)
YooName/Nutch-1.0-dev (YooName Content Crawler; http://www.yooname.com; pythonner-at gmail . com)