funnelweb
Version:
Detect search engine crawlers by their User-Agent strings.
1 lines • 1.42 kB
JavaScript
module.exports = /anyapex|arachmo|B-l-i-t-z-B-O-T|boitho|Cerberian|Charlotte|cosmos|Covario|DataparkSearch|findlinks|holmes|htdig|ia_archiver|ichiro|l\.webis|^PycURL|^tineye|^VYU2|Larbin|LinkWalker|lwp-trivial|mabontland|Mnogosearch|updated\@updated\.com|Morning Paper|mvaclient|NetResearchServer|NewsGator|yahooseeker|NG-Search|NutchCVS|Nymesis|Orbiter|Qseero|Radian6_|ScoutJet|Teoma|Yahoo\! Slurp|yandex|(?:Peww|Pompos|PostPost|SBIder|Scrubby|SearchSight|semanticdiscovery|shopwiki|silk|snappy|sqworm|StackRambler|truwoGPS|vagabondo|vortex|^voyager|webcollage|yeti|yoogliFetchAgent|zao|zyborg|WomlpeFactory|peew|mogimogi|Accoona-AI-Agent|Mediapartners-Google)\/\d|(?:ABACHO|Spider|Become|Beslist|Bim|Blitz|Diamond|Disco|Dot|Sheild\.com Web|Esperanza|Exa|FDSE ro|Furl|Gais|Galaxy|Genie|Giga|google|Girafa|Guruji|HappyFun|IRL|Jaxified |Jyxo|Koepa|Lapozz|lexxe|Linguee |MJ12|Mojeek|Moreover|msn|msr|mx|nice|noxstrum|\; o|omgili|OmniExplorer.?|OOZ|BitesHyper|poly|ps|RAMPy|Rufus|Seek|SEOChat..|Seznam|Shoula ro|site|suggy|survey|synoo|terrawiz|TheSu|\sro|Turnitin|TweetedTimes |Twenga|voila|yacy|Yasakli|You?dao|zeal|noxtrum|billybob|bt|bing|catch|emerald shield\.com web|)bot|(?:Baidu|\[ITS\]|Fyber|hl_ftien_|iask|igde|ld|Im|Nusearch |sogou |soso|speedy |_Web|z|^lm|igde)sp[iy]der|(?:Convera|Enter?prise |FAST-Web|g2|iC|issue|magpie-|NetSeer |sand|sensis web |Shim-)crawler|www\.almaden\.ibm\.com\/cs\/crawler|oegp v/gi