Eliminare i bot malevoli e gli scraper con un comando in htaccess

Lo sapete che il vostro sito è visitato costantemente da BOT e SCRAPER?

Questi due simpatici giocherelloni accedono al nostro sito per esaminarlo e per ciucciare risorse.

Si infatti potrebbe succedere che per colpa di questi due simpatici rompini il vostro sito esaurisca le risorse della CPU, il piu delle volte crashando.

Questi BOT malevoli e gli Scraper sono i peggiori nemici del processore del nostro server.

Cosa fare?

Se siete su server LINUX e APACHE potrete utilizzare il file .htaccess

Potrete crearne uno nuovo o utilizzarne uno già esistente nella root del vostro sito web.

Copiate e incollate il codice qui sotto che contiene una lista aggiornata di bot malevoli e scraper al agosto 2018.

# Block Bad Bots & Scrapers
SetEnvIfNoCase User-Agent "Aboundex" bad_bot
SetEnvIfNoCase User-Agent "80legs" bad_bot
SetEnvIfNoCase User-Agent "360Spider" bad_bot
SetEnvIfNoCase User-Agent "^Java" bad_bot
SetEnvIfNoCase User-Agent "^Cogentbot" bad_bot
SetEnvIfNoCase User-Agent "^Alexibot" bad_bot
SetEnvIfNoCase User-Agent "^asterias" bad_bot
SetEnvIfNoCase User-Agent "^attach" bad_bot
SetEnvIfNoCase User-Agent "^BackDoorBot" bad_bot
SetEnvIfNoCase User-Agent "^BackWeb" bad_bot
SetEnvIfNoCase User-Agent "Bandit" bad_bot
SetEnvIfNoCase User-Agent "^BatchFTP" bad_bot
SetEnvIfNoCase User-Agent "^Bigfoot" bad_bot
SetEnvIfNoCase User-Agent "^Black.Hole" bad_bot
SetEnvIfNoCase User-Agent "^BlackWidow" bad_bot
SetEnvIfNoCase User-Agent "^BlowFish" bad_bot
SetEnvIfNoCase User-Agent "^BotALot" bad_bot
SetEnvIfNoCase User-Agent "Buddy" bad_bot
SetEnvIfNoCase User-Agent "^BuiltBotTough" bad_bot
SetEnvIfNoCase User-Agent "^Bullseye" bad_bot
SetEnvIfNoCase User-Agent "^BunnySlippers" bad_bot
SetEnvIfNoCase User-Agent "^Cegbfeieh" bad_bot
SetEnvIfNoCase User-Agent "^CheeseBot" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^ChinaClaw" bad_bot
SetEnvIfNoCase User-Agent "Collector" bad_bot
SetEnvIfNoCase User-Agent "Copier" bad_bot
SetEnvIfNoCase User-Agent "^CopyRightCheck" bad_bot
SetEnvIfNoCase User-Agent "^cosmos" bad_bot
SetEnvIfNoCase User-Agent "^Crescent" bad_bot
SetEnvIfNoCase User-Agent "^Custo" bad_bot
SetEnvIfNoCase User-Agent "^AIBOT" bad_bot
SetEnvIfNoCase User-Agent "^DISCo" bad_bot
SetEnvIfNoCase User-Agent "^DIIbot" bad_bot
SetEnvIfNoCase User-Agent "^DittoSpyder" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Demon" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Devil" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Wonder" bad_bot
SetEnvIfNoCase User-Agent "^dragonfly" bad_bot
SetEnvIfNoCase User-Agent "^Drip" bad_bot
SetEnvIfNoCase User-Agent "^eCatch" bad_bot
SetEnvIfNoCase User-Agent "^EasyDL" bad_bot
SetEnvIfNoCase User-Agent "^ebingbong" bad_bot
SetEnvIfNoCase User-Agent "^EirGrabber" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^EroCrawler" bad_bot
SetEnvIfNoCase User-Agent "^Exabot" bad_bot
SetEnvIfNoCase User-Agent "^Express\ WebPictures" bad_bot
SetEnvIfNoCase User-Agent "Extractor" bad_bot
SetEnvIfNoCase User-Agent "^EyeNetIE" bad_bot
SetEnvIfNoCase User-Agent "^Foobot" bad_bot
SetEnvIfNoCase User-Agent "^flunky" bad_bot
SetEnvIfNoCase User-Agent "^FrontPage" bad_bot
SetEnvIfNoCase User-Agent "^Go-Ahead-Got-It" bad_bot
SetEnvIfNoCase User-Agent "^gotit" bad_bot
SetEnvIfNoCase User-Agent "^GrabNet" bad_bot
SetEnvIfNoCase User-Agent "^Grafula" bad_bot
SetEnvIfNoCase User-Agent "^Harvest" bad_bot
SetEnvIfNoCase User-Agent "^hloader" bad_bot
SetEnvIfNoCase User-Agent "^HMView" bad_bot
SetEnvIfNoCase User-Agent "^HTTrack" bad_bot
SetEnvIfNoCase User-Agent "^humanlinks" bad_bot
SetEnvIfNoCase User-Agent "^IlseBot" bad_bot
SetEnvIfNoCase User-Agent "^Image\ Stripper" bad_bot
SetEnvIfNoCase User-Agent "^Image\ Sucker" bad_bot
SetEnvIfNoCase User-Agent "Indy\ Library" bad_bot
SetEnvIfNoCase User-Agent "^InfoNaviRobot" bad_bot
SetEnvIfNoCase User-Agent "^InfoTekies" bad_bot
SetEnvIfNoCase User-Agent "^Intelliseek" bad_bot
SetEnvIfNoCase User-Agent "^InterGET" bad_bot
SetEnvIfNoCase User-Agent "^Internet\ Ninja" bad_bot
SetEnvIfNoCase User-Agent "^Iria" bad_bot
SetEnvIfNoCase User-Agent "^Jakarta" bad_bot
SetEnvIfNoCase User-Agent "^JennyBot" bad_bot
SetEnvIfNoCase User-Agent "^JetCar" bad_bot
SetEnvIfNoCase User-Agent "^JOC" bad_bot
SetEnvIfNoCase User-Agent "^JustView" bad_bot
SetEnvIfNoCase User-Agent "^Jyxobot" bad_bot
SetEnvIfNoCase User-Agent "^Kenjin.Spider" bad_bot
SetEnvIfNoCase User-Agent "^Keyword.Density" bad_bot
SetEnvIfNoCase User-Agent "^larbin" bad_bot
SetEnvIfNoCase User-Agent "^LexiBot" bad_bot
SetEnvIfNoCase User-Agent "^lftp" bad_bot
SetEnvIfNoCase User-Agent "^libWeb/clsHTTP" bad_bot
SetEnvIfNoCase User-Agent "^likse" bad_bot
SetEnvIfNoCase User-Agent "^LinkextractorPro" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan/8.1a.Unix" bad_bot
SetEnvIfNoCase User-Agent "^LNSpiderguy" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial" bad_bot
SetEnvIfNoCase User-Agent "^LWP::Simple" bad_bot
SetEnvIfNoCase User-Agent "^Magnet" bad_bot
SetEnvIfNoCase User-Agent "^Mag-Net" bad_bot
SetEnvIfNoCase User-Agent "^MarkWatch" bad_bot
SetEnvIfNoCase User-Agent "^Mass\ Downloader" bad_bot
SetEnvIfNoCase User-Agent "^Mata.Hari" bad_bot
SetEnvIfNoCase User-Agent "^Memo" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft.URL" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft\ URL\ Control" bad_bot
SetEnvIfNoCase User-Agent "^MIDown\ tool" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc" bad_bot
SetEnvIfNoCase User-Agent "^Mirror" bad_bot
SetEnvIfNoCase User-Agent "^Missigua\ Locator" bad_bot
SetEnvIfNoCase User-Agent "^Mister\ PiX" bad_bot
SetEnvIfNoCase User-Agent "^moget" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla/3.Mozilla/2.01" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla.*NEWT" bad_bot
SetEnvIfNoCase User-Agent "^NAMEPROTECT" bad_bot
SetEnvIfNoCase User-Agent "^Navroad" bad_bot
SetEnvIfNoCase User-Agent "^NearSite" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts" bad_bot
SetEnvIfNoCase User-Agent "^Netcraft" bad_bot
SetEnvIfNoCase User-Agent "^NetMechanic" bad_bot
SetEnvIfNoCase User-Agent "^NetSpider" bad_bot
SetEnvIfNoCase User-Agent "^Net\ Vampire" bad_bot
SetEnvIfNoCase User-Agent "^NetZIP" bad_bot
SetEnvIfNoCase User-Agent "^NextGenSearchBot" bad_bot
SetEnvIfNoCase User-Agent "^NG" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^niki-bot" bad_bot
SetEnvIfNoCase User-Agent "^NimbleCrawler" bad_bot
SetEnvIfNoCase User-Agent "^Ninja" bad_bot
SetEnvIfNoCase User-Agent "^NPbot" bad_bot
SetEnvIfNoCase User-Agent "^Octopus" bad_bot
SetEnvIfNoCase User-Agent "^Offline\ Explorer" bad_bot
SetEnvIfNoCase User-Agent "^Offline\ Navigator" bad_bot
SetEnvIfNoCase User-Agent "^Openfind" bad_bot
SetEnvIfNoCase User-Agent "^OutfoxBot" bad_bot
SetEnvIfNoCase User-Agent "^PageGrabber" bad_bot
SetEnvIfNoCase User-Agent "^Papa\ Foto" bad_bot
SetEnvIfNoCase User-Agent "^pavuk" bad_bot
SetEnvIfNoCase User-Agent "^pcBrowser" bad_bot
SetEnvIfNoCase User-Agent "^PHP\ version\ tracker" bad_bot
SetEnvIfNoCase User-Agent "^Pockey" bad_bot
SetEnvIfNoCase User-Agent "^ProPowerBot/2.14" bad_bot
SetEnvIfNoCase User-Agent "^ProWebWalker" bad_bot
SetEnvIfNoCase User-Agent "^psbot" bad_bot
SetEnvIfNoCase User-Agent "^Pump" bad_bot
SetEnvIfNoCase User-Agent "^QueryN.Metasearch" bad_bot
SetEnvIfNoCase User-Agent "^RealDownload" bad_bot
SetEnvIfNoCase User-Agent "Reaper" bad_bot
SetEnvIfNoCase User-Agent "Recorder" bad_bot
SetEnvIfNoCase User-Agent "^ReGet" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey" bad_bot
SetEnvIfNoCase User-Agent "^RMA" bad_bot
SetEnvIfNoCase User-Agent "Siphon" bad_bot
SetEnvIfNoCase User-Agent "^SiteSnagger" bad_bot
SetEnvIfNoCase User-Agent "^SlySearch" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload" bad_bot
SetEnvIfNoCase User-Agent "^Snake" bad_bot
SetEnvIfNoCase User-Agent "^Snapbot" bad_bot
SetEnvIfNoCase User-Agent "^Snoopy" bad_bot
SetEnvIfNoCase User-Agent "^sogou" bad_bot
SetEnvIfNoCase User-Agent "^SpaceBison" bad_bot
SetEnvIfNoCase User-Agent "^SpankBot" bad_bot
SetEnvIfNoCase User-Agent "^spanner" bad_bot
SetEnvIfNoCase User-Agent "^Sqworm" bad_bot
SetEnvIfNoCase User-Agent "Stripper" bad_bot
SetEnvIfNoCase User-Agent "Sucker" bad_bot
SetEnvIfNoCase User-Agent "^SuperBot" bad_bot
SetEnvIfNoCase User-Agent "^SuperHTTP" bad_bot
SetEnvIfNoCase User-Agent "^Surfbot" bad_bot
SetEnvIfNoCase User-Agent "^suzuran" bad_bot
SetEnvIfNoCase User-Agent "^Szukacz/1.4" bad_bot
SetEnvIfNoCase User-Agent "^tAkeOut" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^Telesoft" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot/1.5" bad_bot
SetEnvIfNoCase User-Agent "^The.Intraformant" bad_bot
SetEnvIfNoCase User-Agent "^TheNomad" bad_bot
SetEnvIfNoCase User-Agent "^TightTwatBot" bad_bot
SetEnvIfNoCase User-Agent "^Titan" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot" bad_bot
SetEnvIfNoCase User-Agent "^turingos" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot" bad_bot
SetEnvIfNoCase User-Agent "^URLy.Warning" bad_bot
SetEnvIfNoCase User-Agent "^Vacuum" bad_bot
SetEnvIfNoCase User-Agent "^VCI" bad_bot
SetEnvIfNoCase User-Agent "^VoidEYE" bad_bot
SetEnvIfNoCase User-Agent "^Web\ Image\ Collector" bad_bot
SetEnvIfNoCase User-Agent "^Web\ Sucker" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit" bad_bot
SetEnvIfNoCase User-Agent "^Webclipping.com" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier" bad_bot
SetEnvIfNoCase User-Agent "^WebEMailExtrac.*" bad_bot
SetEnvIfNoCase User-Agent "^WebEnhancer" bad_bot
SetEnvIfNoCase User-Agent "^WebFetch" bad_bot
SetEnvIfNoCase User-Agent "^WebGo\ IS" bad_bot
SetEnvIfNoCase User-Agent "^Web.Image.Collector" bad_bot
SetEnvIfNoCase User-Agent "^WebLeacher" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot" bad_bot
SetEnvIfNoCase User-Agent "^WebReaper" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger" bad_bot
SetEnvIfNoCase User-Agent "^Website\ eXtractor" bad_bot
SetEnvIfNoCase User-Agent "^Website\ Quester" bad_bot
SetEnvIfNoCase User-Agent "^Webster" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper" bad_bot
SetEnvIfNoCase User-Agent "^WebWhacker" bad_bot
SetEnvIfNoCase User-Agent "^WebZIP" bad_bot
SetEnvIfNoCase User-Agent "Whacker" bad_bot
SetEnvIfNoCase User-Agent "^Widow" bad_bot
SetEnvIfNoCase User-Agent "^WISENutbot" bad_bot
SetEnvIfNoCase User-Agent "^WWWOFFLE" bad_bot
SetEnvIfNoCase User-Agent "^WWW-Collector-E" bad_bot
SetEnvIfNoCase User-Agent "^Xaldon" bad_bot
SetEnvIfNoCase User-Agent "^Xenu" bad_bot
SetEnvIfNoCase User-Agent "^Zeus" bad_bot
SetEnvIfNoCase User-Agent "ZmEu" bad_bot
SetEnvIfNoCase User-Agent "^Zyborg" bad_bot
SetEnvIfNoCase User-Agent "AhrefsBot" bad_bot
SetEnvIfNoCase User-Agent "SemrushBot" bad_bot
SetEnvIfNoCase User-Agent "istellabot" bad_bot

# Vulnerability Scanners
SetEnvIfNoCase User-Agent "Acunetix" bad_bot
SetEnvIfNoCase User-Agent "FHscan" bad_bot

# Aggressive Chinese Search Engine
SetEnvIfNoCase User-Agent "Baiduspider" bad_bot

# Aggressive Russian Search Engine
SetEnvIfNoCase User-Agent "Yandex" bad_bot

<Limit GET POST HEAD>
Order Allow,Deny
Allow from all

# Cyveillance
deny from 38.100.19.8/29
deny from 38.100.21.0/24
deny from 38.100.41.64/26
deny from 38.105.71.0/25
deny from 38.105.83.0/27
deny from 38.112.21.140/30
deny from 38.118.42.32/29
deny from 65.213.208.128/27
deny from 65.222.176.96/27
deny from 65.222.185.72/29

Deny from env=bad_bot
</Limit>

FATTO! Se volete potrete anche creare un file robots.txt e inserirlo nella root del vostro sito web con il seguente codice


User-agent: 1 2 3 Submit PRO
 disallow: /
 
User-agent: 200PleaseBot
 disallow: /
 
User-agent: 2ADAMbot
 disallow: /
 
User-agent: 2ADAMbot/1.0
 disallow: /
 
User-agent: 360Spider
 disallow: /
 
user-agent: Abonti
 disallow: /
 
user-agent: Abonti/0.92
 disallow: /
 
user-agent: abot v1.0
 disallow: /
 
user-agent: aboutthedomain
 disallow: /
 
user-agent: Add Catalog
 disallow: /
 
user-agent: Add Catalog/2.1
 disallow: /
 
user-agent: AdvBot
 disallow: /
 
user-agent: AdvBot/2.0
 disallow: /
 
user-agent: AhrefsBot
 disallow: /
 
user-agent: Ahrefs-Bot
 disallow: /
 
user-agent: AhrefsBot/1.0
 disallow: /
 
user-agent: Ahrefs-Bot/1.0
 disallow: /
 
user-agent: Ahrefs-Bot/2.0
 disallow: /
 
user-agent: Ahrefs-Bot/3.0
 disallow: /
 
user-agent: Ahrefs-Bot/4.0
 disallow: /
 
user-agent: Ahrefs-Bot/5.0
 disallow: /
 
user-agent: aiHitBot
 disallow: /
 
user-agent: aiHitBot/2.9
 disallow: /
 
user-agent: Anonymous/0.0
 disallow: /
 
user-agent: Arachnida
 disallow: /
 
user-agent: Associative Spider
 disallow: /
 
User-agent: Baiduspider
 disallow: /
 
User-agent: Baidu Spider
 disallow: /
 
User-agent: Battleztar Bazinga
 disallow: /
 
User-agent: Battleztar Bazinga/0.01
 disallow: /
 
User-agent: BDFetch
 disallow: /
 
User-agent: betaBot
 disallow: /
 
User-agent: bieshu
 disallow: /
 
User-agent: Bigli SEO
 disallow: /
 
User-agent: Blackboard Safeassign
 disallow: /
 
User-agent: Blazer 1.0
 disallow: /
 
User-agent: BLEXBot
 disallow: /
 
User-agent: BLEXBot/1.0
 disallow: /
 
User-agent: BLP_bbot
 disallow: /
 
User-agent: BLP_bbot/0.1
 disallow: /
 
User-agent: BOIA-Accessibility-Agent/PR 1.0
 disallow: /
 
User-agent: BOT for JCE
 disallow: /
 
User-agent: BOT/0.1 (BOT for JCE)
 disallow: /
 
User-agent: BPImageWalker
 disallow: /
 
User-agent: BPImageWalker/2.0
 disallow: /
 
User-agent: BUbiNG
 disallow: /
 
User-agent: BuiBui-Bot
 disallow: /
 
User-agent: BuiBui-Bot/1.0
 disallow: /
 
User-agent: ca-crawler
 disallow: /
 
User-agent: ca-crawler/1.0
 disallow: /
 
User-agent: CakePHP
 disallow: /
 
User-agent: Calypso v/0.01
 disallow: /
 
User-agent: Calypso
 disallow: /
 
User-agent: CB/Nutch-1.7
 disallow: /
 
User-agent: CCBot
 disallow: /
 
User-agent: CCBot/2.0
 disallow: /
 
User-agent: Checkbot
 disallow: /
 
User-agent: checkgzipcompression.com
 disallow: /
 
User-agent: chushou
 disallow: /
 
User-agent: CloudServerMarketSpider
 disallow: /
 
User-agent: CloudServerMarketSpider/1.0
 disallow: /
  
User-agent: Clushbot/3.x-BinaryFury
 disallow: /
 
User-agent: CMS Crawler
 disallow: /
 
User-agent: CMS Crawler: http://www.cmscrawler.com
 disallow: /
 
User-agent: coccoc
 disallow: /
 
User-agent: CoinCornerBot
 disallow: /
 
User-agent: CoinCornerBot/1.1
 disallow: /
 
User-agent: Copyscape
 disallow: /
 
User-agent: crawler4j
 disallow: /
 
User-agent: CRAZYWEBCRAWLER 0.9.0
 disallow: /
 
User-agent: CRAZYWEBCRAWLER 0.9.1
 disallow: /
 
User-agent: CRAZYWEBCRAWLER 0.9.7
 disallow: /
 
User-agent: CrazyWebCrawler
 disallow: /
 
User-agent: CrazyWebCrawler-Spider
 disallow: /
 
User-agent: Crowsnest
 disallow: /
 
User-agent: Crowsnest/0.5
 disallow: /
 
User-agent: Curious George - www.analyticsseo.com/crawler
 disallow: /
 
User-agent: Curious George
 disallow: /
 
User-agent: cuwhois
 disallow: /
 
User-agent: cuwhois/1.0
 disallow: /
 
User-agent: dahoms
 disallow: /
 
User-agent: datagnionbot
 disallow: /
 
User-agent: DeuSu/5.0.2
 disallow: /
 
User-agent: Digincore
 disallow: /
 
User-agent: Digincore bot
 disallow: /
 
User-agent: Dispatch/0.11.0
 disallow: /
 
User-agent: Domain Re-Animator Bot
 disallow: /
 
User-agent: DomainAppender /1.0
 disallow: /
 
User-agent: DomainAppender
 disallow: /
  
 User-agent: DomainCrawler/3.0
 disallow: /
 
User-agent: DomainSigmaCrawler
 disallow: /
 
User-agent: DomainSigmaCrawler/0.1
 disallow: /
 
User-agent: Domnutch
 disallow: /
 
User-agent: Domnutch-Bot
 disallow: /
 
User-agent: Domnutch-Bot/Nutch
 disallow: /
 
User-agent: Domnutch-Bot/Nutch-1.0
 disallow: /
 
User-agent: dotbot
 disallow: /
 
User-agent: ECCP/1.2.1
 disallow: /
 
User-agent: eCommerceBot
 disallow: /
 
User-agent: enlle punto com/Nutch-1.9
 disallow: /
 
User-agent: EPiServer Link Checker
 disallow: /
 
User-agent: EuripBot
 disallow: /
 
User-agent: EuripBot/2.0
 disallow: /
 
User-agent: evc/2.0
 disallow: /
 
User-agent: evc-batch
 disallow: /
 
User-agent: evc-batch/2.0
 disallow: /
 
User-agent: Express WebPictures
 disallow: /
 
User-agent: Faraday v0.8.8
 disallow: /
 
User-agent: Faraday
 disallow: /
 
User-agent: Findxbot
 disallow: /
 
User-agent: Findxbot/1.0
 disallow: /
 
User-agent: Flamingo_SearchEngine
 disallow: /
 
User-agent: Flipboard Robot
 disallow: /
 
User-agent: GetProxi.es-bot
 disallow: /
 
User-agent: GetProxi.es-bot/1.1
 disallow: /
 
User-agent: GigablastOpenSource
 disallow: /
 
User-agent: GigablastOpenSource/1.0
 disallow: /
 
User-agent: Girafabot
 disallow: /
 
User-agent: Gluten Free Crawler
 disallow: /
 
User-agent: Gluten Free Crawler/1.0
 disallow: /
 
User-agent: GriffinBot
 disallow: /
 
User-agent: GrifinBot/0.01
 disallow: /
 
User-agent: GWPImages
 disallow: /
 
User-agent: GWPImages/1.0
 disallow: /
 
User-agent: Haiula
 disallow: /
 
User-agent: Haiula/1.4
 disallow: /
 
User-agent: HaosouSpider
 disallow: /
 
User-agent: Hivemind
 disallow: /
 
User-agent: HostHarvest
 disallow: /
 
User-agent: HostHarvest/0.4.28
 disallow: /
 
User-agent: HRCrawler
 disallow: /
 
User-agent: HRCrawler/2.0
 disallow: /
 
User-agent: http://git.io/tl_S2w
 disallow: /
 
User-agent: http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
 disallow: /
 
User-agent: HubSpot Links Crawler 1.0
 disallow: /
 
User-agent: HubSpot Webcrawler
 disallow: /
 
User-agent: HubSpot
 disallow: /
 
User-agent: hunchan
 disallow: /
 
User-agent: HyperCrawl
 disallow: /
 
User-agent: HyperCrawl/0.2
 disallow: /
 
User-agent: ICAP-IOD
 disallow: /
 
User-agent: ICC-Crawler
 disallow: /
 
User-agent: ICC-Crawler/2.0
 disallow: /
 
User-agent: Ichiro Robot
 disallow: /
 
User-agent: image.coccoc/1.0
 disallow: /
 
User-agent: Image2play
 disallow: /
 
User-agent: Image2play/0.1
 disallow: /
 
User-agent: Indy Library
 disallow: /
 
User-agent: InsightsCollector
 disallow: /
 
User-agent: InsightsCollector/0.1
 disallow: /
 
User-agent: InsightsCollector/0.1beta
 disallow: /
 
User-agent: integrity/5
 disallow: /
 
User-agent: InterNaetBoten
 disallow: /
 
User-agent: InterNaetBoten/0.99
 disallow: /
 
User-agent: IRL Crawler
 disallow: /
 
User-agent: James BOT - WebCrawler
 disallow: /
 
User-agent: James BOT
 disallow: /
 
User-agent: JamesBOT
 disallow: /
 
User-agent: JetBrains 5.0
 disallow: /
 
User-agent: JetBrains
 disallow: /
 
User-agent: Kraken
 disallow: /
 
User-agent: Kraken/0.1
 disallow: /
 
User-agent: Kyoto-Tohoku-Crawler/v1
 disallow: /
 
User-agent: larbin
 disallow: /
 
User-agent: lechenie
 disallow: /
 
User-agent: libwww-perl
 disallow: /
 
User-agent: link checker
 disallow: /
 
User-agent: Link/1.0
 disallow: /
 
User-agent: linkCheck
 disallow: /
 
User-agent: linkCheckV3.0
 disallow: /
 
User-agent: Linkdex
 disallow: /
 
User-agent: linkdex.com/v2.0
 disallow: /
 
User-agent: linkdex.com/v2.1
 disallow: /
 
User-agent: LinkdexBot
 disallow: /
 
User-agent: linkdexbot/2.0
 disallow: /
 
User-agent: linkdexbot/2.1
 disallow: /
 
User-agent: linkdexbot-mobile/2.1
 disallow: /
 
User-agent: LinkpadBot
 disallow: /
 
User-agent: LinkpadBot/1.06
 disallow: /
 
User-agent: LinqiaScrapeBot
 disallow: /
 
User-agent: LinqiaScrapeBot/1.0
 disallow: /
 
User-agent: Lipperhey SEO Service
 disallow: /
 
User-agent: Lipperhey
 disallow: /
 
User-agent: Lipperhey-Kaus-Australis
 disallow: /
 
User-agent: Lipperhey-Kaus-Australis/5.0
 disallow: /
 
User-agent: listicka
 disallow: /
 
User-agent: LSSRocketCrawler
 disallow: /
 
User-agent: LSSRocketCrawler/1.0 LightspeedSystems
 disallow: /
 
User-agent: LSSRocketCrawler/1.0
 disallow: /
 
User-agent: ltx71
 disallow: /
 
User-agent: LWNutch/Nutch-1.4
 disallow: /
 
User-agent: Mail.RU
 disallow: /
 
User-agent: Mail.RU_Bot
 disallow: /
 
User-agent: Mail.RU_Bot/2.0
 disallow: /
 
User-agent: Mail.RU_Bot/Fast/2.0
 disallow: /
 
User-agent: md5sum
 disallow: /
 
User-agent: md5sum\x22
 disallow: /
 
User-agent: meanpathbot
 disallow: /
 
User-agent: MegaIndex.ru
 disallow: /
 
User-agent: MegaIndex.ru/2.0
 disallow: /
 
User-agent: mezhpozvonochnoi
 disallow: /
 
User-agent: Mike-Crawler
 disallow: /
 
User-agent: MixBot
 disallow: /
 
User-agent: MixrankBot
 disallow: /
 
User-agent: MJ12bot
 disallow: /
 
User-agent: Monkeybot/0.1
 disallow: /
 
User-agent: my crawler
 disallow: /
 
User-agent: My Nutch Spider/Nutch-1.9
 disallow: /
 
User-agent: mycrowl/Nutch-1.9
 disallow: /
 
User-agent: MyGreatUA/2.0
 disallow: /
 
User-agent: MyIPTest
 disallow: /
 
User-agent: NameProtect Robot
 disallow: /
 
User-agent: NerdyBot
 disallow: /
 
User-agent: Netcraft Spider
 disallow: /
 
User-agent: netEstate NE Crawler
 disallow: /
 
User-agent: NetLyzer FastProbe
 disallow: /
 
User-agent: NetResearchServer
 disallow: /
 
User-agent: NetResearchServer/4.0
 disallow: /
 
User-agent: Nmap Scripting Engine
 disallow: /
 
User-agent: node.io
 disallow: /
 
User-agent: node.js
 disallow: /
 
User-agent: Node/simplecrawler 0.5.2
 disallow: /
 
User-agent: Node/simplecrawler
 disallow: /
 
User-agent: oBot/2.3.1
 disallow: /
 
User-agent: omgilibot
 disallow: /
 
User-agent: omgilibot/0.4
 disallow: /
 
User-agent: Online Domain Tools - Online Website Link Checker
 disallow: /
 
User-agent: Online Domain Tools - Online Website Link Checker/1.2
 disallow: /
 
User-agent: Openfind Robot
 disallow: /
 
User-agent: OpenHoseBot
 disallow: /
 
User-agent: OpenHoseBot/2.1
 disallow: /
 
User-agent: Openstat
 disallow: /
 
User-agent: Openstat/0.1
 disallow: /
 
User-agent: OptimizationCrawler
 disallow: /
 
User-agent: OptimizationCrawler/0.2
 disallow: /
 
User-agent: Page Analyzer v4.0
 disallow: /
 
User-agent: Page Analyzer
 disallow: /
 
User-agent: PageAnalyzer
 disallow: /
 
User-agent: PageAnalyzer/1.1
 disallow: /
 
User-agent: PageAnalyzer/1.5
 disallow: /
 
User-agent: PagesInventory
 disallow: /
 
User-agent: Pagespeed/1.1 Fetcher
 disallow: /
 
User-agent: Pagespeed/1.1
 disallow: /
 
User-agent: Pagespeedbot
 disallow: /
 
User-agent: Perl LWP
 disallow: /
 
User-agent: PHPCrawl
 disallow: /
 
User-agent: phpSiteCheck 1.0
 disallow: /
 
User-agent: phpSiteCheck
 disallow: /
 
User-agent: Plukkie
 disallow: /
 
User-agent: POGS/2.0
 disallow: /
 
User-agent: Powermarks
 disallow: /
 
User-agent: PowerPivot
 disallow: /
 
User-agent: PRIVACY_ENFAQ.jsp
 disallow: /
 
User-agent: Prlog
 disallow: /
 
User-agent: Prlog/1.0
 disallow: /
 
User-agent: publiclibraryarchive.org
 disallow: /
 
User-agent: publiclibraryarchive.org/1.0
 disallow: /
 
User-agent: Pu_iN Crawler
 disallow: /
 
User-agent: Putin
 disallow: /
 
User-agent: Putin spider
 disallow: /
 
User-agent: qingdao
 disallow: /
 
User-agent: QlikView
 disallow: /
 
User-agent: quipu
 disallow: /
 
User-agent: quipu/1.0
 disallow: /
 
User-agent: quipu/2.0
 disallow: /
 
User-agent: R6_CommentReader
 disallow: /
 
User-agent: R6_FeedFetcher
 disallow: /
 
User-agent: Riddler
 disallow: /
 
User-agent: RivalSeek.com-Bot
 disallow: /
 
User-agent: rogerbot
 disallow: /
 
User-agent: rogerbot/1.0
 disallow: /
 
User-agent: rootlink
 disallow: /
 
User-agent: RU_Bot/2.0
 disallow: /
 
User-agent: Scopia
 disallow: /
 
User-agent: Scopia crawler
 disallow: /
 
User-agent: Scopia crawler 1.0
 disallow: /
 
User-agent: Scopia crawler 1.1
 disallow: /
 
User-agent: Scopia crawler 1.2
 disallow: /
 
User-agent: Scrapy
 disallow: /
 
User-agent: Scrapy/0.16.5
 disallow: /
 
User-agent: Scrapy/0.24.4
 disallow: /
 
User-agent: Scrapy/0.24.5
 disallow: /
 
User-agent: Scrapy/0.24.6
 disallow: /
 
User-agent: Scrapy/1.0.1
 disallow: /
 
User-agent: Screaming Frog SEO Spider
 disallow: /
 
User-agent: Screaming Frog SEO Spider/2,55
 disallow: /
 
User-agent: Screaming Frog SEO Spider/2.55
 disallow: /
 
User-agent: Screaming Frog SEO Spider/3.1
 disallow: /
 
User-agent: Screaming Frog SEO Spider/3.3
 disallow: /
 
User-agent: Screaming Frog SEO Spider/4.1
 disallow: /
 
User-agent: Screaming Frog SEO Spider/5.0
 disallow: /
 
User-agent: Screaming Frog SEO Spider/5.1
 disallow: /
 
User-agent: Screaming Frog SEO Spider/5.1 Beta 2
 disallow: /
 
User-agent: scrutiny/4
 disallow: /
 
User-agent: SemrushBot
 disallow: /
 
User-agent: SemrushBot-SA
 disallow: /
  
User-agent: SEOdiver/1.0
 disallow: /
 
User-agent: SEOkicks
 disallow: /
 
User-agent: SEOkicks-Robot
 disallow: /
 
User-agent: SEOlyticsCrawler
 disallow: /
 
User-agent: SEOlyticsCrawler/3.0
 disallow: /
 
User-agent: seoscanners
 disallow: /
 
User-agent: seoscanners.net/1
 disallow: /
 
User-agent: SEOstats 2.1.0
 disallow: /
 
User-agent: Seosys/Nutch-2.3
 disallow: /
 
User-agent: SetCronJob/1.0
 disallow: /
 
User-agent: SeznamBot
 disallow: /
 
User-agent: SheerBoredom.Experimental.Robot
 disallow: /
 
User-agent: SheerBoredom.Experimental.Robot/0.2
 disallow: /
 
User-agent: ShowyouBot
 disallow: /
 
User-agent: Simplecrawler
 disallow: /
 
User-agent: SISTRIX Crawler
 disallow: /
 
User-agent: sistrix
 disallow: /
 
User-agent: SiteBot
 disallow: /
 
User-agent: SiteBot/0.1
 disallow: /
 
User-agent: SiteExplorer
 disallow: /
 
User-agent: SiteExplorer/1.0
 disallow: /
 
User-agent: SiteExplorer/1.0b
 disallow: /
 
User-agent: Siteluxbot
 disallow: /
 
User-agent: Siteluxbot/1.0
 disallow: /
 
User-agent: SkimBot
 disallow: /
 
User-agent: SkimBot/1.0
 disallow: /
 
User-agent: sky nutch crawler/Nutch-1.9
 disallow: /
 
User-agent: SMTBot
 disallow: /
 
User-agent: SMTBot/1.0
 disallow: /
 
User-agent: SNK Screenshot Bot
 disallow: /
 
User-agent: SNK Screenshot Bot/0.20
 disallow: /
 
User-agent: Sogou Spider
 disallow: /
 
User-agent: Sogou web spider
 disallow: /
 
User-agent: SpamBayes
 disallow: /
 
User-agent: SpamBayes/1.1a3+
 disallow: /
 
User-agent: spbot
 disallow: /
 
User-agent: spbot/4.4.2
 disallow: /
 
User-agent: spiderbot
 disallow: /
 
User-agent: SpiderLing
 disallow: /
 
User-agent: Spiderbot/Nutch-1.7
 disallow: /
 
User-agent: spray-can
 disallow: /
 
User-agent: spray-can/1.2.1
 disallow: /
 
User-agent: SSG/3.0
 disallow: /
 
User-agent: Statastico
 disallow: /
 
User-agent: Statastico/4.0
 disallow: /
 
User-agent: Steeler
 disallow: /
 
User-agent: Steeler/3.5
 disallow: /
 
User-agent: Stratagems Kumo
 disallow: /
 
User-agent: Stratagems
 disallow: /
 
User-agent: StudioFACA Search
 disallow: /
 
User-agent: StudioFACA
 disallow: /
 
User-agent: sukibot
 disallow: /
 
User-agent: sukibot_heritrix
 disallow: /
 
User-agent: sukibot_heritrix/3.1.1
 disallow: /
 
User-agent: SurveyBot
 disallow: /
 
User-agent: Synapse
 disallow: /
 
User-agent: Synthesio Crawler release MonaLisa
 disallow: /
 
User-Agent: tbot-nutch/Nutch-1.10
 disallow: /
 
User-Agent: Traackr.com bot
 disallow: /
 
User-Agent: trendictionbot
 disallow: /
 
User-Agent: Trendiction-Bot
 disallow: /
 
User-Agent: TrueBot
 disallow: /
 
User-Agent: TrueBot/1.0
 disallow: /
 
User-Agent: TulipChain/5.xx
 disallow: /
 
User-Agent: TWMBot/0.1
 disallow: /
 
User-Agent: Typhoeus
 disallow: /
 
User-Agent: UCMore Crawler App
 disallow: /
 
User-Agent: uMBot-LN
 disallow: /
 
User-Agent: uMBot-LN/1.0
 disallow: /
 
User-Agent: updown_tester
 disallow: /
 
User-Agent: URLChecker
 disallow: /
 
User-agent: V1.0/1.2
 disallow: /
 
User-agent: w3af.org
 disallow: /
 
User-agent: WASALive
 disallow: /
 
User-agent: WASALive-Bot
 disallow: /
 
User-agent: vBSEO
 disallow: /
 
User-agent: WBSearchBot
 disallow: /
 
User-agent: WBSearchBot/1.1
 disallow: /
 
User-agent: WeAreNotEvil
 disallow: /
 
User-agent: WebAlta
 disallow: /
 
User-agent: WebAlta Crawler
 disallow: /
 
User-agent: Web corpus crawler
 disallow: /
 
User-agent: WebCookies
 disallow: /
 
User-agent: WebCookies/1.0
 disallow: /
 
User-agent: WebCopier vx.xa
 disallow: /
 
User-agent: Webnest 0.9
 disallow: /
 
User-agent: WebQL
 disallow: /
 
User-agent: Webscout
 disallow: /
 
User-agent: Webscout/1.0
 disallow: /
 
User-agent: Web-sniffer
 disallow: /
 
User-agent: Web-sniffer/1.1.0
 disallow: /
 
User-agent: Webster Pro V3.4
 disallow: /
 
User-agent: WebTarantula.com Crawler
 disallow: /
 
User-agent: WeCrawlForThePeace
 disallow: /
 
User-agent: VegeBot
 disallow: /
  
 User-agent: Vegi bot
 disallow: /
 
User-agent: WeLikeLinks
 disallow: /
 
User-agent: VeriCiteCrawler
 disallow: /
 
User-agent: VeriCiteCrawler/Nutch-1.9
 disallow: /
 
User-agent: WhatWeb
 disallow: /
 
User-agent: WhatWeb/0.4.8-dev
 disallow: /
 
User-agent: Visited by http://tools.geek-tools.org
 disallow: /
 
User-agent: Voila Robot
 disallow: /
 
User-agent: voltron
 disallow: /
 
User-agent: woobot
 disallow: /
 
User-agent: woobot/1.1
 disallow: /
 
User-agent: woobot/2.0
 disallow: /
 
User-agent: Vorboss Web Crawler
 disallow: /
 
User-agent: Vorboss Web Crawler/Nutch-2.3
 disallow: /
 
User-agent: WorldBrewBot
 disallow: /
 
User-agent: WorldBrewBot/2.1
 disallow: /
 
User-agent: worldwebheritage.org
 disallow: /
 
User-agent: worldwebheritage.org/1.0
 disallow: /
 
User-agent: wscheck.com
 disallow: /
 
User-agent: wscheck.com/1.0.0
 disallow: /
 
User-agent: www.deadlinkchecker.com
 disallow: /
 
User-agent: www.petitsage.fr site detector 0.4
 disallow: /
 
User-agent: WWW-Mechanize
 disallow: /
 
User-agent: WWW-Mechanize/1.74
 disallow: /
 
User-agent: Xenu Link Sleuth
 disallow: /
 
User-agent: Xenu's Link Sleuth
 disallow: /
 
User-agent: XoviBot
 disallow: /
 
User-agent: XoviBot/2.0
 disallow: /
 
User-agent: XSpider
 disallow: /
 
User-agent: Yandex Robot
 disallow: /
 
User-agent: Yandex
 disallow: /
 
User-agent: Yetibot
 disallow: /
 
User-agent: YisouSpider
 disallow: /
 
User-agent: yoozBot
 disallow: /
 
User-agent: yoozBot-2.2
 disallow: /
 
User-agent: zgrab/0.x
 disallow: /
 
User-agent: zzabmbot
 disallow: /
 
User-agent: zzabmbot/1.0
 disallow: /