1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
|
User-agent: 008
User-agent: ahrefsbot
User-agent: bygglobot
User-agent: comodospider
User-agent: discobot
User-agent: docomo
User-agent: dotbot
User-agent: exabot
User-agent: ezooms
User-agent: findlinks
User-agent: garlikcrawler
User-agent: hapax newscrawler
User-agent: huaweisymantecspider
User-agent: mj12bot
User-agent: mlbot
User-agent: nerdbynature.bot
User-agent: nigma.ru
User-agent: pflab
User-agent: psbot
User-agent: purebot
User-agent: rankurbot
User-agent: scoutjet
User-agent: sistrix crawler
User-agent: sitebot
User-agent: speedy spider
User-agent: testnutch
User-agent: trends crawler
User-agent: unchaos real time trends bot
User-agent: web.idrc.ca/challenge
User-agent: webvac
Disallow: /
|