File: crawlers-user-agents.data

package info (click to toggle)
modsecurity-crs 3.0.2-1~bpo9+1
  • links: PTS
  • area: main
  • in suites: stretch-backports
  • size: 1,064 kB
  • sloc: ansic: 727; perl: 428; python: 218; ruby: 69; makefile: 15
file content (34 lines) | stat: -rw-r--r-- 659 bytes parent folder | download | duplicates (6)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# Search engine crawlers and other bots

# site ripper
# http://www.softbytelabs.com/en/BlackWidow/
black widow
blackwidow
# crawler
# 2006
prowebwalker
# generic crawler
pymills-spider/
# SEO
# https://ahrefs.com/robot
AhrefsBot
# people database
# https://pipl.com/bot/
PiplBot
# advertising targeting
# https://www.grapeshot.com/crawler/
GrapeshotCrawler/2.0
grapeFX
# SEO
# http://www.searchmetrics.com/searchmetricsbot/
SearchmetricsBot
# SEO
# https://moz.com/help/guides/moz-procedures/what-is-rogerbot
rogerbot
# SEO
# http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
MJ12bot
# news service
Owlin bot
# misbehaving spider
Lingewoud-550-Spyder