File: www.xing.com

package info (click to toggle)
python-protego 0.5.0%2Bdfsg-1
  • links: PTS, VCS
  • area: main
  • in suites: forky, sid
  • size: 30,052 kB
  • sloc: python: 1,579; perl: 190; cpp: 33; sh: 4; makefile: 3
file content (19 lines) | stat: -rw-r--r-- 416 bytes parent folder | download | duplicates (4) 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
 
User-agent: CrazyWebCrawler-Spider
Disallow: /

User-agent: *
Disallow: /img/users
Disallow: /app/search
Disallow: /cgi-bin/search.fpl
Disallow: /patterns
Disallow: /jobs/posting_clicks
Disallow: /user-ass/jobs/posting_pdf
Disallow: /publicsearch/
Disallow: /search/
Disallow: /people/search/
Disallow: /r/
Disallow: /xbp/search
Disallow: /profile/signup/
Disallow: /go/login
Disallow: /ads/
Disallow: /jobs/search/