File: www.fark.com

package info (click to toggle)
python-protego 0.5.0%2Bdfsg-1
  • links: PTS, VCS
  • area: main
  • in suites: forky, sid
  • size: 30,052 kB
  • sloc: python: 1,579; perl: 190; cpp: 33; sh: 4; makefile: 3
file content (95 lines) | stat: -rw-r--r-- 2,647 bytes parent folder | download | duplicates (4)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
User-agent: Googlebot
Disallow: /nospam
Disallow: /nospam/
Disallow: /nomirror
Disallow: /nomirror/
Disallow: /admin/admin.php
Disallow: /submit
Disallow: /comments/8008135
Disallow: /confirm
Disallow: /unsub
Disallow: /passwordreset
Disallow: /ajax
Disallow: /login
Disallow: /archives/index-*
Disallow: /archives/index.1*
Disallow: /archives/index.2*
Disallow: /*/archives/index-*
Disallow: /*/archives/index.1*
Disallow: /*/archives/index.2*
Allow: /users
Allow: /cgi/users.pl
Allow: /cgi/fark/users.pl
Allow: /comments
Allow: /cgi/comments.pl
Allow: /cgi/fark/comments.pl
Disallow: /cgi/
User-agent: Mediapartners-Google
Disallow: /nospam
Disallow: /nospam/
Disallow: /nomirror
Disallow: /nomirror/
Disallow: /admin/admin.php
Disallow: /comments/8008135
Disallow: /passwordreset
Disallow: /ajax
Disallow: /login
Disallow: /archives/index-*
Disallow: /archives/index.1*
Disallow: /archives/index.2*
Disallow: /*/archives/index-*
Disallow: /*/archives/index.1*
Disallow: /*/archives/index.2*
Allow: /users
Allow: /cgi/users.pl
Allow: /cgi/fark/users.pl
Allow: /comments
Allow: /cgi/comments.pl
Allow: /cgi/fark/comments.pl
Allow: /confirm
Allow: /unsub
Allow: /submit
Allow: /cgi/submit.pl
Allow: /cgi/feedback.pl 
Allow: /cgi/forgotpassword.pl
Allow: /cgi/newuser.pl
Disallow: /cgi/
User-agent: *
Crawl-delay: 1
Disallow: /nospam
Disallow: /nospam/
Disallow: /nomirror
Disallow: /nomirror/
Disallow: /admin/admin.php
Disallow: /submit
Disallow: /users
Disallow: /comments/8008135
Disallow: /confirm
Disallow: /unsub
Disallow: /passwordreset
Allow: /ajax/headlines
Disallow: /ajax
Disallow: /login
Disallow: /archives/index-*
Disallow: /archives/index.1*
Disallow: /archives/index.2*
Disallow: /*/archives/index-*
Disallow: /*/archives/index.1*
Disallow: /*/archives/index.2*
Allow: /comments
Allow: /cgi/comments.pl
Allow: /cgi/fark/comments.pl
Disallow: /cgi/
# $Id: robots.txt 28844 2018-07-24 21:43:04Z mandrews $
#
# IMPORTANT NOTE:
# Fark user profiles have a meta tag on the page to tell search engines to NOT
# index them.  But to read the meta tag, the engines have to be able to crawl
# the page.  A disallow means "don't crawl", NOT "don't index" -- if some other
# site has a link to a URL we have in our disallow list, search engines may
# still index it anyway.  So, counterintuitively, the reason we allow
# Googlebot to crawl user profiles is so that they WON'T index them.
# Our intent is that Fark.com user profiles NOT appear in search engines.
# This is the same reason we allow /go and /goto through here, as well as
# /api and /ajax.  We block all these from indexing with either the meta tag
# or the X-Robots-Tag header.