1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
|
User-agent: Googlebot
Disallow: /nospam
Disallow: /nospam/
Disallow: /nomirror
Disallow: /nomirror/
Disallow: /admin/admin.php
Disallow: /submit
Disallow: /comments/8008135
Disallow: /confirm
Disallow: /unsub
Disallow: /passwordreset
Disallow: /ajax
Disallow: /login
Disallow: /archives/index-*
Disallow: /archives/index.1*
Disallow: /archives/index.2*
Disallow: /*/archives/index-*
Disallow: /*/archives/index.1*
Disallow: /*/archives/index.2*
Allow: /users
Allow: /cgi/users.pl
Allow: /cgi/fark/users.pl
Allow: /comments
Allow: /cgi/comments.pl
Allow: /cgi/fark/comments.pl
Disallow: /cgi/
User-agent: Mediapartners-Google
Disallow: /nospam
Disallow: /nospam/
Disallow: /nomirror
Disallow: /nomirror/
Disallow: /admin/admin.php
Disallow: /comments/8008135
Disallow: /passwordreset
Disallow: /ajax
Disallow: /login
Disallow: /archives/index-*
Disallow: /archives/index.1*
Disallow: /archives/index.2*
Disallow: /*/archives/index-*
Disallow: /*/archives/index.1*
Disallow: /*/archives/index.2*
Allow: /users
Allow: /cgi/users.pl
Allow: /cgi/fark/users.pl
Allow: /comments
Allow: /cgi/comments.pl
Allow: /cgi/fark/comments.pl
Allow: /confirm
Allow: /unsub
Allow: /submit
Allow: /cgi/submit.pl
Allow: /cgi/feedback.pl
Allow: /cgi/forgotpassword.pl
Allow: /cgi/newuser.pl
Disallow: /cgi/
User-agent: *
Crawl-delay: 1
Disallow: /nospam
Disallow: /nospam/
Disallow: /nomirror
Disallow: /nomirror/
Disallow: /admin/admin.php
Disallow: /submit
Disallow: /users
Disallow: /comments/8008135
Disallow: /confirm
Disallow: /unsub
Disallow: /passwordreset
Allow: /ajax/headlines
Disallow: /ajax
Disallow: /login
Disallow: /archives/index-*
Disallow: /archives/index.1*
Disallow: /archives/index.2*
Disallow: /*/archives/index-*
Disallow: /*/archives/index.1*
Disallow: /*/archives/index.2*
Allow: /comments
Allow: /cgi/comments.pl
Allow: /cgi/fark/comments.pl
Disallow: /cgi/
# $Id: robots.txt 28844 2018-07-24 21:43:04Z mandrews $
#
# IMPORTANT NOTE:
# Fark user profiles have a meta tag on the page to tell search engines to NOT
# index them. But to read the meta tag, the engines have to be able to crawl
# the page. A disallow means "don't crawl", NOT "don't index" -- if some other
# site has a link to a URL we have in our disallow list, search engines may
# still index it anyway. So, counterintuitively, the reason we allow
# Googlebot to crawl user profiles is so that they WON'T index them.
# Our intent is that Fark.com user profiles NOT appear in search engines.
# This is the same reason we allow /go and /goto through here, as well as
# /api and /ajax. We block all these from indexing with either the meta tag
# or the X-Robots-Tag header.
|