File: www.goodreads.com

package info (click to toggle)
python-protego 0.6.0%2Bdfsg-1
  • links: PTS, VCS
  • area: main
  • in suites: forky, sid
  • size: 30,048 kB
  • sloc: python: 1,492; perl: 190; cpp: 33; sh: 4; makefile: 3
file content (44 lines) | stat: -rw-r--r-- 1,731 bytes parent folder | download | duplicates (4)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
User-agent: *
Disallow: /about/team_member/
Disallow: /admin
Disallow: /api
Disallow: /blog/list_rss
Disallow: /book/reviews/
Disallow: /book_link/follow/
Disallow: /buy_buttons/
Disallow: /ebooks
Disallow: /event/show/
Disallow: /home/index_rss
Disallow: /oggiPlayerLoader.htm
Disallow: /photo/group/
Disallow: /quotes/list_rss
Disallow: /reader
Disallow: /review/list_rss
Disallow: /review/rate
Disallow: /shelf/user_shelves
Disallow: /story
Disallow: /tooltips
Disallow: /track
Disallow: /trivia/answer
Disallow: /user/updates_rss

Sitemap: https://www.goodreads.com/siteindex.author.xml
Sitemap: https://www.goodreads.com/siteindex.author_community_question.xml
Sitemap: https://www.goodreads.com/siteindex.award.xml
Sitemap: https://www.goodreads.com/siteindex.blog.xml
Sitemap: https://www.goodreads.com/siteindex.book_community_question.xml
Sitemap: https://www.goodreads.com/siteindex.genre.xml
Sitemap: https://www.goodreads.com/siteindex.giveaway.xml
Sitemap: https://www.goodreads.com/siteindex.group.xml
Sitemap: https://www.goodreads.com/siteindex.index.xml
Sitemap: https://www.goodreads.com/siteindex.interview.xml
Sitemap: https://www.goodreads.com/siteindex.list.xml
Sitemap: https://www.goodreads.com/siteindex.quiz.xml
Sitemap: https://www.goodreads.com/siteindex.quote.xml
Sitemap: https://www.goodreads.com/siteindex.quote_tag.xml
Sitemap: https://www.goodreads.com/siteindex.related_work.xml
Sitemap: https://www.goodreads.com/siteindex.topic.xml
Sitemap: https://www.goodreads.com/siteindex.trivia.xml
Sitemap: https://www.goodreads.com/siteindex.user.xml
Sitemap: https://www.goodreads.com/siteindex.video.xml