File: www.news.com.au

package info (click to toggle)
python-protego 0.4.0%2Bdfsg-1
  • links: PTS, VCS
  • area: main
  • in suites: trixie
  • size: 30,200 kB
  • sloc: python: 1,598; perl: 190; cpp: 33; sh: 4; makefile: 3
file content (20 lines) | stat: -rw-r--r-- 382 bytes parent folder | download | duplicates (4) 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
 
User-agent: *

Disallow: /*/comments-*
Disallow: /*/print/
Disallow: /*SIT*
Disallow: /*.swf
Disallow: /printpage/
Disallow: /double-rainbow/
Disallow: /*-simplify/*
Disallow: /it-test-only/
Disallow: /enewsletters/

Sitemap: https://www.news.com.au/sitemap.xml
Sitemap: https://www.news.com.au/news-sitemap.xml

#Agent Specific Disallowed Sections

User-agent: NewsNow

Disallow: /