File: robots.txt

package info (click to toggle)
linkchecker 10.6.0-2
  • links: PTS, VCS
  • area: main
  • in suites: forky, sid
  • size: 3,132 kB
  • sloc: python: 13,154; makefile: 134; sh: 71; xml: 36; sql: 20; javascript: 19; php: 2
file content (14 lines) | stat: -rw-r--r-- 387 bytes parent folder | download | duplicates (6) 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
 
# A test robots.txt file. It resides in the root directory to be found
# by the unittest HTTP server.
# Note that while unit tests are running, the contents of this file
# can change.

User-agent: LinkChecker
Crawl-delay: 1
Disallow: /secret
# a comment line inside a rule, followed by non-ascii
Disallow: /umlaut/รถ

# The following rule should not match
User-agent: *
Crawl-delay: 100