# https://en.wikipedia.org/wiki/Robots_exclusion_standard -*-conf-colon-*- # http://www.robotstxt.org/orig.html (original site, now ad infested) # # root@workstation> e /home/mike/robots.txt /mnt/lan/server/root/var/www/list/htdocs/robots.txt # # Blank lines are record separators; I mark them explicitly below. # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # a l l h o s t s # ─────────────────── # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ↰ Blank line is record separator # Temporarily block all robots # ────────────────────────────── # #User-agent: * #Disallow: / # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # r e l u k . c a # ───────────────── # ━━━━━━━━━━━━━━━━━━━━━━━━ ↰ Blank line is record separator # Allow diagnostic tools that I sometimes use # ──────────────────────── # User-agent: Browsershots Crawl-delay: 1 Disallow: # Disallow nothing, that is. # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ↰ Blank line is record separator # Allow the Internet Archiver (Wayback Machine) # ───────────────────────────── # I rely on it to safeguard the moral rights to some of my work. # See e.g. `~/work/ethics/normative/notes.brec` § safeguarding moral rights. User-agent: ia_archiver Crawl-delay: 1 Disallow: # Disallow nothing, that is. # ━━━━━━━━━━━━━━ ↰ Blank line is record separator # Allow others depending on requested path # ────────────── # User-agent: * Crawl-delay: 1 # # H h h h h h h h h h h h h h h h h h h h h h h h h h # Disallow: /home/mike/ # Redundant alias. # # # P p p p p p p p p p p p p p p p p p p p p p p p p p # # Redundant. Disallow: /project/_/archive/autonomy/other/ Disallow: /project/_/archive/autonomy/wiki/ # # Apparently the Javadoc compiler is botching a large number of external hyperlinks in *class use* pages: # Google’s *search console* complains about broken links. E.g. see the external links of # `http://reluk.ca/project/_/archive/lib-gwt-svg/release/javadoc-0.5.5+mca.1/org/vectomatic/dom/svg/itf/class-use/ISVGTransformable.html`. Disallow: /project/_/archive/lib-gwt-svg/release/javadoc- # Disallow: /project/Votorola/_/javadoc/src-html/ # Redundant. # # # S s s s s s s s s s s s s s s s s s s s s s s s s s # # Presently these proper names are equivalent to `server` and `workstation`, where robots instead should # crawl in order to avoid churn of their indeces on the next proper name change. Disallow: /sys/computer/Halfpenny/ Disallow: /sys/computer/Primeval/ # # Redundant. Disallow: /sys/computer/server/home/v/votorola/code/ Disallow: /sys/computer/server/home/v/votorola/in/ Disallow: /sys/computer/server/home/v/votorola/out/ Disallow: /sys/computer/server/home/voff/votorola/code/ Disallow: /sys/computer/server/home/voff/votorola/in/ Disallow: /sys/computer/server/home/voff/votorola/out/ # # Redundant and large. Disallow: /sys/computer/server/var/cache/pollwiki/backup-partial/wiki-dump-current.xml Disallow: /sys/computer/server/var/cache/pollwiki/backup-partial/wiki-dump-full.xml # # Redundant. Disallow: /sys/computer/server/var/cache/votorola/v/in/ Disallow: /sys/computer/server/var/cache/votorola/v/out/ Disallow: /sys/computer/server/var/www/public/htdocs/mediawiki-c/extensions Disallow: /sys/computer/server/var/www/public/htdocs/mediawiki-c/includes Disallow: /sys/computer/workstation/home/vdev/votorola/code Disallow: /sys/computer/workstation/home/voff/votorola/code # Disallow: /sys/computer/workstation/opt/gwt/2.8.0+mca.1/doc/javadoc/ # Redundant for the most part. # # Redundant. Disallow: /sys/computer/workstation/opt/javamail-1.4/docs/javadocs/ Disallow: /sys/computer/workstation/opt/openid4java-0.9.6.662/apidoc/ Disallow: /sys/computer/workstation/opt/wicket-1.5.4/apidocs/ # # # V v v v v v v v v v v v v v v v v v v v v v v v v v # Disallow: /var/cache/rhinohide-demo/_signed/ # Redundant. # # Redundant, [crawl] the released (cache) of `textbender/a/b/rhinohideDemo` only, not the source, # nor other released files. Cannot soft-link-include released files in source, # because they have relative links to built jars. [What?] Disallow: /var/cache/rhinohide-demo/textbender/_/ Disallow: /var/cache/rhinohide-demo/textbender/a/b/base/ Disallow: /var/cache/rhinohide-demo/textbender/a/b/demo/ Disallow: /var/cache/rhinohide-demo/textbender/a/b/rhinohide/ # # Extinct targets, referents of `_source.txt` files. Disallow: /var/cache/rhinohide-demo/textbender/a/b/rhinohideDemo/_/_/ # # Redundant, [crawl] the released (cache) of `textbender/a/b/rhinohideDemo` only, not the source, # nor other released files. Cannot soft-link-include released files in source, # because they have relative links to built jars. [What?] Disallow: /var/cache/rhinohide-demo/textbender/a/r/ Disallow: /var/cache/rhinohide-demo/textbender/a/u/ Disallow: /var/cache/rhinohide-demo/textbender/d/ Disallow: /var/cache/rhinohide-demo/textbender/g/ Disallow: /var/cache/rhinohide-demo/textbender/o/ # # Redundant, [crawl] the released (cache) of `textbender/a/b/demo` only, not the source, # nor other released files. So found docs compatible with current release. [What?] Disallow: /var/cache/textbender-demo/textbender/_/ Disallow: /var/cache/textbender-demo/textbender/a/b/base/ Disallow: /var/cache/textbender-demo/textbender/a/b/rhinohide/ Disallow: /var/cache/textbender-demo/textbender/a/b/rhinohideDemo/ Disallow: /var/cache/textbender-demo/textbender/a/r/ Disallow: /var/cache/textbender-demo/textbender/a/u/ Disallow: /var/cache/textbender-demo/textbender/d/ Disallow: /var/cache/textbender-demo/textbender/g/ Disallow: /var/cache/textbender-demo/textbender/o/ # # Redundant. Disallow: /var/cache/textbender-javadoc/src-html/ Disallow: /var/cache/textbender-javadoc/textbender/a/u/brancher/ Disallow: /var/cache/textbender-javadoc/textbender/a/u/lineCode/ Disallow: /var/cache/textbender-javadoc/textbender/a/u/lineCoder/ Disallow: /var/cache/textbender-javadoc/textbender/a/u/shadowPoint/ Disallow: /var/cache/textbender-javadoc/textbender/a/u/shadowSelect/ Disallow: /var/cache/textbender-javadoc/textbender/a/u/fragmentTransfer/ Disallow: /var/cache/textbender-javadoc/textbender/a/u/transfer/SourceSeries Disallow: /var/cache/textbender-javadoc/textbender/a/u/transfer/SP Disallow: /var/cache/textbender-javadoc/textbender/a/r/page/navmod/ Disallow: /var/cache/textbender-javadoc/textbender/o/browser/ Disallow: /var/repo/git/ Disallow: /var/repo/hg/ # Disallow: /var/post/ # Volatile. # # # W w w w w w w w w w w w w w w w w w w w w w w w w w # # In the original wiki, these requests were merely redirected to `/mediawiki/images/`. # Here they are copied as files by `http://reluk.ca/project/Votorola/b/wiki-copy/wget`. # In any case, they are redundant with their canonical locations under `/mediawiki/images/`. Disallow: /w/Special:FilePath/