✳flâneur — a map of the web's best reading

gwern.net/robots.txt

gwern.net · 81 words · saved by 1 readers

Sitemap: https://gwern.net/sitemap.xml User-agent: ia_archiver Disallow: / Allow: /modafinil Allow: /dnm-arrest User-agent: * Disallow: /fulltext # Hide copies: # Duplicate content is bad for SEO (and clutters search results), so no Markdown sources, WWW archives, metadata snippets, or link-bibliography compilations. Disallow: /*.md Disallow: /*.md.html # disallow syntax-highlighted versions of source code as duplicates: Disallow: /static/*.*.html # spurious google hits for filenames cluttering results: Disallow: /static/nginx/* Disallow: /static/redirect/* Disallow: /metadata/* Disallow: /metadata/annotation/backlink/* Disallow: /metadata/annotation/similar/* Disallow: /metadata/annotation/link-bibliography/* Disallow: /confidential/* Disallow: /private/* Disallow: /secret/* Disallow: /doc/www/* # # I disallow doc/*/index pages because those keep cluttering up Google Scholar # Disallow: /doc/*/index # Allow: /doc/rotten.com/*

Sitemap: https://gwern.net/sitemap.xml User-agent: ia_archiver Disallow: / Allow: /modafinil Allow: /dnm-arrest User-agent: * Disallow: /fulltext # Hide copies: # disallow syntax-highlighted versions of source code as duplicates: Disallow: /static/*.*.html # spurious google hits for filenames cluttering results: Disallow: /static/nginx/* Disallow: /static/redirect/* Disallow: /metadata/* Disallow: /metadata/annotation/backlink/* Disallow: /metadata/annotation/similar/* Disallow: /metadata/annotation/link-bibliography/* Disallow: /confidential/* Disallow: /private/* Disallow: /secret/* Disallow

Explore this link on the map →

related reading