How come search engines don't crawl the list and I can't find 'robots.txt'?

• To: [email protected]
• Subject: How come search engines don't crawl the list and I can't find 'robots.txt'?
• From: [email protected] (Linux User)
• Date: Sun, 13 Apr 2014 20:41:33 +0000
• In-reply-to: <CAJVRA1Qg_PLXCStjPcWkHU4TcGNvG6m8_O79M0d9AH_nXxM5VA@mail.gmail.com>
• References: <[email protected]$> <CAJVRA1Qg_PLXCStjPcWkHU4TcGNvG6m8_O79M0d9AH_nXxM5VA@mail.gmail.com>

On Sun, 13 Apr 2014 07:08:27 -0700
coderman <[email protected]> wrote:

> On Sun, Apr 13, 2014 at 6:53 AM, Georgi Guninski
> <[email protected]> wrote:
> > AFAICT the list archives don't show in search engines
> > and in addition don't see /robots.txt.
> >
> > How so?
> 
> back when traffic was handled on the al-qaeda.net domain, indexing by
> western search engines was considered to be "material support to
> Al-qaeda"...
> 
> now they've just forgotten to undo their filters.
> 
> ;)



https://www.google.com/search?q=test+site%3Acpunks.org%2Fpipermail%2Fcypherpunks%2F


the archives are indeed indexed?

• Follow-Ups:
  • How come search engines don't crawl the list and I can't find 'robots.txt'?
    • From: [email protected] (Georgi Guninski)

• References:
  • How come search engines don't crawl the list and I can't find 'robots.txt'?
    • From: [email protected] (Georgi Guninski)
  • How come search engines don't crawl the list and I can't find 'robots.txt'?
    • From: [email protected] (coderman)

• Prev by Date: NSA good guys
• Next by Date: [cryptome] snowman news
• Previous by thread: How come search engines don't crawl the list and I can't find 'robots.txt'?
• Next by thread: How come search engines don't crawl the list and I can't find 'robots.txt'?
• Index(es):
  • Date
  • Thread