# Borrowed from http://www.diveintomark.org/robots.txt User-agent: * Disallow: /cgi-sys/ Disallow: /images/ Disallow: korean.php User-agent: Zao User-agent: semanticdiscovery User-agent: PubCrawl User-agent: TurnitinBot User-agent: NPbot User-agent: psbot User-agent: baiduspider User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm) User-agent: larbin User-agent: NationalDirectory User-agent: LNSpiderguy User-agent: Teleport User-agent: MIIxpc User-agent: asterias User-agent: lwp-trivial User-agent: LinkWalker User-agent: cosmos User-agent: MSIECrawler User-agent: sitecheck.internetseer.com User-agent: pompos User-agent: Generic User-agent: WebSearchBench User-agent: almaden User-agent: k2spider User-agent: curl User-agent: Wget User-agent: QuepasaCreep User-agent: grub-client User-agent: grub User-agent: Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org) User-agent: Googlebot-Image User-agent: TrackBack Disallow: /