# robots.txt # 15 april 2002 # laat alle zoekmachines toe User-agent: * Disallow: # Robots.txt Tutorial # http://www.searchengineworld.com/robots/robots_tutorial.htm # # Robots.txt Validator # http://www.searchengineworld.com/cgi-bin/robotcheck.cgi # # Robots Exclusion Meta Tag : Using robots metatags # http://searchengineworld.com/metatag/robots.htm # # Search Indexing Robots and Robots.txt, article at searchtools.com # http://www.searchtools.com/robots/robots-txt.html # # Root of Robots Exclusion Standard directory with some interesting files # http://info.webcrawler.com/mak/projects/robots/ # # Robots.txt : The Big Crawl # http://www.searchengineworld.com/misc/robots_txt_crawl.htm # # Robots.txt on 'big' websites # http://www.cnn.com/robots.txt # http://nl.altavista.com/robots.txt # http://www.google.com/robots.txt