# See http://web.nexor.co.uk/mak/doc/robots/norobots.html for a description # of what this file is, and it's syntax. - CAD 4/30/95 # # Generally, it's: # # User-agent: agent-name # Disallow: directory on the server you wish to not have robots in # # To keep all robot's out of directory "blah", use: # # User-agent: * # Disallow: /blah/ # # CAD 4/30/95 I think that robots should stay our of the cgi-bin directory: User-agent: * Disallow: /web_new/ Disallow: /web_old/ Disallow: /torah/ Disallow: /cgi/ Disallow: /errors/ Disallow: /downloads/ Disallow: /logs/ Disallow: /chart/ Disallow: /BatMitzvah/ Disallow: /web_thru-05-2007/