User-agent: gigabot Crawl-delay: 120 User-agent: googlebot Crawl-delay: 30 User-agent: msnbot Crawl-delay: 120 User-agent: teoma Crawl-delay: 120 User-agent: slurp Crawl-delay: 120 # Define access-restrictions for robots/spiders # http://www.robotstxt.org/wc/norobots.html User-Agent: Googlebot Disallow: /*sendto_form$ Disallow: /*folder_factories$ User-agent: * Disallow: /*sendto_form$ Disallow: /*folder_factories$