## GENERAL SETTINGS User-agent: 360Spider Disallow: / User-agent: 80bot Disallow: / User-agent: Goodzer Disallow: / user-agent: WBSearchBot Disallow: / User-agent: ShopWiki Disallow: / User-agent: TwengaBot-2.0 Disallow: / User-agent: FatBot Disallow: / User-agent: Baiduspider Disallow: / ## Enable robots.txt rules for all crawlers User-agent: * ## Crawl-delay parameter: number of seconds to wait between successive requests to the same server. ## Set a custom crawl rate if you're experiencing traffic problems with your server. Crawl-delay: 60 ## DEVELOPMENT RELATED SETTINGS ## Do not check any generated pages Disallow: /* ### But allow the game pages Allow: /*-game$