Robots.txt Rules ConfiguratorGenerate clean, Googlebot-compliant crawl directives and sitemap tags for your website. Default Crawl Access (All Bots) Allow All Web CrawlersDisallow All Crawlers (Block Indexing) Crawl-Delay (Seconds) No Delay (Recommended)5 Seconds10 Seconds20 Seconds XML Sitemap URL Restricted / Blocked Directories (one per line) Generated robots.txt Preview Copied to clipboard! Copy Robots.txt Download .txt File