Robots.txt Generator

Build a robots.txt file with user-agent presets, custom allow/disallow rules, crawl delay, and sitemap URL. Includes AI crawler presets.

⚠️ Important: A robots.txt file only suggests that crawlers not access certain paths. It does not remove pages from search results. To prevent indexing, use a noindex meta tag or X-Robots-Tag header.

User-Agent Configuration

Allow / Disallow Rules

AI Crawler Toggles

Quickly block or allow AI crawlers that scrape content for training data. These are added alongside your main rules.

Additional Settings

Leave empty to omit. Note: Googlebot ignores crawl-delay.

Generated robots.txt

# Configure rules above to generate robots.txt

About robots.txt

The robots.txt file is placed in your website's root directory and tells search engine crawlers which pages they can and cannot request. It uses the Robots Exclusion Protocol (REP), a standard respected by all major search engines.

Key Facts

  • robots.txt is a request, not a rule — malicious crawlers can ignore it.
  • It does not prevent indexing — use noindex for that.
  • Blocked pages may still appear in search results if linked from other sites.
  • Place your robots.txt at the root: https://example.com/robots.txt.
  • Each user-agent block must start with User-agent: followed by Disallow: or Allow: directives.
  • Use Disallow: / to block an entire site for a specific bot.