Robots.txt Tester

Paste your robots.txt, choose a user-agent and test URL path to see if it's allowed or blocked. Includes syntax validation, rule parsing, and group visualization.

Robots.txt Content

Enter your full robots.txt content. The tool automatically parses and validates it as you type.

Test a URL Path

Or type a custom user-agent below
Only the path part starting with /, e.g., /blog/post or /admin/login

Parsed Groups

Paste a robots.txt above to see parsed rule groups.

How Robots.txt Works

When a crawler visits your site, it first checks your robots.txt file at https://yoursite.com/robots.txt to see which paths it's allowed to request. The parser evaluates rules using the longest matching path principle: the most specific rule for the matching user-agent wins.

Why Robots.txt Matters for SEO

Properly configured robots.txt files are essential for technical SEO because they:

  • Manage crawl traffic — Prevent search engines from wasting resources on admin pages, duplicate content, or infinite URL spaces (search results, calendars, filters)
  • Control AI crawler access — Many AI training bots respect robots.txt directives, making it one method to opt out of having your content used for model training. For search-specific AI products, different bots (OAI-SearchBot, Claude-SearchBot) serve different purposes.
  • Direct crawlers to sitemaps — The Sitemap: directive in robots.txt is one of the primary ways search engines discover your XML sitemaps
  • Set crawl rate expectations — The Crawl-delay directive (supported by Bing, Yandex, and others) prevents server overload from aggressive crawlers. Google does not support Crawl-delay and automatically adjusts its crawl rate.

Common Syntax Rules

  • Each block starts with User-agent:
  • Disallow: blocks paths — Disallow: / blocks everything.
  • Allow: overrides a Disallow for more specific paths.
  • An empty Disallow: rule (no path after the colon) has no blocking effect. It does not cancel other rules in the group. Use Allow: / to explicitly grant access.
  • Crawl-delay: sets delay between requests. Google does not support this directive.
  • Sitemap: points to your XML sitemap location.
  • Lines starting with # are comments.
  • User-agent * acts as a wildcard matching all bots without specific rules.

Common Mistakes

  • Using noindex in robots.txt — The noindex directive is not part of the robots.txt standard. Use <meta name="robots" content="noindex"> or X-Robots-Tag: noindexHTTP header instead.
  • Blocking CSS/JS files — Google needs these for rendering. Blocking them can hurt your mobile-first indexing and rankings.
  • Using incorrect line endings — Use standard Unix (LF) or Windows (CRLF) line endings. The parser normalizes both; neither causes problems.
  • Placing Sitemap inside user-agent blocks — Sitemap directives should be at the file level, not inside specific user-agent groups.
  • Assuming specific user-agents inherit wildcard rules — Googlebot uses only its own rules and does NOT inherit restrictions from the wildcard (*) group.

Frequently Asked Questions

A robots.txt file is a text file placed at the root of a website (e.g., https://example.com/robots.txt) that tells web crawlers which parts of the site they are allowed or disallowed to access. It follows the Robots Exclusion Protocol and is the first file most crawlers request when visiting a site. It is a directive, not an enforcement mechanism — well-behaved crawlers obey it, but malicious bots may ignore it.
No — robots.txt blocks crawling, not indexing. If you disallow a page in robots.txt but an external site links to it, Google may still index the URL (without being able to see the content). To prevent indexing, use the noindex meta tag or X-Robots-Tag HTTP header instead. Robots.txt only controls whether crawlers can access the page to read its content.
Disallow tells crawlers which paths they cannot access. Allow explicitly permits access to a path, typically used to override a broader Disallow rule for a specific subdirectory or file. For example, you can Disallow the entire /private/ directory but Allow /private/public/ to grant selective access. When two rules conflict, the longest matching path wins (most specific rule takes precedence).
Order does not directly matter because the Robots Exclusion Protocol uses the longest matching path rule — the rule with the most specific (longest) path match wins, regardless of where it appears in the file. However, it is best practice to group rules by user-agent and place Allow rules after broader Disallow rules for readability and maintainability.
You typically need at least a wildcard (*) group for general rules, plus specific groups for bots you want to restrict differently. Common user-agents include Googlebot (Google Search), Bingbot (Bing), and purpose-specific crawlers like GPTBot/ClaudeBot (AI training), OAI-SearchBot/Claude-SearchBot (AI search), AhrefsBot/SemrushBot (SEO tools), and CCBot (dataset crawlers). Each group should be tailored to what that crawler needs or should not access.
Yes — Googlebot and most modern search engines support basic pattern matching in robots.txt rules. An asterisk (*) matches any sequence of characters (e.g., Disallow: /*.pdf$ blocks all PDF files). The dollar sign ($) denotes the end of a URL pattern. For example, Disallow: /*?session= blocks all URLs containing a session parameter. Note that not all search engines support pattern matching, and the syntax is more limited than full regular expressions.
To block AI training crawlers, add dedicated user-agent groups: User-agent: GPTBot followed by Disallow: /. You can also block Google-Extended (Google's AI training and grounding product token) the same way. Note that search-specific AI crawlers like OAI-SearchBot and Claude-SearchBot are separate from training crawlers — blocking them may remove your site from AI search results. Always test with a tool like this one to verify your rules work as expected.
Generally no — search engines like Google need CSS and JavaScript files to render pages correctly for their mobile-first indexing. Blocking these can harm your search rankings because Google cannot see the page as a user would. Only block static assets if they are in a private area (e.g., /admin/). For any other case, allow crawlers to access your CSS, JS, and image files.
An empty Disallow rule has no blocking effect. It does not erase or cancel other rules in the group. If no other matching rule blocks the URL, crawling is allowed. To explicitly allow everything, use Allow: / instead.

Suggested Workflow

Manage how search engines crawl your site with our Crawl Control Workflow — a step-by-step guide to configuring robots.txt, sitemaps, and crawl rate settings.

Reviewed Jun 2026 · Sources and limitations

Review details: 2026-06-12 · Marc LaClear · v1.2

Reference sources:

Known limits:

  • This tool parses and tests robots.txt rules based on pasted text — it does not fetch or validate live robots.txt files from your site.
  • The parser implements RFC 9309 rules plus Google-specific behavior. Results may differ from other parsers.
  • Some AI crawlers may ignore robots.txt directives entirely. A "blocked" result here does not guarantee the crawler will respect it.
  • Test results are diagnostic aids, not guarantees. Validate critical access rules with Google Search Console and live crawl data.

Report an issue with this tool