Deployment risk review
Robots.txt Change Impact Simulator
Compare a current and proposed robots.txt file against a real URL inventory. See exactly which crawler decisions change, which rule wins, and which important pages or resources need review.
Add current and proposed files
Leave the current file empty for single-file audit mode.
Plain text, up to 1 MiB.
Parsed as text only. Uploaded content is never executed.
Current file review
Proposed file review
Add and map the URL inventory
Paste one URL per line or use CSV/TSV with optional crawl metadata.
Only scheme and host are used. Rows on another origin are clearly labeled.
Maximum 50,000 valid URLs, 20 MiB input, and 500,000 URL-agent evaluations per run.
Column mapping
Add inventory data to detect columns.
Select user-agent tokens
Tokens are editable comparison inputs, not claims about current crawler policies.
Evidence-based results
Change impact
| URL / evidence | User agent | Current | Proposed | Change / issues | Trace |
|---|
Rule coverage and removal candidates
Unused means the rule matched no supplied URL-agent decision. Redundant means removing it would not change supplied decisions. Broader inventories may produce different results.
Current file
Proposed file
Sitemap directive comparison
Export deployment evidence
Exports use the active search and change filters. CSV cells are protected against spreadsheet-formula injection.
Implemented semantics, limitations, and deployment use
How the simulator evaluates a change
The standards-versioned analyzer uses documented Google-style matching: the most specific matching user-agent token is selected, equally specific groups are merged, the URL path and query are matched case-sensitively, * matches any sequence, a final $ anchors the end, and the longest matching rule wins with Allow winning an equal-length tie.
Fragments are ignored because they are not sent in HTTP requests. Empty Allow and Disallow values have no access effect. Unsupported fields such as Crawl-delay and Host are preserved as diagnostics but do not receive invented access behavior.
What the result does not prove
The result does not predict indexing, ranking, crawler identity, or whether a crawler will honor the supplied file. Treat it as pre-deployment evidence and follow it with a live robots.txt request, representative URL tests, a crawl, and search-platform validation.
Reference specifications
Review the Robots Exclusion Protocol (RFC 9309) and Google robots.txt interpretation documentation before deploying crawler-specific changes.
Post-deployment checklist
- Verify the live
/robots.txtresponse, status, encoding, and final content. - Retest important newly blocked and newly allowed examples for every affected crawler token.
- Confirm CSS, JavaScript, image, font, API, and sitemap URLs remain intentionally accessible.
- Monitor crawl errors, server logs, and search-platform reports after release.
Frequently asked questions
Does an allowed result mean a URL will be indexed?
No. The simulator evaluates crawl access under the supplied robots.txt rules. Indexing also depends on page availability, canonicalization, page directives, content, links, and crawler decisions.
What matching rules does the simulator use?
The default mode selects the most specific matching user-agent group, merges equally specific groups, evaluates the URL path plus query string case-sensitively, supports * and a final $ anchor, and chooses the longest matching rule with Allow winning an equal-length tie.
Why does the tool warn about blocked noindex URLs?
If a crawler is blocked from requesting a URL, it may not be able to read page-level directives. The warning is a review prompt, not a prediction of indexing behavior.
Are uploaded files sent to the server?
No. Files and pasted data are parsed in your browser. The tool does not send file contents, URLs, rules, or result examples to the server or analytics.
What does a redundant rule candidate mean?
For the supplied inventory and selected user agents, removing that rule would not change an access decision. Test broader inventories before removing a production rule.
Can I use the tool without a current baseline file?
Yes. Leave the current file empty and provide the proposed file to run single-file audit mode.