A practical release check
How to compare two SEO crawl exports
- Create a trustworthy baseline. Use a crawl from before a migration, redesign, template update, content deployment, or other material change. If possible, use a crawl that was reviewed and accepted.
- Repeat the same crawl setup. Keep the crawler, start URL, rendering mode, authentication, robots behavior, include/exclude rules, user agent, and crawl limits consistent.
- Map equivalent fields. The URL is required. Map only columns that mean the same thing in both files; an “Indexability” label is not always equivalent to a robots-directive column.
- Review evidence, then validate live. Start with high-priority regressions and broad patterns. Confirm important changes in the live page, crawler, server response, Search Console, or deployment record before assigning a cause.
Worked crawl-comparison example
An HVAC company crawls its 480-URL website before and after a CMS theme release. The files match 463 URLs. The report does not call every difference an error; it separates changes that deserve different kinds of review.
| Page and field | Baseline | Current | Interpretation |
|---|---|---|---|
/services/furnace-repair/Indexability |
Indexable | Noindex | High-priority regression Confirm whether the directive was intentional. |
/financing/URL presence |
Not present | Present, 200 | Added URL Review indexability and internal discovery. |
/air-conditioning/Internal inlinks |
42 | 18 | Material decline Check changed navigation and template links. |
/about/Title |
About Lakeside Heating & Air | About Us | Review change Compare the intended title brief and live template. |
Reasonable conclusion: the noindex deserves immediate validation, the inlink decline suggests a template-level navigation check, and the title change needs editorial review. The export alone does not prove traffic loss or explain why the changes occurred.
What each crawl-comparison result means
- Added URL
- The normalized URL appears only in the current export. It may be a valid new page, an unintended crawl path, a parameter variation, or a scope difference.
- Removed URL
- The normalized URL appears only in the baseline. Check whether it was intentionally redirected, deleted, excluded, blocked, orphaned, or simply missed by a different crawl setup.
- High-priority regression
- A mapped signal moved in a direction that can obstruct access, indexability, consolidation, or page identity—for example, 200 to 404, indexable to noindex, or a changed canonical target.
- Material change
- A title, description, H1, word count, crawl depth, or inlink value crossed the selected comparison rule. The change needs context and is not automatically harmful.
- Improvement
- A mapped signal moved in a commonly safer direction, such as an error URL returning 200 again. Validate that the current state matches the intended implementation.
- Coverage warning
- The two exports may not be directly comparable because too few URLs matched, duplicate normalized URLs were found, rows were invalid, or useful columns were not mapped.
How matching and thresholds work
The checker does not create a vague 0–100 SEO score. It normalizes URLs using the settings you select, joins baseline and current rows by that normalized key, and compares only fields mapped in both files.
Exact and directional checks
URL presence, HTTP status, indexability, robots directives, canonical targets, titles, descriptions, and H1 values are compared directly. Directional labels describe review priority; they do not predict a ranking outcome.
Numeric-change checks
Word-count and inlink declines use the percentages you choose. Crawl-depth regressions use the selected level increase. The baseline value supplies the comparison reference.
Duplicate handling
If multiple rows collapse to one normalized URL, the report warns you instead of silently treating ambiguous rows as clean evidence. Review normalization and the raw export.
Common crawl-comparison mistakes
Changing the crawl setup
A JavaScript-rendered current crawl and an HTML-only baseline can make valid pages, links, canonicals, and content look added or removed. Match settings before interpreting the result.
Comparing incomplete exports
Row limits, filters, interrupted crawls, blocked sections, or exporting only “HTML” in one file can create a false coverage gap.
Normalizing URLs too aggressively
Query parameters, hostnames, protocols, and trailing slashes can represent intentional differences. Merge them only when they are equivalent for this site and test.
Treating every removal as broken
A removed URL may be intentionally redirected or consolidated. Check its live response, destination relevance, internal links, sitemap, and canonical signals.
Assigning a cause from one export
A crawl difference shows what changed in supplied data, not who changed it, when it changed, whether Google processed it, or what caused performance to move.
Fixing rows without checking the pattern
Many similar title, canonical, noindex, or depth changes often point to one template, CMS rule, or navigation deployment. Investigate the shared source first.
Practical use cases
- Website migrations: compare pre-launch and post-launch crawl coverage, statuses, canonicals, and indexability.
- CMS or theme releases: find template-wide metadata, robots, navigation, or content changes.
- Content deployments: confirm planned additions and removals without losing titles, H1s, or substantial copy.
- Technical SEO QA: turn a recrawl into an evidence-based developer or agency handoff.
- Recurring monitoring: repeat the same export comparison after major releases or scheduled quality checks.
- Incident review: connect a traffic-change timeline to measured crawl changes without claiming the crawl proves causation.
Limitations
- The checker analyzes only rows and columns present in the two files. It does not crawl, render, fetch, or discover missing URLs on its own.
- Different crawler settings, licenses, versions, user agents, rendering modes, authentication, robots handling, or crawl completion can produce false differences.
- A crawl export cannot confirm Google’s selected canonical, index status, rankings, clicks, conversions, backlinks, server-log activity, or business impact.
- Text and numeric thresholds are review heuristics, not Google requirements. A flagged change may be planned, beneficial, or irrelevant.
- Automatic column detection is based on common headings. Confirm mappings when headers are localized, renamed, or contain similar fields.
- Browser memory and performance vary by device. Very large enterprise crawls may need a desktop crawler, database, command-line workflow, or paid platform.
Frequently asked questions
Does this tool crawl my website?
No. It compares two crawl exports that you provide. It does not fetch, render, or crawl any URL, so it cannot discover a page missing from both files.
Are my crawl files uploaded or stored?
No. Parsing, URL matching, comparison, filtering, and exports happen in this browser tab. Your file contents are not sent to Your SEO Toolbox or a third-party API. Closing or resetting the page clears the in-memory analysis.
Which crawl-export formats and columns can I use?
Use CSV, TSV, or a supported delimited text export. Each file needs a URL column. Status, indexability, robots, canonical, title, meta description, H1, word count, crawl depth, and internal inlinks are optional, but a field can be compared only when an equivalent column is mapped in both files.
Can I compare Screaming Frog, Sitebulb, or other crawler exports?
Yes. Automatic detection recognizes common headings, and the manual mapper can pair differently named columns. For reliable evidence, keep crawl scope and settings comparable and make sure the mapped fields mean the same thing.
How are baseline and current URLs matched?
The URL column is cleaned and normalized according to the settings you select. Fragments are removed. Protocol, www hostnames, trailing slashes, and common tracking parameters are treated according to your choices. The report warns when normalization creates duplicate keys.
Is every removed URL a serious regression?
No. A page may have been intentionally redirected, consolidated, removed, excluded by a crawl rule, or replaced. Treat “removed” as a validation queue, then check live status, destination, internal links, sitemap inclusion, and business purpose.
Does a regression mean rankings or traffic will fall?
No. A regression label means a crawl signal moved in a potentially harmful direction. The report cannot predict how a search engine will process the page or whether the change affects traffic. Validate important findings with live checks and first-party data.
Can two exports have different column names?
Yes. Confirm each file’s mapping separately. The checker compares the meaning you assign to each column, not the literal header spelling.
Is every feature free?
Yes. File comparison, column mapping, normalization settings, finding filters, the copyable brief, and CSV, JSON, and Markdown downloads are free. No signup, email address, or paid API is required.
Related free SEO tools
Use a focused live-page check to validate the most important changes found in the crawl comparison.
Method, review details, and tool facts
The implementation uses deterministic browser-side parsing and comparison rules. Priority labels organize manual review; they are not ranking factors, Google rules, or guarantees. Product specification prepared July 14, 2026.
- Input
- Two crawl CSV or TSV exports
- Processing
- Browser only
- Account
- Not required
- Paid API
- None
- Output
- Prioritized findings, CSV, JSON, Markdown
- Data storage
- None
Useful references: Google crawling and indexing overview, robots meta and X-Robots-Tag guidance, canonicalization guidance, and MDN File API.