Private technical SEO analysis

Crawl-Versus-Bot Gap Analyzer

Join what your crawler knows, what your sitemap declares, and what selected claimed bots requested. Find important no-hit URLs, requests outside inventory, and observed waste patterns without uploading raw logs.

Browser-only processingMultiple filesCSV, JSON, Markdown exportsNo paid API

Add logs and preview the parser

Import multiple rotated files or paste a small sample. Raw lines are aggregated inside a Web Worker and are not kept in page state.

Apache, Nginx, CSV/custom-delimited, GZIP, or ZIP. Up to 50 MB compressed, 256 MB expanded per archive, and 100 ZIP entries.

Processed locally in your browser. Your files are not uploaded.

No log files selected.

Useful for a parser preview or a small analysis. Use files for large jobs.

Add crawl, sitemap, and priority inventory

URL columns are auto-detected. Open mapping when your export uses different headers.

Crawler CSV Recommended

Processed locally in your browser. Your files are not uploaded.

Paste CSV instead

Sitemap XML or CSV

Processed locally in your browser. Your files are not uploaded.

Paste XML or CSV instead

Priority or performance CSV Optional

Processed locally in your browser. Your files are not uploaded.

Paste CSV instead
Edit crawl and priority column mapping

Choose scope, bot groups, and normalization

These settings define what can match. Exported JSON records the active settings for reproducibility.

Required for path-only request targets and site-scope filtering.

Analysis window

Leave blank to use the full imported coverage.

Request methods
URL normalization

Comma-separated. Raw targets remain available as evidence examples.

One group per line: Group name|token,token. Tokens are literal case-insensitive substrings, not regular expressions.

Run the local join

Load sample data for a complete demonstration, or analyze your selected files and settings.

URL evidence

How the analysis works

  1. Parse safely. The worker reads selected text, GZIP, or ZIP entries, validates archive limits, parses mapped fields, and reports excluded-line reasons.
  2. Normalize deterministically. Request targets and inventory URLs use the same origin, query, port, fragment, and trailing-slash settings while raw examples remain evidence.
  3. Aggregate early. Requests are reduced by normalized URL and bot group before the join, including first and last request, unique days, status mix, bytes, response time, referrers, and daily counts.
  4. Join evidence. Aggregates are compared with crawl, sitemap, and optional business or traffic signals. Priority remains decomposable into named evidence.
  5. Export actions. Active tables export formula-safe CSV; JSON contains normalized settings and aggregates rather than raw logs; Markdown includes caveats, examples, and validation actions.

What this tool does not prove

  • It does not verify crawler identity through live DNS.
  • It does not prove that a search engine never requested, indexed, or ranked a URL outside the imported coverage.
  • It does not fetch logs, crawl your site, connect to Search Console, or store a project remotely.
  • It does not estimate a precise crawl budget or promise that cleanup changes will improve rankings.
  • Custom formats require one logical record per line; multiline quoted fields and arbitrary user regular expressions are deliberately unsupported.

Frequently Asked Questions

Does this tool upload my server logs?

No. Files and pasted data are processed locally in your browser. The analyzer does not send imported URLs, log lines, user agents, titles, or issue examples to the application server, analytics providers, or third parties.

Does a Googlebot user-agent prove the request came from Google?

No. User-agent strings can be spoofed. Unless your imported custom log data includes an independently verified bot flag, the tool labels crawler identity as a claimed user agent. The MVP does not perform reverse DNS verification.

What log formats can I import?

The analyzer accepts Apache Common or Combined and compatible Nginx lines, CSV-style CDN logs with editable field mapping, and custom-delimited logs. Log files may be plain text, GZIP, or ZIP. Unsupported or malformed lines are counted with a rejection reason.

What does a no-hit gap mean?

It means no normalized request from the selected claimed user-agent groups matched that URL in the imported log window. It does not prove that Google never crawled the URL, that retention is complete, or that the URL is not indexed.

How are waste opportunities calculated?

The tool counts observed imported requests associated with redirects, errors, parameters, assets, canonicalized URLs, non-indexable URLs, or conservative legacy patterns. It does not invent a crawl-budget number or promise a specific saving.

How large can imported files be?

Each compressed archive is limited to 50 MB, expanded archive content to 256 MB, and ZIP archives to 100 entries. Plain log files are limited to 256 MB each. Split larger logs into multiple rotated files. The Web Worker aggregates as it parses so raw lines do not enter page state.

Reviewed Jul 2026 · Sources and limitations

Review details: 2026-07-13 · Marc LaClear · v1.0

Reference sources:

Known limits:

  • Crawler grouping is based on claimed user-agent strings unless a custom independently verified flag is mapped. The tool does not perform reverse DNS.
  • Results cover only the imported retention window and can be incomplete when logs are sampled, transformed, missing hosts, or missing CDN/origin traffic.
  • ZIP entries using encryption, unsupported compression methods, or trailing data descriptors must be extracted locally before import.
  • Redirect chains are reported only when crawler data supplies enough status and destination evidence to follow each hop.
  • Observed request concentration is evidence from the imported logs, not a guaranteed crawl-budget saving, indexing outcome, or ranking result.

Report an issue with this tool