Meta Robots Checker

Check meta robots directives and HTTP X-Robots-Tag headers. Detect noindex, nofollow, noarchive, nosnippet, and conflicts between HTML and HTTP-level directives — with scoring and actionable advice.

Check Robots Directives

The tool fetches the page and checks for meta robots tags, Googlebot-specific tags, and X-Robots-Tag HTTP headers.
Quick test:

About the Meta Robots Checker

This tool analyzes all robots directives affecting a page — from meta tags in the HTML to X-Robots-Tag HTTP headers. It detects conflicts, highlights Googlebot-specific rules, checks Bingbot and Yandex overrides, and gives a clear score-based assessment.

Comprehensive Analysis

Scans HTML meta robots, Googlebot meta, Bingbot/Yandex overrides, and X-Robots-Tag HTTP headers.

Conflict Detection

Identifies mismatches between HTML and HTTP-level directives that can cause unexpected indexing behavior.

Score & Prioritization

Get a numerical score with prioritized checks so you know exactly what to fix first.

Common Robots Directives

  • noindex — Do not show this page in search results
  • nofollow — Do not follow links on this page for ranking purposes
  • noarchive — Do not show a cached copy of this page
  • nosnippet — Do not show a text snippet or video preview in search results
  • max-snippet:N — Limit the text snippet to N characters
  • max-image-preview:N — Control image preview size (none/standard/large)
  • max-video-preview:N — Limit video preview duration in seconds
  • none — Equivalent to noindex, nofollow combined
  • unavailable_after:date — Stop showing this page in results after a specific date

Why Robots Directives Matter for SEO

Properly configured robots directives are fundamental to technical SEO. They control how search engines interact with your content and can have dramatic effects on your site's visibility in search results:

  • Prevent wasteful indexing — Block thin content, admin pages, and duplicate URLs from appearing in search results, preserving crawl budget for important pages.
  • Protect sensitive content — Use noindex to keep private areas, staging sites, and member-only areas out of search results.
  • Control link equity flow — Use nofollow on user-generated content, affiliate links, or paid links to prevent passing ranking signals.
  • Avoid accidental de-indexing — A single misconfigured X-Robots-Tag header can remove your entire site from Google. Regular audits prevent this.
  • Fine-tune search appearance — Use max-snippet and max-image-preview to control how your pages appear in search results without removing them entirely.

Best Practices

  • Every page should have one consistent set of directives — don't let HTML and HTTP headers disagree.
  • Use bot-specific tags only when needed — Googlebot, Bingbot, and Yandex meta tags should only override the general robots tag when necessary.
  • Never use noindex on pages that should rank — double-check before adding noindex to any production page.
  • To remove a web page from Google, allow Googlebot to crawl the URL and return a noindex meta tag or X-Robots-Tag header. Do not block that page in robots.txt while waiting for Google to process the noindex directive.
  • Audit regularly — Automatic updates, security plugins, and CDN configurations can silently inject unwanted X-Robots-Tag headers.
  • Use max-snippet instead of nosnippet if you want to control snippet length rather than removing snippets entirely.
  • Test changes — After modifying robots directives, use the URL Inspection Tool in Google Search Console to verify the change.

Meta Robots Examples: Indexable vs. Blocked

These examples show the difference between a page that is open to search engines and a page that intentionally blocks indexing.

Valid example

https://example.com/ is a simple public page that normally has no blocking meta robots tag. The expected result is indexable by default: no noindex, no nofollow, and no restrictive X-Robots-Tag header.

Broken example

Use a staging, thank-you, filtered search, or duplicate page on your own site that contains <meta name="robots" content="noindex, nofollow">. This should produce a blocked or high-risk result.

If you do not have a test page, inspect the source of a non-production page before using it so you know whether the noindex directive is really present.

Preview: What the Result Should Show

A strong result should show the final URL, the active meta robots and X-Robots-Tag directives, any bot-specific overrides, and a clear indexable-or-blocked summary.

How to Interpret the Result

  • No robots tag: search engines generally default to index and follow if nothing else blocks the page.
  • Noindex: the page is requesting removal from organic search results.
  • Nofollow: search engines are told not to follow links on the page for discovery or ranking signals.
  • HTML and HTTP conflict: assume the most restrictive directive wins.

Common Failure Cases

  • X-Robots-Tag added by server config: developers may not see it in page source because it is sent as an HTTP header.
  • Googlebot-specific override:Google may see different instructions than other crawlers.
  • Robots.txt confusion: robots.txt controls crawling; meta robots controls indexing after a crawler can see the page.

What warnings mean

Warnings mean the page may still be crawlable, but the indexing instructions are mixed, restrictive, or easy to misread. Treat them as a signal to confirm intent before rollout, especially on production templates and headers.

What this tool cannot know

  • Whether the page is blocked elsewhere by authentication, canonicals, redirects, or non-robots technical issues.
  • Whether Google has already processed a recent directive change; use URL Inspection for that confirmation.
  • Whether sitewide rules are intentionally segmenting different page groups; that requires template, CDN, and CMS review beyond one URL.

Download sample meta robots report

Frequently Asked Questions

Meta robots tags (and X-Robots-Tag HTTP headers) tell search engines how to handle your page. Common directives include noindex (don't index), nofollow (don't follow links), noarchive (don't cache), nosnippet (don't show snippets), and max-snippet/max-image-preview restrictions. These directives control how your pages appear in search results and how search engines crawl your site.
When both meta robots (HTML) and X-Robots-Tag (HTTP header) are present, the most restrictive directive typically wins. For example, if one says "index, follow" and the other says "noindex", the page will NOT be indexed. The same applies to nofollow, noarchive, and nosnippet. This is why checking both sources is critical — many server configurations silently inject X-Robots-Tag headers.
When there's a conflict between HTML meta robots and the HTTP X-Robots-Tag header, search engines apply the most restrictive combination. A conflict arises when one source sets a directive and the other does not. For instance, if the HTML says "noindex" but the HTTP header doesn't mention noindex, the page will be noindexed. Always ensure both sources agree on every directive to avoid unexpected behavior.
This tool checks both the standard meta robots tag (applies to all bots) and the googlebot-specific meta tag. Some sites serve different rules to Google specifically — for example, allowing Google to index content while blocking Bing. The googlebot-specific meta tag (<meta name="googlebot" content="...">) overrides the general robots tag for Google's crawler only.
Noindex tells search engines "do not show this page in search results."Nofollow tells search engines "do not follow the links on this page for ranking purposes."They are independent directives and can be used together or separately. A page with noindex, nofollow will neither appear in search results nor pass link equity through its outbound links. A page with just noindex (but no nofollow) will still pass link equity through its links.
The nosnippet directive prevents search engines from showing a text snippet or video preview in search results. If a page has nosnippet, Google may still show the page title and URL but not a description. This can reduce click-through rates from search results. The related max-snippet:[number] directive allows you to control snippet length instead of removing it entirely.
Yes, you can combine multiple directives in the same meta robots content attribute, separated by commas. For example: <meta name="robots" content="index, follow, max-snippet:150, max-image-preview:standard">. The max-snippet directive limits the text snippet to a specified number of characters, and max-image-preview controls whether images can appear in search results (none/standard/large). These give you granular control over your search result appearance.
The "none" directive is a shorthand equivalent to "noindex, nofollow". When a page has robots content="none", search engines will neither index the page nor follow any links on it. This is a strong directive that effectively removes the page from search consideration entirely. Use it only for pages you want to completely hide from search engines, such as admin areas or duplicate content pages.
No — robots directives only affect organic (unpaid) search results. Google Ads landing pages can still be shown in paid search results even if they have a noindex tag. However, if you want your landing pages to also appear in organic results, you should ensure they don't have blocking robots directives. Quality Score evaluations for ads do consider landing page experience but are independent of robots tags.
Robots.txt prevents crawling, while meta robots/X-Robots-Tag can prevent indexing only after the crawler can access the page and see the directive. If a page is disallowed in robots.txt, Google may still index the URL based on links or other information, but it cannot see the noindex tag. To remove a page from Google, allow Googlebot to crawl the URL and return a noindex meta tag or X-Robots-Tag header until Google processes it.
Reviewed Jun 2026 · Sources and limitations

Review details: 2026-06-10 · Marc LaClear · v1.0

Reference sources:

Known limits:

  • Checks are based on publicly fetchable HTML, response headers, and browser-side input. They do not use private Google Search Console, analytics, or ranking data.
  • Scores and warnings are diagnostic aids, not guarantees of ranking improvement or Google indexation.
  • Pages blocked by robots.txt, login walls, bot protection, heavy JavaScript, or network timeouts may return incomplete results.
  • Validate critical fixes with official Google tools such as Search Console, Rich Results Test, Lighthouse, and your own crawl data.

Report an issue with this tool