Robots.txt Generator, Validator and Tester

Build a robots.txt file without guessing. Control search engines and AI crawlers by purpose, test individual URLs against your rules, and catch dangerous mistakes before publishing.

Choose a simple starting point

Start with a safe preset, then use Advanced options only when you need custom user agents, crawl-delay, or AI crawler policies.

Use the path tester below the generated file to test a URL path.

Advanced options: custom rules, AI crawlers, crawl-delay, and sitemap

User-Agent Configuration

The user-agent your rules apply to. * applies to all crawlers. AI crawler blocks are added separately below.

Allow / Disallow Rules

AI Crawler Policy

Control AI crawlers by purpose. No separate policy generates no rule for that category. Select Block or Allow entire site — overrides wildcard restrictions only when you need a different policy for a specific category. No separate policy means this crawler follows the wildcard rules.

Bot Category Allow Block No policy
Traditional search engines
AI search and answer engines
AI model training
User-requested AI retrieval
SEO tools
Archive and dataset crawlers

Additional Settings

Leave empty to omit. Google does not support the Crawl-delay directive — Google automatically adjusts its crawl rate. For serious crawling problems, inspect server errors and Googlebot activity in Search Console.
Full URL to your XML sitemap (optional). Helps crawlers discover your sitemap.
Quick samples:

Generated robots.txt



        
        

        

            Usage:Copy the generated content and save it as robots.txt in your website root directory (e.g., https://example.com/robots.txt). Use FTP, cPanel File Manager, or your hosting control panel to upload the file.
        


        
        

            
Test a URL Against This File

            

                

                    
                    
                

                

                    
                    
                

            

            

                

                    
                    
                

            

            
            
        


        

    


    
    

        How to use and interpret robots.txt
        
This tool helps you create a properly formatted robots.txt file that controls how search engine crawlers and other bots interact with your website. With support for multiple user-agents, AI crawler presets, and common CMS patterns, you can generate production-ready robots.txt files in seconds.


        

            

                
                

                    User-Agent Presets
                    
Choose from presets for Googlebot, Bingbot, GPTBot, ClaudeBot, and more — no need to remember exact user-agent strings.

                

            

            

                
                

                    AI Crawler Controls
                    
Control AI crawlers by purpose — separate search crawlers, training crawlers, and user-triggered retrieval bots with a purpose-based policy matrix.

                

            

            

                
                

                    Standard Disallows
                    
One-click to add common disallow patterns for WordPress admin, temporary files, and CMS paths.

                

            

            
        


        
Why Proper Crawl Control Matters for SEO

        
Controlling how search engines crawl your site helps manage server resources and ensures crawlers focus on your important pages. Without proper robots.txt rules, crawlers may spend time on:

        
    
            
  • Admin panels — /wp-admin/, /admin/ — these contain no useful content for search results.
    • 
            
  • Duplicate content — Print-friendly pages, pagination parameters, session IDs — crawling these wastes server resources.
    • 
            
  • Internal search results — Pages like /search?q=... — indexing search results provides no value and creates thin content.
    • 
            
  • Temporary files — /tmp/, /cache/, log files — these are never meant to be indexed.
    • 
            
  • AJAX/API endpoints — Backend endpoints that serve data but not complete pages.
    • 
        

        
By carefully crafting your robots.txt rules, you guide crawlers toward your most important content, improving crawl efficiency. For very large or frequently updated sites, this can also help with crawl budget management.


        
How Robots.txt Works

        
The Robots Exclusion Protocol (REP) is a standard that defines how crawlers should interpret robots.txt files. Key principles:

        
    
            
  • Crawlers check yourdomain.com/robots.txt before crawling any pages on your site.
    • 
            
  • Each set of rules starts with a User-agent: line that names the crawler.
    • 
            
  • Rules are processed in order of specificity — the most specific matching rule (longest path) wins.
    • 
            
  • An empty robots.txt or a missing file means all pages are allowed by default.
    • 
            
  • Use Disallow: to block paths and Allow: to override blocks for specific paths.
    • 
            
  • The Sitemap: directive can be placed anywhere in the file (not tied to a user-agent).
    • 
            
  • Critical:Googlebot does not inherit rules from the wildcard (*) group. A specific user-agent group like Googlebot uses only its own rules.
    • 
        


        
AI Crawlers and Content Protection

        
With the rise of AI training data collection, many website owners want to control how AI systems access their content. It is important to distinguish between different AI crawler purposes:

        
    
            
  • Search and answer discovery — OAI-SearchBot, Claude-SearchBot, PerplexityBot — these power AI search features. Blocking them may remove your site from AI-generated answers.
    • 
            
  • Model training — GPTBot, ClaudeBot, Google-Extended, Applebot-Extended — these collect data to train AI models. Many site owners choose to block these.
    • 
            
  • User-requested retrieval — ChatGPT-User, Claude-User — these fetch content when a user specifically asks an AI assistant to access a page.
    • 
            
  • General web crawl — CCBot (Common Crawl) — provides open web datasets used by many AI companies.
    • 
        

        
Not all AI crawlers respect robots.txt directives. For the strongest protection, consider combining robots.txt with IP blocking and terms of service restrictions.


        
Important Caveats

        
    
            
  • robots.txt is a voluntary standard — malicious crawlers and scrapers can ignore it entirely.
    • 
            
  • Blocking crawlers via robots.txt does not hide pages — anyone who knows the URL can still visit it. Use authentication for sensitive content.
    • 
            
  • If you block a page via robots.txt but other sites link to it, Google may still index the URL (with a "result not available" description).
    • 
            
  • For true page removal from search results, use a noindex meta tag or X-Robots-Tag: noindex HTTP header — and keep the page crawlable.
    • 
            
  • Always test your robots.txt after making changes — a mistake could accidentally block important pages or open up restricted areas.
    • 
        

    




    

        Examples and common mistakes
        
Use these practical examples to understand how rules work in real scenarios.

        

            
WordPress safe default
Standard disallows for /wp-admin/ with admin-ajax.php allowed for the front end.

            
Allow search, block AI training
Allow Googlebot and Bingbot while blocking GPTBot, ClaudeBot, and Google-Extended from scraping for model training.

            
Block parameter patterns
Use Disallow: /*?sort= to prevent crawling of sorted or filtered URLs that create duplicate content.

            
        

        
How to Interpret the Result

        
The generated file is ready to deploy. Always test your robots.txt after publishing. Use the built-in validation panel and URL tester above to verify that specific URLs are allowed or blocked as intended.

        
Common Mistakes

        
    
            
  • Accidental sitewide Disallow: /
    • 
            
  • BlockingCSS/JS needed for rendering
    • 
            
  • Wrong sitemap URL
    • 
            
  • Assuming a specific user-agent inherits wildcard (*) group rules — it does not for Googlebot.
    • 
        

    


    
    

        FAQ
                

            
            

                
A robots.txt file is a text file placed in your website's root directory that tells search engine crawlers which pages they can and cannot access. It uses the Robots Exclusion Protocol (REP), a standard respected by all major search engines. A robots.txt file helps you manage crawl budget by preventing crawlers from wasting resources on administrative pages, duplicate content, or sensitive directories. While it is not a security measure (malicious bots can ignore it), it is essential for proper SEO and crawl efficiency on any website of more than a few pages.

            

        

                

            
            

                
Disallow tells crawlers they should not access a specific path or file. For example, Disallow: /wp-admin/ tells crawlers to stay out of the WordPress admin area. Allow overrides a disallow for a specific path — useful when you want to block an entire directory but allow a specific file within it. In the standard robots.txt implementation, all paths are allowed by default, so you only need Allow when you have a broader Disallow that you need to make an exception for. The order matters: for Googlebot, the most specific rule wins.

            

        

                

            
            

                
No — this is one of the most common misconceptions about robots.txt. The robots.txt file only controls whether a crawler can access a page; it does not control whether that page appears in search results. If a page is blocked by robots.txt but linked from other sites, Google may still index it (without being able to see the content) and show it in search results with a description like "A description for this result is not available because of the site's robots.txt."To prevent indexing, you must use a noindex meta tag or X-Robots-Tag HTTP header instead.

            

        

                

            
            

                
AI crawler blocks work the same as any other robots.txt rule — they use the User-agent directive to target specific bots. Different AI bots serve different purposes: OAI-SearchBot and Claude-SearchBot power AI search features, while GPTBot, ClaudeBot, and Google-Extended collect training data. ChatGPT-User and Claude-User fetch content when a user explicitly asks the AI to retrieve a page. Our tool separates these by purpose so you can make informed decisions — for example, allowing AI search discovery while blocking training data collection. Not all AI crawlers respect robots.txt — some may ignore the exclusion protocol entirely.

            

        

                

            
            

                
Your robots.txt file must be placed in the root directory of your website. For most websites, this means it should be accessible at https://yourdomain.com/robots.txt. The file must be named exactly robots.txt (case-sensitive on some servers) and should be a plain text file. If you use a subdomain, each subdomain needs its own robots.txt file at its root. For example, blog.example.com needs its own robots.txt at https://blog.example.com/robots.txt. You can check your current robots.txt by visiting https://yourdomain.com/robots.txt in your browser.

            

        

                

            
            

                
Crawl-delay tells a crawler how many seconds to wait between successive requests to your server. This can help prevent your server from being overwhelmed by aggressive crawlers. Google does not support the Crawl-delay directive — Google automatically adjusts its crawl rate. Bingbot, Yandex, and many other crawlers do respect Crawl-delay, so it is still worth including if you need to manage crawl rates for non-Google crawlers. A typical crawl delay is between 5 and 30 seconds.

            

        

                

            
            

                
There are several ways to test your robots.txt: (1) Use the Robots.txt report in Google Search Console under Settings → Crawl stats to see which URLs are blocked. (2) Use the URL Inspection tool in Search Console to test access to specific URLs. (3) Visit https://yourdomain.com/robots.txt directly in your browser to verify the file is accessible. (4) Use a robots.txt testing tool (like our Robots.txt Tester) to check if specific URLs on your site are allowed or blocked. (5) Use curl -A "Googlebot" https://yourdomain.com/robots.txt to test the raw response. See Google's official guide: https://support.google.com/webmasters/answer/6062598

            

        

                

            
            

                
Yes — Googlebot and most modern search engines support basic pattern matching in robots.txt rules. The asterisk (*) matches any sequence of characters (e.g., Disallow: /*.pdf$ blocks all PDF files). The dollar sign ($) denotes the end of a URL pattern. For example, Disallow: /*?session= blocks all URLs containing a session parameter. Note that not all search engines support pattern matching, and the syntax is more limited than full regular expressions. Google also supports the Allow directive for overriding specific paths within a blocked directory.

            

        

                

            
            

                
Technically, no — if a site does not have a robots.txt file, crawlers will assume all pages are allowed. However, for any site with more than a handful of pages, having a robots.txt file is strongly recommended. It gives you control over crawl traffic, can reduce unnecessary server load from resource-heavy crawlers, and lets you specify sitemap locations. Even a minimal robots.txt with just a sitemap reference (Sitemap: https://example.com/sitemap.xml) is beneficial as it helps crawlers discover your sitemap.

            

        

                

            
            

                
The Sitemap directive tells crawlers where to find your XML sitemap(s). It is optional but highly recommended as it provides a direct signal to search engines about the URLs you want indexed. The format is: Sitemap: https://example.com/sitemap.xml. You can include multiple Sitemap directives for multiple sitemaps. This directive does not belong to any specific user-agent block — it is typically placed at the end of the file. Google, Bing, and other major search engines support this directive as an alternative to submitting sitemaps through their webmaster tools.

            

        

            





        

        

            Reviewed Jun 2026 · Sources and limitations
            

                
Review details: 2026-06-12                · Marc LaClear                · v1.2


                                
Reference sources:

                
                
                                
Known limits:

                
    
                                        
  • This tool generates robots.txt rules based on your selections — it does not fetch or analyze an existing robots.txt file from your site.
    • 
                                        
  • The validation engine checks syntax, safety issues, and common misconfigurations, but is not a substitute for testing with Google Search Console or live crawl data.
    • 
                                        
  • AI crawler policies assume crawlers respect robots.txt. Some AI retrieval crawlers may ignore the exclusion protocol.
    • 
                                        
  • Rules are generated for the current user-agent selection only. Multiple user-agent groups require separate configurations.
    • 
                                    

                
                
Report an issue with this tool