Verifying staging blocks
Ensure staging, admin, and test directories are blocked correctly to prevent search indexers from indexing duplicate site copies.
SEO tools
Robots.txt Tester
Paste robots.txt content to inspect common directives and spot risky crawl blocks.
How to use
- 1
Paste robots.txt content.
- 2
Run the tester.
- 3
Review directives and warnings.
Quick answer
Robots.txt Tester parses your robots.txt file to highlight active rules, User-agent targets, sitemap links, and crawl blocks. It is best for auditing crawl configurations before Google, Bing, or AI agents index your site.
Key features
- Checks directive validity (Allow, Disallow, Sitemap, Crawl-delay).
- Groups rules by User-agent blocks for clear, multi-crawler auditing.
- Highlights empty directives or formatting issues in real-time.
- Identifies absolute sitemap URLs and verifies standard layout compliance.
- Processes text fully in-browser, preventing site crawl configs from leaking.
- Requires zero connections to production DNS, domains, or Search Console.
About this tool
A site's robots.txt file is the initial gatekeeper for search engines and AI scrapers, instructing them on where they are allowed to wander and what directories must remain private. A single misplaced slash, incorrect wildcard star, or malformed Disallow path can completely block entire sections of your domain from search indexing—or accidentally expose secret staging directories to public indexers. This tester translates raw robots directives into clear, structured tables, highlighting which agents have crawl clearance and warning you of dangerous crawl-blocks. Because validation runs inside your local browser window, you can safely test and tweak your rules offline before deploying them live.
Common scenarios
Auditing AI scraper rules
Configure custom rules to allow mainstream search crawlers while specifically blocking or throttling aggressive AI model content scrapers.
Validating sitemap paths
Verify that your sitemap link is fully declared as an absolute URL and placed correctly outside of user-agent directive blocks.
FAQ
Does this replace Google Search Console testing?
No. It is a quick local check for common mistakes before deeper validation.
Can it fetch a live robots.txt file?
Live fetching can be added later as a server-assisted feature.
What is the standard placement for a sitemap rule?
A Sitemap directive is independent of User-agent blocks. It should be declared as an absolute URL (e.g. Sitemap: https://yourdomain.com/sitemap.xml) and typically sits at the very top or bottom of the file.
Are crawl-delay rules supported by Google?
No. Googlebot ignores Crawl-delay directives entirely. However, other search bots (like Bingbot) and minor site scrapers still support and respect Crawl-delay settings.
Can I test wildcards?
Yes. Standard wildcards (like * and $) are supported by modern search engines and can be parsed to audit matches.
Is my crawl file private?
Yes. All robots.txt parsing is handled strictly within your browser window using client-side JavaScript. None of your directories or directives are logged.