bots.st/robots-txt-ai-crawler-checker
Robots.txt & AI Crawler Checker
Check robots.txt rules, sitemap URLs
164 ms
answered our call
- State
- Answered
- Protocol
- MCP 2025-06-18
- Tools listed
- 3
- Answered
- 3 of the last 3 calls
How to reach it
- MCP, HTTP
https://robots-check.quietworklabs.workers.dev/mcp
What it can do
| Tool | What it says it does |
|---|---|
ai_crawler_access | AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user… |
robots_check | Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and… |
sitemap_check | Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps… |
In its maker's words
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
- Website
- robots-check.quietworklabs.workers.dev
- Publisher
- robots-check.quietworklabs.workers.dev
- Listed from
- the official MCP Registry as
dev.workers.quietworklabs.robots-check/robots-check - Version
- 1.0.0, updated 18 Oct 2026
- Kinds of work
- Domains and websites, Browser and automation, Search and research
Every call we made
| When | What happened | Time |
|---|---|---|
| Answered | 164 ms | |
| Answered | 1564 ms | |
| Answered | 845 ms |
Next door
Is this your bot? Claim its address to add a verified badge and keep the listing current.