bots.st/crawl-census
Crawl Census
Robots/crawl policy audits and preflight decisions
1867 ms
answered our call
- State
- Answered
- Protocol
- MCP 2025-06-18
- Tools listed
- 7
- Answered
- 3 of the last 3 calls
How to reach it
- MCP, HTTP
https://crawlcensus.com/mcp
What it can do
| Tool | What it says it does |
|---|---|
agent_profile | What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of… |
census_facts | Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the… |
census_stats | Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption. |
crawl_preflight | Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots… |
scan_site | Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail. |
site_report | Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site. |
submit_domains | Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so… |
In its maker's words
Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?
- Website
- crawlcensus.com/for-crawlers
- Source
- github.com/taylorsmithgg/crawl-census-client
- Publisher
- taylorsmithgg
- Listed from
- the official MCP Registry as
io.github.taylorsmithgg/crawl-census - Version
- 1.0.0, updated 23 Aug 2026
- Kinds of work
- Browser and automation, Search and research, Security and trust
The domain it lives on
- crawlcensus.com
- registered 22 Aug 2026 (3 months ago), at Cloudflare, Inc., its home page is a working site
From our own crawl of domain names: registry records and the home page as our crawler last saw it.
Every call we made
| When | What happened | Time |
|---|---|---|
| Answered | 1867 ms | |
| Answered | 1684 ms | |
| Answered | 526 ms |
Next door
- Bowmark web data: LinkedIn, Amazon, YouTube, Reddit, Airbnb, InstagramScrape and automate logged-in web data
- SkyelightPin feedback to rendered elements with code
- TapAutomate and validate logged-in browser tasks
- String Web AccessFetch URLs, crawl sites, search web, return clean Markdown
- ShipvoSimulate a phone and automate UI actions
Is this your bot? Claim its address to add a verified badge and keep the listing current.