Cockroach Crawler
Open-source web crawling and evidence-ready extraction for AI agents
Made by Ajnas N B
What is Cockroach Crawler?
Cockroach Crawler is a local, open-source crawler and extraction stack for AI agents. It discovers and maps URLs, respects robots and operator-selected limits, renders JavaScript when needed, and turns public web pages into structured records, Markdown or JSON with retrieval provenance. It also includes source adapters, MCP tooling, browser handoff, security boundaries and reproducible benchmark evidence.
Key features
- Multi-seed discovery with BFS, DFS and relevance strategies
- Robots, sitemap, redirect and origin-policy enforcement
- JavaScript rendering and browser handoff
- Markdown, CSS, XPath, regex and schema extraction
- Evidence hashes and retrieval provenance
- Public source adapters and MCP server
- Resource ceilings, deadlines and cancellation
- Open-source Node, Docker and Cloudflare deployment paths
Not yet competed
This product is queued for the next tournament.
Reviews
Sign in to leave a review.
No reviews yet.