crw
Lightweight Rust-based web scraper, crawler, and search API with a built-in MCP server for AI agents. Drop-in Firecrawl/Tavily-compatible API; agents use it to extract and search web content at 2x+ the speed of alternatives.
Connects to: MCP clients, Claude, Cursor, Firecrawl API, Tavily API, self-hosted deployments · · AGPL-3.0 1,056★
Use it with an AI agent
Loadbay is an MCP server, so an agent can search the catalog and find this harness:
claude mcp add --transport http loadbay https://loadbay.xyz/api/mcp
- Source: https://github.com/us/crw
- This harness as JSON: /api/harnesses/crw
- Agent setup: /setup.md
- Browse all 780+ harnesses on Loadbay