# Web search and scraping for your AI agent: find the page, read it, pull the records

RouterGrowth gives an agent the open web through one MCP server: a search that returns sources it can cite, any URL as clean markdown, and structured records pulled from a page against a template. Firecrawl and Apify sit behind the same call, with failover between them.

HTML page: https://www.routergrowth.com/tools/web-scraping

## What your agent can do

| Capability | What it returns | Providers | From | Status |
|---|---|---|---|---|
| [`web.search`](https://www.routergrowth.com/catalog/web.search) | Semantic or keyword web search returning sources an agent can cite. | DataForSEO, Firecrawl, Apify | $0.0035 per search | live |
| [`web.scrape`](https://www.routergrowth.com/catalog/web.scrape) | Clean markdown of a page or site for agent context. | Firecrawl, Apify | $0.0056 per page | live |
| [`web.extract`](https://www.routergrowth.com/catalog/web.extract) | One page in, dated records out: a leadership page, an official bio, an org chart, a press release or a job listing extracted against a template (or your own JSON schema), each record with the sentence on the page that supports it, the page date and the source URL. Watch-friendly: records key on the person or posting, so a weekly refresh reports who joined, left or changed title. | Firecrawl | $0.028 per page | live |
| [`news.search`](https://www.routergrowth.com/catalog/news.search) | Google News articles for a query and location: title, source, snippet, URL and publication time. | DataForSEO, Firecrawl | $0.0035 per search | live |

## Ask it like this

- Read the pricing pages of these 12 competitors and put the plans in one table. (`web.scrape`)
- Search the web for independent reviews of this product and summarise them with links. (`web.search`, `web.scrape`)
- Pull the leadership team from each of these company pages: name, title and the sentence that says so. (`web.extract`)

## Connect

MCP server: `https://api.routergrowth.com/mcp` (OAuth, or an API key header).

```bash
claude mcp add --transport http routergrowth https://api.routergrowth.com/mcp
```

```bash
npm install -g routergrowth && routergrowth setup
routergrowth run -c web.search -i '{"query": "independent reviews of Notion", "limit": 10}'
```

```bash
curl -X POST https://api.routergrowth.com/v1/run \
  -H "Authorization: Bearer $ROUTERGROWTH_API_KEY" \
  -d '{"capability": "web.search", "input": {"query": "independent reviews of Notion", "limit": 10}}'
```

## Worth knowing

- **Two providers, one call.** When one scraper fails on a page, the router tries the other. Only the one that returns the page is charged.
- **Records with evidence.** web.extract returns each record with the sentence on the page that supports it and the page date, so the agent can check its own work.
- **Markdown, not HTML.** Pages come back as clean markdown sized for a model's context.

## Questions

### Is there a web search MCP server?

RouterGrowth is an MCP server with web search, page scraping, structured extraction and news search as tools.

### Is this a Firecrawl MCP?

Firecrawl is one of the providers behind web.scrape, web.search and web.extract, alongside Apify. You use one key and the router handles failover.

### Can it scrape any website?

It reads public pages and returns markdown. Pages behind a login are out of scope, and some sites block automated readers: when both providers fail, nothing is charged.

### How is web.extract different from scraping?

Scraping returns the page. Extraction returns records from it against a template or your own schema, each with its supporting sentence.

## Related

- Workflow: [Track competitor ads for DTC skincare brands](https://www.routergrowth.com/workflows/ads-dtc-skincare.md)
- Workflow: [Track competitor ads for B2B SaaS](https://www.routergrowth.com/workflows/ads-b2b-saas.md)
- Workflow: [SEO research for the AI sales tools category](https://www.routergrowth.com/workflows/seo-ai-sales-tools.md)
- Workflow: [SEO research for sales engagement software](https://www.routergrowth.com/workflows/seo-sales-engagement.md)
- Guide: [AI web scraping: Firecrawl and Apify behind one call, with failover](https://www.routergrowth.com/use-cases/multi-provider-web-scraping.md)
- Guide: [Web search API for AI agents: search, scrape and cite the web](https://www.routergrowth.com/use-cases/web-research-for-agents.md)
- Skill: [Brand mention sweep](https://www.routergrowth.com/skills/brand-mention-sweep.md)