Web search and scraping for agents

$ Set up https://www.routergrowth.com/SKILL.md, then research "SOC 2 for seed-stage startups": search the web, read the five best pages as markdown and give me a sourced brief.

Every agent that does research needs the same two primitives: search the web and get back results it can cite, then read a page without drowning in HTML. Most teams build both twice, once against a search API and once against a scraping service, each with its own key, its own invoice and its own idea of what a result looks like.

RouterGrowth exposes them as two capabilities on one key. web.search returns structured results. web.scrape returns a page as clean markdown. When the question is "who ranks" rather than "what is true", seo.serp returns the raw Google results page as JSON.

The pipeline

web.search     →  structured results (title, URL, snippet, content) for a query
web.scrape     →  one URL as clean markdown for the model's context
seo.serp       →  the live Google or Bing results page as JSON, with SERP features
news.search    →  recent news for a query, when recency is the point

Step 1: search and get structured results

curl -X POST https://api.routergrowth.com/v1/run \
  -H "Authorization: Bearer $ROUTERGROWTH_API_KEY" \
  -d '{"capability": "web.search",
       "input": {"query": "SOC 2 timeline for seed-stage startups", "limit": 10}}'

The response is a list of results with a title, URL, snippet and, where the provider returns it, the page content. It is the same shape whichever provider served the call. Routing is by best_value unless you pin a provider: DataForSEO and Apify are live today, and the semantic engines (Exa, Tavily, Brave, Firecrawl search) are wired behind the same call and switch on as their accounts open. Pin with "routing": {"provider": "..."} when a given engine is the point of the experiment.

Priced per search. A search that returns nothing is not billed.

Two habits make the results usable by a model instead of merely readable by a human:

• Ask the question, not the keyword. Semantic engines reward a full sentence. Keyword engines tolerate one. "how long does a SOC 2 Type 1 take for a 10-person startup" beats "soc 2 startup".
• Keep the URLs. The model should cite the URL it read, not the search snippet. Snippets are truncated and sometimes stale. The scrape step is what turns a result into a source.

Step 2: read the page as markdown

curl -X POST https://api.routergrowth.com/v1/run \
  -H "Authorization: Bearer $ROUTERGROWTH_API_KEY" \
  -d '{"capability": "web.scrape", "input": {"url": "https://example.com/soc2-guide"}}'

Title and body as markdown, with navigation, cookie banners and scripts stripped. Headings survive as headings, tables as tables, links as links. That is what the model wants in its context: a document, not a DOM.

Priced per page. Routed across Apify today with Firecrawl behind the same call, so a "scrape this into markdown" step is one capability rather than a choice between Firecrawl, Jina Reader and a headless browser you maintain.

Scrape only what you will read. The economical pattern is search once, scrape the three to five results the model chose, and stop. Scraping every result on the page multiplies cost for pages the model will never quote.

Step 3: rankings are a different question

curl -X POST https://api.routergrowth.com/v1/run \
  -H "Authorization: Bearer $ROUTERGROWTH_API_KEY" \
  -d '{"capability": "seo.serp",
       "input": {"keyword": "soc 2 for startups", "location": "United States", "engine": "google"}}'

web.search answers "what are the best sources for this question". seo.serp answers "what does Google show for this keyword, in this country, right now": the organic results in order, the SERP features present, and whether an AI Overview appeared. Use it when the output is an SEO decision (is this keyword winnable, who owns it) rather than a research brief. It is also the cheapest way to get Google results as JSON when what you need is the ranking, not the reading.

Step 4: recency

curl -X POST https://api.routergrowth.com/v1/run \
  -H "Authorization: Bearer $ROUTERGROWTH_API_KEY" \
  -d '{"capability": "news.search", "input": {"query": "SOC 2 requirements change", "limit": 20}}'

Web search indexes lag. When the question is "what happened this week", Google News through news.search returns dated articles with the source, which the model can order by time before it reads anything.

Putting it in an agent loop

The whole loop, as the agent runs it:

1. web.search with the question, limit around 10.
2. The model picks the results worth reading. Usually three to five.
3. web.scrape each of those.
4. The model writes the brief, citing the URLs it read.
5. If the brief needs "as of this week", news.search first and scrape the two newest.

Cost per brief is one search plus a handful of pages, which lands in cents. The expensive version of this loop is the one that scrapes every result, so the selection step in the middle is where an agent earns its keep.

Where it breaks

• Paywalled and login-walled pages. The scrape returns the wall, not the article. Detect it (short body, "subscribe" in the first 200 characters) and drop the source rather than citing it.
• JavaScript-only pages. Providers render most of them. A page that returns a near-empty body is the signal to try the other provider by pinning routing.provider.
• Search results are not sources. A snippet is a preview. Cite what you scraped, and treat a result you did not read as a lead, not a fact.
• Rate and respect. The providers handle robots and rate limits. Your agent should still not re-scrape the same URL in a loop; cache by URL for the length of the task.

Run it as an agent

routergrowth run -c web.search -i '{"query":"SOC 2 timeline for seed-stage startups","limit":10}'
routergrowth run -c web.scrape -i '{"url":"https://example.com/soc2-guide"}'
routergrowth run -c seo.serp -i '{"keyword":"soc 2 for startups","location":"United States"}'

The MCP server and the SKILL.md expose the same three calls, so a Claude Code or ChatGPT agent runs the loop above as native tools with the spend quoted before each call.

What does each call cost?#

Prices below are the starting price per successful call. /v1/inspect returns the exact figure before the run and reserves it against your balance; failures and unbilled no-matches release the hold in full.

One prepaid balance covers every row. Capabilities marked coming soon are listed but not yet executable.

FAQ#

How do I run a web search from inside my agent and get structured results?

Call web.search with the question as the query. The result is a list of title, URL, snippet and content fields, the same shape for every provider behind it, priced per search. Through the MCP server or the SKILL.md the agent gets it as a native tool, quotes the cost first, and gets the results back as JSON it can pass straight into the next step.

What API scrapes any web page into clean markdown for an LLM?

web.scrape takes a URL and returns the page's title and body as markdown with navigation, banners and scripts removed. It is routed across Apify and Firecrawl, so it covers the same job as Firecrawl or Jina Reader without a separate account, and it is priced per page on the same balance as the search that found the page.

Is this cheaper than a search API subscription?

For agent workloads, usually. Search API plans price a monthly quota you rarely fill evenly. Here a search and a page are each priced individually, a search that returns nothing is not billed, and there is no monthly minimum. The exact per-call price is returned by /v1/inspect before you run anything.

When should I use seo.serp instead of web.search?

When the output is a ranking decision. seo.serp returns the live Google or Bing results page for a keyword and country, in order, with SERP features and AI Overview presence: what an SEO tool needs. web.search returns the best sources for a question across engines: what a research agent needs. They are priced separately and answer different questions.

Can I pin a specific provider, like Firecrawl or Exa?

Yes, with "routing": {"provider": "firecrawl"} on the run, as long as that provider is live for the capability. The catalog page for each capability shows which providers are live and which are wired and waiting on an account. Auto routing picks the best value among live providers and fails over if one errors.