Web scraping for agents: why one provider is not enough

$ Set up https://www.routergrowth.com/SKILL.md, then scrape https://example.com/pricing into markdown and show me which provider answered and how long it took.

Every scraping provider fails. Not often, and not on purpose: a render times out, a rate limit trips at the wrong minute, a site ships a new bot wall, the provider has an incident. When a human is scraping, a failure is a retry. When an agent is scraping, a failure is a broken step in a loop that had twelve more steps to run, and the agent either stops or, worse, carries on with an empty page as if it were the source.

The fix is not a better provider. It is more than one provider behind the same call, with a router that tries the next one when the first one fails and only charges for the attempt that delivered. Since 14 September 2026, web.scrape runs that way on RouterGrowth: Firecrawl first, Apify behind it, one call, one balance.

What breaks when an agent scrapes through one provider?

Three things, in order of how often they happen.

• The page did not come back. Timeouts and rate limits are the common case. A scraping API that succeeds 98% of the time fails one page in fifty, and an agent reading twenty pages per brief hits that wall on most briefs.
• The page came back empty. A JavaScript-only page that the provider did not render, or a bot wall that returned a 200 with nothing in it. Your agent gets a "success" with a 40-word body and cites it.
• The provider is down. Rare, and total. Every scrape in every loop fails until someone notices.

With one provider, each of these is your problem: a retry loop you write, a second account you keep funded, a fallback you maintain and never test until the day you need it. With two providers behind one capability, each of these is a routing decision the platform already made.

How does one call route across Firecrawl and Apify?

The agent calls web.scrape with a URL. The router ranks the live providers for that capability, tries the first, and on a failure moves to the next. The response is the same shape whichever provider served it: title, URL and the page as clean markdown.

curl -X POST https://api.routergrowth.com/v1/run \
  -H "Authorization: Bearer $ROUTERGROWTH_API_KEY" \
  -d '{"capability": "web.scrape", "input": {"url": "https://example.com/pricing"}}'

What happens underneath, per attempt:

1. The provider's price is quoted and a hold of that size is reserved against your balance.
2. The provider is called. The response is classified: succeeded, no match, provider error, rate limited, timed out.
3. On success the hold settles at cost plus the platform margin, and the run ends.
4. On anything else the hold is released in full and the next provider is tried, up to three attempts.

At most one attempt ever settles. A failed attempt is never charged. That is the whole contract, and it is what makes a second provider free to have and cheap to use: you pay for the page, not for the path to it.

What does a failover look like in the run record?

Every run carries its attempts, so an agent (or you, reading the log) can see which provider answered and which ones did not. A failover reads like this:

{
  "status": "succeeded",
  "provider": "apify",
  "routing_reason": "highest weighted score within max cost",
  "attempts": [
    {"provider": "firecrawl", "outcome": "provider_rate_limited", "duration_ms": 412},
    {"provider": "apify", "outcome": "succeeded", "duration_ms": 6210}
  ],
  "billing": {"quoted": "0.015", "charged": "0.015"}
}

Attempt one was released. Attempt two was charged, once, at the second provider's price. The agent got its markdown and never had to know.

Which provider goes first?

The default strategy is best_value: a weighted score of measured success rate (45%), price (25%), latency (20%) and freshness (10%), computed over the last 30 days of real runs on the platform. A provider with no measurements yet gets a neutral prior, so a new provider is neither favored nor punished before it has a track record.

Three other strategies exist, and you can pin a provider outright:

routergrowth run -c web.scrape -i '{"url":"https://example.com"}' --strategy fastest
routergrowth run -c web.scrape -i '{"url":"https://example.com"}' --strategy cheapest
routergrowth run -c web.scrape -i '{"url":"https://example.com"}' --strategy highest_success_rate
routergrowth run -c web.scrape -i '{"url":"https://example.com"}' --provider apify

Pinning turns failover off: you asked for one provider, you get that provider's answer or its error. Use it when a page is known to render only on one of them, and use --strategy the rest of the time.

What did the two providers measure on 14 September 2026?

The day Firecrawl went live, both providers scraped the same page through the production router. Measured, not quoted:

provider    p50 latency   p95 latency   success   sample   role
firecrawl   744 ms        744 ms        100%      1        first (best_value)
apify       5.9 s         101 s         100%      7        failover

Firecrawl is the faster path by an order of magnitude and the cheaper one per page. Apify is the one that has already proven itself across arbitrary sites for a month, with a long tail on latency because its actors run synchronously. Together they are a better scraper than either is alone: the fast one answers first, the patient one answers when the fast one cannot. The catalog page shows both measurements live, and they will move as the sample grows.

web.search gained the same second provider the same day, next to DataForSEO and Apify.

How do you run it from an agent?

routergrowth run -c web.scrape -i '{"url":"https://example.com/pricing"}'
routergrowth inspect -c web.scrape

inspect is free and returns each provider's price, health and measured latency before you spend anything. Through the MCP server or the SKILL.md, a Claude Code or ChatGPT agent gets web.scrape as a native tool with the quote shown first and the attempts trail in the result, so it can reason about what happened rather than guess.

Where does it still break?

• Both providers fail. It happens: a page that neither renders, or a site that walls both. The run fails, nothing is charged, and the error names the last provider's reason. Your agent should treat that as "no source", not retry in a loop.
• A wall that returns a 200. A short body with "subscribe" or "verify you are human" in the first 200 characters is a failed scrape wearing a success code. Detect it and pin the other provider for that URL, or drop the source.
• Latency budgets. Failover adds the failed attempt's time to the run. If the first provider times out at its limit and the second needs 6 seconds, the agent waited for both. Use --strategy fastest when the loop is latency-bound, and --max-cost to keep the failover inside the budget you set.

What does each call cost?#

Prices below are the starting price per successful call. /v1/inspect returns the exact figure before the run and reserves it against your balance; failures and unbilled no-matches release the hold in full.

One prepaid balance covers every row. Capabilities marked coming soon are listed but not yet executable.

FAQ#

Why route web scraping across two providers instead of picking the best one?

Because the best one still fails on some pages and some minutes. Two providers behind one call turn a scraping failure from a broken agent step into a second attempt the router makes for you. You pay only for the attempt that delivered the page.

Does failover cost extra?

No. Each attempt reserves a hold and a failed attempt releases it in full. At most one attempt settles per run, at that provider's quoted price. A run where both providers fail is not charged at all.

Which scraping providers are live behind web.scrape?

Firecrawl and Apify, both live in production since 14 September 2026. Firecrawl is ranked first by the default strategy on speed and price. Apify is the failover, with a month of verified runs across arbitrary sites. inspect -c web.scrape shows both with live health.

Can I force a specific scraper?

Yes. Pass --provider firecrawl or --provider apify on the CLI, or "routing": {"provider": "..."} on the API. Pinning turns failover off for that run. --strategy fastest, cheapest or highest_success_rate keeps failover on and changes the order.

What happens when a page returns a bot wall with a 200 status?

The provider reports a success and the router treats it as one, because the page did come back. Check the body length and the first few hundred characters before citing it. For a URL that walls one provider, pin the other one, and if both wall it, drop the source.