# Web scraping for agents: why one provider is not enough Every scraping provider fails. Not often, and not on purpose: a render times out, a rate limit trips at the wrong minute, a site ships a new bot wall, the provider has an incident. When a human is scraping, a failure is a retry. When an agent is scraping, a failure is a broken step in a loop that had twelve more steps to run, and the agent either stops or, worse, carries on with an empty page as if it were the source. The fix is not a better provider. It is more than one provider behind the same call, with a router that tries the next one when the first one fails and only charges for the attempt that delivered. Since 14 September 2026, `web.scrape` runs that way on RouterGrowth: Firecrawl first, Apify behind it, one call, one balance. ![One web.scrape call routed across Firecrawl and Apify, with failover and a single charge](/static/use-cases/multi-provider-web-scraping.png) ## What breaks when an agent scrapes through one provider? Three things, in order of how often they happen. - **The page did not come back.** Timeouts and rate limits are the common case. A scraping API that succeeds 98% of the time fails one page in fifty, and an agent reading twenty pages per brief hits that wall on most briefs. - **The page came back empty.** A JavaScript-only page that the provider did not render, or a bot wall that returned a 200 with nothing in it. Your agent gets a "success" with a 40-word body and cites it. - **The provider is down.** Rare, and total. Every scrape in every loop fails until someone notices. With one provider, each of these is your problem: a retry loop you write, a second account you keep funded, a fallback you maintain and never test until the day you need it. With two providers behind one capability, each of these is a routing decision the platform already made. ## How does one call route across Firecrawl and Apify? The agent calls `web.scrape` with a URL. The router ranks the live providers for that capability, tries the first, and on a failure moves to the next. The response is the same shape whichever provider served it: title, URL and the page as clean markdown. ```bash curl -X POST https://api.routergrowth.com/v1/run \ -H "Authorization: Bearer $ROUTERGROWTH_API_KEY" \ -d '{"capability": "web.scrape", "input": {"url": "https://example.com/pricing"}}' ``` What happens underneath, per attempt: 1. The provider's price is quoted and a hold of that size is reserved against your balance. 2. The provider is called. The response is classified: succeeded, no match, provider error, rate limited, timed out. 3. On success the hold settles at cost plus the platform margin, and the run ends. 4. On anything else the hold is released in full and the next provider is tried, up to three attempts. At most one attempt ever settles. A failed attempt is never charged. That is the whole contract, and it is what makes a second provider free to have and cheap to use: you pay for the page, not for the path to it. ## What does a failover look like in the run record? Every run carries its attempts, so an agent (or you, reading the log) can see which provider answered and which ones did not. A failover reads like this: ```json { "status": "succeeded", "provider": "apify", "routing_reason": "highest weighted score within max cost", "attempts": [ {"provider": "firecrawl", "outcome": "provider_rate_limited", "duration_ms": 412}, {"provider": "apify", "outcome": "succeeded", "duration_ms": 6210} ], "billing": {"quoted": "0.015", "charged": "0.015"} } ``` Attempt one was released. Attempt two was charged, once, at the second provider's price. The agent got its markdown and never had to know. ## Which provider goes first? The default strategy is `best_value`: a weighted score of measured success rate (45%), price (25%), latency (20%) and freshness (10%), computed over the last 30 days of real runs on the platform. A provider with no measurements yet gets a neutral prior, so a new provider is neither favored nor punished before it has a track record. Three other strategies exist, and you can pin a provider outright: ```bash routergrowth run -c web.scrape -i '{"url":"https://example.com"}' --strategy fastest routergrowth run -c web.scrape -i '{"url":"https://example.com"}' --strategy cheapest routergrowth run -c web.scrape -i '{"url":"https://example.com"}' --strategy highest_success_rate routergrowth run -c web.scrape -i '{"url":"https://example.com"}' --provider apify ``` Pinning turns failover off: you asked for one provider, you get that provider's answer or its error. Use it when a page is known to render only on one of them, and use `--strategy` the rest of the time. ## What did the two providers measure on 14 September 2026? The day Firecrawl went live, both providers scraped the same page through the production router. Measured, not quoted: ``` provider p50 latency p95 latency success sample role firecrawl 744 ms 744 ms 100% 1 first (best_value) apify 5.9 s 101 s 100% 7 failover ``` Firecrawl is the faster path by an order of magnitude and the cheaper one per page. Apify is the one that has already proven itself across arbitrary sites for a month, with a long tail on latency because its actors run synchronously. Together they are a better scraper than either is alone: the fast one answers first, the patient one answers when the fast one cannot. The catalog page shows both measurements live, and they will move as the sample grows. `web.search` gained the same second provider the same day, next to DataForSEO and Apify. ## How do you run it from an agent? ```bash routergrowth run -c web.scrape -i '{"url":"https://example.com/pricing"}' routergrowth inspect -c web.scrape ``` `inspect` is free and returns each provider's price, health and measured latency before you spend anything. Through the MCP server or the SKILL.md, a Claude Code or ChatGPT agent gets `web.scrape` as a native tool with the quote shown first and the attempts trail in the result, so it can reason about what happened rather than guess. ## Where does it still break? - **Both providers fail.** It happens: a page that neither renders, or a site that walls both. The run fails, nothing is charged, and the error names the last provider's reason. Your agent should treat that as "no source", not retry in a loop. - **A wall that returns a 200.** A short body with "subscribe" or "verify you are human" in the first 200 characters is a failed scrape wearing a success code. Detect it and pin the other provider for that URL, or drop the source. - **Latency budgets.** Failover adds the failed attempt's time to the run. If the first provider times out at its limit and the second needs 6 seconds, the agent waited for both. Use `--strategy fastest` when the loop is latency-bound, and `--max-cost` to keep the failover inside the budget you set. ## FAQ ### Why route web scraping across two providers instead of picking the best one? Because the best one still fails on some pages and some minutes. Two providers behind one call turn a scraping failure from a broken agent step into a second attempt the router makes for you. You pay only for the attempt that delivered the page. ### Does failover cost extra? No. Each attempt reserves a hold and a failed attempt releases it in full. At most one attempt settles per run, at that provider's quoted price. A run where both providers fail is not charged at all. ### Which scraping providers are live behind web.scrape? Firecrawl and Apify, both live in production since 14 September 2026. Firecrawl is ranked first by the default strategy on speed and price. Apify is the failover, with a month of verified runs across arbitrary sites. `inspect -c web.scrape` shows both with live health. ### Can I force a specific scraper? Yes. Pass `--provider firecrawl` or `--provider apify` on the CLI, or `"routing": {"provider": "..."}` on the API. Pinning turns failover off for that run. `--strategy fastest`, `cheapest` or `highest_success_rate` keeps failover on and changes the order. ### What happens when a page returns a bot wall with a 200 status? The provider reports a success and the router treats it as one, because the page did come back. Check the body length and the first few hundred characters before citing it. For a URL that walls one provider, pin the other one, and if both wall it, drop the source.