Tools / Web search and scraping / API
In Python, send requests.post to https://api.routergrowth.com/v1/run with a Bearer key, capability web.scrape and input url. The JSON response carries result.markdown, result.title, result.description and result.status_code, plus the run_id and what was charged. No headless browser, proxy pool or HTML parser to maintain in the script.
Ask your agent
Write a Python script that scrapes every URL in urls.txt and saves each page as a markdown file named after its domain.
Connect RouterGrowth once, then ask in plain words. New workspaces start with $1 of credit. Get a key or read the setup guide.
A real run
One call to web.scrape (website scrape), run through RouterGrowth on October 26, 2026. These are the first rows it returned; fields that name a person are masked.
| url | title | language | status_code |
|---|---|---|---|
| https://www.shopify.com/pos | Point of Sale (POS) for Business - Shopify | en | 200 |
The Shopify POS page as the script receives it: status 200 and 22,934 characters of markdown with 46 headings, ready to split into sections.
Step by step
Read ROUTERGROWTH_API_KEY with os.environ and send it as Authorization: Bearer.
requests.post('https://api.routergrowth.com/v1/run', json={'capability': 'web.scrape', 'input': {'url': url}}, headers=headers, timeout=120).
data = r.json(); data['result']['markdown'] is the page text and data['result']['status_code'] the HTTP status.
If status is running, GET the poll_url in the response until done is true.
Run the list in a for loop or a thread pool, write each page to a .md file and keep the run_id.
Price
List prices on this call, read from the catalog when this page was built. The exact figure is quoted before every run, and a monthly plan lowers it.
In practice
You describe the job; the agent picks the calls, shows the price and runs them.
Write a Python script that scrapes every URL in urls.txt and saves each page as a markdown file named after its domain.
In Python, search the web for three queries about our category, then scrape the top five results of each into one CSV.
Connect
One MCP server for Claude, ChatGPT, Claude Code, Codex and Cursor, or the same call over the CLI and REST with the input above.
claude mcp add --transport http routergrowth https://api.routergrowth.com/mcp
npm install -g routergrowth && routergrowth setup
routergrowth run -c web.scrape -i '{"url": "https://www.shopify.com/pos"}'curl -X POST https://api.routergrowth.com/v1/run \
-H "Authorization: Bearer $ROUTERGROWTH_API_KEY" \
-d '{"capability": "web.scrape", "input": {"url": "https://www.shopify.com/pos"}}'Worth knowing
A 403 or 404 page still returns its text. Keep rows whose status_code is 200.
Lines starting with # mark the sections, which makes chunking for a model a few lines of code.
Change the capability and input to search the web or the news with the same function.
FAQ
None is needed. The API is JSON over HTTPS, so requests or httpx is enough.
Yes. The page is rendered on the provider side before it is turned into markdown.
Yes. Send requests concurrently; each URL is its own run and its own charge.
No. It returns the page as markdown. Pull the fields in your script or with a model.
More on Web search and scraping
Reading this as an agent? This page as markdown: /tools/web-scraping/api-python.md
Connect your agent once. The first calls run on the $1 that comes with signup.