# Scrape public pages from a Python script with one POST request

In Python, send requests.post to https://api.routergrowth.com/v1/run with a Bearer key, capability web.scrape and input url. The JSON response carries result.markdown, result.title, result.description and result.status_code, plus the run_id and what was charged. No headless browser, proxy pool or HTML parser to maintain in the script.

HTML page: https://www.routergrowth.com/tools/web-scraping/api-python
Part of: [Web search and scraping tools](https://www.routergrowth.com/tools/web-scraping.md)

## The call

[`web.scrape`](https://www.routergrowth.com/catalog/web.scrape): Clean markdown of a page or site for agent context.

Price on this network: $0.0056 per page. The exact quote is shown before every run.

## What a real run returned

Run on 2026-10-26: 1 records in 0.7 s for $0.0064. Personal fields are masked. The Shopify POS page as the script receives it: status 200 and 22,934 characters of markdown with 46 headings, ready to split into sections.

| url | title | language | status_code |
|---|---|---|---|
| https://www.shopify.com/pos | Point of Sale (POS) for Business - Shopify | en | 200 |

## How to do it

1. **Keep the key in the environment.** Read ROUTERGROWTH_API_KEY with os.environ and send it as Authorization: Bearer.
2. **POST the URL.** requests.post('https://api.routergrowth.com/v1/run', json={'capability': 'web.scrape', 'input': {'url': url}}, headers=headers, timeout=120).
3. **Read the result.** data = r.json(); data['result']['markdown'] is the page text and data['result']['status_code'] the HTTP status.
4. **Poll when it is still running.** If status is running, GET the poll_url in the response until done is true.
5. **Loop over a URL list.** Run the list in a for loop or a thread pool, write each page to a .md file and keep the run_id.

## Ask it like this

- Write a Python script that scrapes every URL in urls.txt and saves each page as a markdown file named after its domain. (`web.scrape`)
- In Python, search the web for three queries about our category, then scrape the top five results of each into one CSV. (`web.search`, `web.scrape`)

## Connect

MCP server: `https://api.routergrowth.com/mcp`

```bash
claude mcp add --transport http routergrowth https://api.routergrowth.com/mcp
```

```bash
npm install -g routergrowth && routergrowth setup
routergrowth run -c web.scrape -i '{"url": "https://www.shopify.com/pos"}'
```

```bash
curl -X POST https://api.routergrowth.com/v1/run \
  -H "Authorization: Bearer $ROUTERGROWTH_API_KEY" \
  -d '{"capability": "web.scrape", "input": {"url": "https://www.shopify.com/pos"}}'
```

## Worth knowing

- **Skip pages that did not load.** A 403 or 404 page still returns its text. Keep rows whose status_code is 200.
- **Split on headings.** Lines starting with # mark the sections, which makes chunking for a model a few lines of code.
- **Same helper for other calls.** Change the capability and input to search the web or the news with the same function.

## Questions

### Is there a Python SDK?

None is needed. The API is JSON over HTTPS, so requests or httpx is enough.

### Does it handle pages built with JavaScript?

Yes. The page is rendered on the provider side before it is turned into markdown.

### Can I scrape many URLs in parallel?

Yes. Send requests concurrently; each URL is its own run and its own charge.

### Does it extract fields like price or SKU?

No. It returns the page as markdown. Pull the fields in your script or with a model.

## Related

- [Convert a web page into clean markdown your model can read](https://www.routergrowth.com/tools/web-scraping/website-to-markdown.md)
- [Search the web from code and get citable sources back as JSON](https://www.routergrowth.com/tools/web-scraping/web-search-api.md)
- [Ground an agent in sources from the domains you trust](https://www.routergrowth.com/tools/web-scraping/web-search-api-for-llm.md)
- [Turn a public page into agent ready text with a single request](https://www.routergrowth.com/tools/web-scraping/api.md)
- [Get the latest headlines on a listed company from its name and ticker](https://www.routergrowth.com/tools/web-scraping/stock-news-api.md)
- [Pull news articles on any topic into your code as JSON](https://www.routergrowth.com/tools/web-scraping/news-api.md)
- [Fetch news articles into a Python script with one POST](https://www.routergrowth.com/tools/web-scraping/news-api-python.md)
- [Get the articles Google News lists for a query, in one API call](https://www.routergrowth.com/tools/web-scraping/google-news-api.md)
- [Read a public Amazon product page from your agent, with no scraper to maintain](https://www.routergrowth.com/tools/web-scraping/amazon.md)