Tools / Web search and scraping / How to
Pass a page URL and get its content back as markdown, with headings as # lines, links as [text](url) and the markup, scripts and styles removed. The title, meta description and language come with it. Use it to put a page into a prompt, a RAG index or a notes folder without writing an HTML parser.
Ask your agent
Turn these ten documentation pages into markdown files and build a table of contents from their headings.
Connect RouterGrowth once, then ask in plain words. New workspaces start with $1 of credit. Get a key or read the setup guide.
A real run
One call to web.scrape (website scrape), run through RouterGrowth on October 26, 2026. These are the first rows it returned; fields that name a person are masked.
| level | heading |
|---|---|
| h2 | Learning Center |
| h5 | Learn about DDoS attacks |
| h5 | Learn about CDNs |
One call on a Cloudflare Learning Center URL: 5,039 characters of markdown with 19 headings and 3 links; the rows are headings as the markdown carries them.
Step by step
Use the exact URL. If it redirects, the markdown is of the page it lands on, as here.
Send the url. The response holds markdown, title, description, language and status_code.
Drop menu and footer lines you do not need, so the model reads the body.
Split on # lines to keep each section whole when you index or summarize it.
Price
List prices on this call, read from the catalog when this page was built. The exact figure is quoted before every run, and a monthly plan lowers it.
In practice
You describe the job; the agent picks the calls, shows the price and runs them.
Turn these ten documentation pages into markdown files and build a table of contents from their headings.
Search for three well cited guides on zero trust networking, convert each to markdown and compare them side by side.
Connect
One MCP server for Claude, ChatGPT, Claude Code, Codex and Cursor, or the same call over the CLI and REST with the input above.
claude mcp add --transport http routergrowth https://api.routergrowth.com/mcp
npm install -g routergrowth && routergrowth setup
routergrowth run -c web.scrape -i '{"url": "https://www.cloudflare.com/learning/what-is-cloudflare/"}'curl -X POST https://api.routergrowth.com/v1/run \
-H "Authorization: Bearer $ROUTERGROWTH_API_KEY" \
-d '{"capability": "web.scrape", "input": {"url": "https://www.cloudflare.com/learning/what-is-cloudflare/"}}'Worth knowing
Each call converts one URL. For a site, list its pages and run one call per page.
Images come through as markdown image links, not as files or descriptions of the picture.
Columns, tabs and styling become plain reading order, so a visual grid reads as a sequence of blocks.
FAQ
Markdown: headings, lists, links and image links keep their markdown syntax.
Not always. Menu and footer text can come through; filter those lines before chunking.
Yes. The page is rendered before it is converted, so text a public visitor sees is included.
Not in one call. Pass each URL; a sitemap or a site: search gives you the list.
More on Web search and scraping
Reading this as an agent? This page as markdown: /tools/web-scraping/website-to-markdown.md
Connect your agent once. The first calls run on the $1 that comes with signup.