switch402

Scrape a Docs Site to Markdown for Your LLM: Map, Then Lite per Page

Turn a documentation site into markdown files for an LLM or a RAG index in two steps: /v1/map lists the site’s URLs ($0.001 per call, up to 1,000 URLs) and /v1/scrape/lite converts each page ($0.003 per call). You pay for one map and the pages you actually convert.

  • Step 1: map

    POST {"url": "<docs root>", "limit": 1000} to /v1/map: same-site links plus sitemap URLs, deduplicated.

  • Step 2: filter

    Keep only the docs paths you need (for example /docs/ or /reference/) before you pay for any page.

  • Step 3: convert

    POST each URL to /v1/scrape/lite and save the markdown with its title and source URL.

  • Failures are free

    A page that does not come back returns 502 and is not charged; retry it later.

Cost: $0.001 for the map + pages x $0.003. A 200-page docs site costs $0.601; 1,000 pages cost $3.001. Re-running only the pages that changed costs $0.003 each.

Code examples

Python: docs site to a folder of markdown files

# pip install "x402[requests,evm]"
import os
from eth_account import Account
from x402 import x402ClientSync
from x402.http.clients import x402_requests
from x402.mechanisms.evm import EthAccountSigner
from x402.mechanisms.evm.exact import ExactEvmScheme

client = x402ClientSync()
client.register("eip155:8453", ExactEvmScheme(signer=EthAccountSigner(Account.from_key(os.environ["AGENT_KEY"]))))
session = x402_requests(client)
session.headers["User-Agent"] = "my-agent/1.0"

import pathlib, re

ROOT = "https://example.com/docs/"
out = pathlib.Path("docs_md"); out.mkdir(exist_ok=True)

m = session.post("https://api.switch402.com/v1/map", json={"url": ROOT, "limit": 1000})  # $0.001
m.raise_for_status()
urls = [u for u in m.json()["urls"] if u.startswith(ROOT)]

for u in urls:
    r = session.post("https://api.switch402.com/v1/scrape/lite", json={"url": u})  # $0.003 per page
    if r.status_code != 200:
        continue  # 502 = no content, not charged
    page = r.json()
    name = re.sub(r"[^a-z0-9]+", "-", u[len(ROOT):].lower()).strip("-") or "index"
    (out / f"{name}.md").write_text(f"# {page.get('title') or name}\n\nSource: {u}\n\n{page['markdown']}")
print(len(urls), "pages mapped")

curl: free quote for the lite tier before you start

curl -s 'https://api.switch402.com/v1/quote?url=https://example.com/docs/&tier=lite' -H 'User-Agent: my-agent/1.0'

FAQ

Is this a crawler?
No. map reads one page plus the sitemaps, and you choose which URLs to convert. That keeps the cost to the pages you need.
Can I respect robots.txt and terms?
You decide which URLs to fetch. Convert only docs you are allowed to copy, and follow each site’s terms.
What about very large docs sites?
One map call returns up to 1,000 URLs. For more, map section roots separately (for example /docs/api/ and /docs/guides/).

Related

More API pages