How to Build a LangChain Web Scraper Tool That Pays per Page
Give a LangChain agent two tools: web_search (up to 10 results, $0.008 per call) and read_url (page to markdown, $0.003 per call). Your code pays each call over x402 in USDC on Base, so there is no API key to manage and failed calls cost nothing.
Steps
- Install. pip install "x402[requests,evm]" langchain-core. The x402 client wraps requests and pays HTTP 402 responses.
- Fund a wallet on Base. Hold a little USDC on Base and export the private key as AGENT_KEY. Set a spending cap of your own: the client pays every 402 it receives.
- Define the tools. Wrap /v1/search and /v1/scrape/lite with the @tool decorator so the model sees a name, a description and typed arguments.
- Bind them to your agent. Pass [web_search, read_url] to your chat model's bind_tools or to your agent constructor.
Code
Python: two LangChain tools
# pip install "x402[requests,evm]"
import os
from eth_account import Account
from x402 import x402ClientSync
from x402.http.clients import x402_requests
from x402.mechanisms.evm import EthAccountSigner
from x402.mechanisms.evm.exact import ExactEvmScheme
client = x402ClientSync()
client.register("eip155:8453", ExactEvmScheme(signer=EthAccountSigner(Account.from_key(os.environ["AGENT_KEY"]))))
session = x402_requests(client)
session.headers["User-Agent"] = "my-agent/1.0"
from langchain_core.tools import tool
@tool
def web_search(query: str, limit: int = 5) -> list:
"""Search the web. Returns up to 10 results with rank, title, url and snippet."""
r = session.post("https://api.switch402.com/v1/search", json={"q": query, "limit": limit}) # $0.008
r.raise_for_status()
return r.json()["results"]
@tool
def read_url(url: str) -> str:
"""Read a public web page and return it as markdown."""
r = session.post("https://api.switch402.com/v1/scrape/lite", json={"url": url}) # $0.003
if r.status_code != 200:
return f"Could not read {url} (HTTP {r.status_code}, not charged)"
return r.json()["markdown"][:20000]
tools = [web_search, read_url]
# llm_with_tools = your_chat_model.bind_tools(tools)Free quote first (curl)
curl -s 'https://api.switch402.com/v1/quote?url=https://example.com&tier=lite' -H 'User-Agent: my-agent/1.0'You pay only for the pages you actually fetch: $0.008 per standard page on /v1/scrape, $0.003 per lite call on /v1/scrape/lite, in USDC on Base and charged only on success. No subscription or unused credits; optional prepaid packs and a $10 monthly plan.
FAQ
- Why cut the markdown at 20,000 characters?
- It keeps one long page from filling the context window. Adjust it to your model.
- Can I use a prepaid key instead of a wallet?
- Yes. Use plain requests with Authorization: Bearer <key>. Each successful call debits its own price.
- Is there an MCP option?
- Yes: the same tools are on https://api.switch402.com/mcp for MCP-aware agents. Paid tools need an x402-capable client.
Related
- API docs
- Use cases
- Pricing calculators
- All guides
- Scrape a website to markdown in Python
- Add web scraping to Claude Desktop (MCP)
- Add web scraping to Cursor (MCP)
- Pay per request with x402 and USDC
- Web search MCP in Claude Code
- Web scraping in n8n, pay per use
- CrewAI web scraping tool
- PDF text extraction in Python