How to Add a Pay-per-Page Web Scraping Tool to CrewAI

Write a small CrewAI custom tool that turns any public URL into markdown for $0.003 per successful call over x402 (USDC on Base). No API key, no plan, and nothing is charged when a page does not come back.

Steps

1. Install. pip install "x402[requests,evm]" crewai.
2. Fund a wallet on Base. Hold a little USDC on Base and export the private key as AGENT_KEY. Keep the balance small: the client pays every 402 it receives.
3. Subclass BaseTool. Give the tool a name and description, and call /v1/scrape/lite in _run.
4. Give it to an agent. Pass tools=[ReadUrlTool()] when you create the Agent.

Code

# pip install "x402[requests,evm]"
import os
from eth_account import Account
from x402 import x402ClientSync
from x402.http.clients import x402_requests
from x402.mechanisms.evm import EthAccountSigner
from x402.mechanisms.evm.exact import ExactEvmScheme

client = x402ClientSync()
client.register("eip155:8453", ExactEvmScheme(signer=EthAccountSigner(Account.from_key(os.environ["AGENT_KEY"]))))
session = x402_requests(client)
session.headers["User-Agent"] = "my-agent/1.0"

from crewai.tools import BaseTool

class ReadUrlTool(BaseTool):
    name: str = "read_url"
    description: str = "Read a public web page by URL and return it as markdown."

    def _run(self, url: str) -> str:
        r = session.post("https://api.switch402.com/v1/scrape/lite", json={"url": url})  # $0.003 per successful page
        if r.status_code != 200:
            return f"Could not read {url} (HTTP {r.status_code}, not charged)"
        return r.json()["markdown"][:20000]

# from crewai import Agent
# researcher = Agent(role="Researcher", goal="...", backstory="...", tools=[ReadUrlTool()])

r = session.post("https://api.switch402.com/v1/search", json={"q": "your query", "limit": 5})
results = r.json()["results"]  # rank, title, url, snippet

You pay only for the pages you actually fetch: $0.008 per standard page on /v1/scrape, $0.003 per lite call on /v1/scrape/lite, in USDC on Base and charged only on success. No subscription or unused credits; optional prepaid packs and a $10 monthly plan.