Scrape Tool

Turn open-web URLs into clean datasets at scale - fetch, crawl, extract, clean, and export - with a Rust hot path when navin-core is installed.

For the Scraping studio UI and /scrape workflows, see navin_scraping.

Why You Need It

web_fetch is great for one page. Corpora need parallelism, cleaning, and file exports. The scrape tool gives the agent:

Parallel fetch of many URLs with title, text, markdown, links, and meta
Same-domain crawl (BFS) with depth and page caps
Cleaning of HTML chrome into readable text/markdown
Exports to csv, json, jsonl, xml, xlsx, md, and HTML report
Autonomy: dry-run a few pages, scale with pipeline, escalate to browser for JS shells
Assisted walls: detect captcha / Cloudflare / paywall / login, pause, ask the user to solve or sign in via a browser session - never auto-bypass

Configuration

Enabled by default.

tools:
  scrape:
    enabled: true          # default: true
    max_pages: 50          # default crawl/pipeline page cap (hard max 200)
    concurrency: 8         # parallel workers (1-32)
    timeout_seconds: 30
    max_bytes: 5242880     # 5 MiB per response
    user_agent: "Mozilla/5.0 (compatible; NavinScrape/0.1; +https://navin.ai) AppleWebKit/537.36"
    proxy: null            # optional HTTP(S) proxy
    respect_same_domain: true
    assisted: true         # pause + ask user on captcha/CF/paywall/login (no bypass)

Disable with tools.scrape.enabled: false. Set assisted: false only if you want a wall report without the approval pause (still no bypass).

Build the Rust accelerator (recommended):

make native

Without it, the same actions run through a pure-Python fallback.

Actions

fetch - Parallel GET + extract

scrape(action="fetch", url="https://example.com/docs")
scrape(action="fetch", urls='["https://a.example/x","https://a.example/y"]')
scrape(action="fetch", urls="https://a.example/x\nhttps://a.example/y", concurrency=16)

Returns JSON { "pages": [ ... ] } with url, finalUrl, status, title, text, markdown, links, meta, error.

crawl - Same-domain BFS

scrape(action="crawl", url="https://example.com/docs", max_depth=2, max_pages=40)

Follows in-domain links up to max_depth / max_pages. Returns pages plus queued / seen counts.

extract - HTML → structured

scrape(action="extract", html="<html>...</html>", url="https://example.com/page")

Use when you already have HTML (e.g. from browser action=content).

clean - Normalize text

scrape(action="clean", text="  messy \n\n  text  ")

export - Write files

scrape(action="export", records='{"pages":[...]}', format="xlsx", path="scrape/out.xlsx")
scrape(action="export", records='[...]', format="csv", path="scrape/out.csv")
scrape(action="export", records='[...]', format="report", path="scrape/report.html")

path must stay inside the workspace. Formats: csv, json, jsonl, xml, xlsx, md, report.

pipeline - Crawl then export

scrape(
  action="pipeline",
  url="https://example.com/docs",
  format="xlsx",
  path="scrape/docs.xlsx",
  max_depth=2,
  max_pages=30
)

Returns a short summary (pages, ok, export, sample) - not the full corpus in chat.

Parameters

Parameter	Used by	Description
`action`	all	`fetch` \| `crawl` \| `extract` \| `clean` \| `export` \| `pipeline` \| `diagnose`
`url`	fetch, crawl, pipeline, extract	Single seed / base URL
`urls`	fetch, crawl, pipeline	JSON array or newline/comma list
`html`	extract	Raw HTML
`text`	clean	Raw text
`records`	export	JSON array or `{ "pages": [...] }`
`format`	export, pipeline	Export format
`path`	export, pipeline	Workspace-relative output path
`max_pages`	crawl, pipeline	Page cap
`max_depth`	crawl, pipeline	Link depth (`0` = seeds only)
`concurrency`	fetch	Parallel workers

diagnose - Classify walls

scrape(action="diagnose", url="https://example.com")
scrape(action="diagnose", html="<html>Just a moment...</html>")

Returns wall.kind (captcha, cloudflare, paywall, login, empty_shell, etc.) and the no-bypass policy.

Walls (assisted, no bypass)

On fetch / crawl / pipeline / diagnose, each page may include a wall object. The response also has walls, bypass_policy, and next_steps.

Kind	Human?	Agent behaviour
`empty_shell`	no	Escalate to `browser` (render / API) - not a security bypass
`forbidden` / `error`	no	Report; optional browser retry
`captcha` / `cloudflare` / `paywall` / `login`	yes	Pause (approval card when `assisted: true`). User solves or signs in via browser. Never bypass.

Flow for human walls:

Tool detects the wall and requests approval.
User allows, then completes the challenge / login in the browser tool session.
Agent runs browser action=content, then scrape action=extract / export.

When to use browser instead

Situation	Tool
Static / server-rendered HTML	`scrape`
Empty shell / heavy client render	`browser` (`action=content`)
JSON API behind the page	`browser` network + `response_body`, then `scrape action=export`
Captcha / Cloudflare / paywall / login	Pause → user in `browser` session → resume (no bypass)

Security

Only http / https URLs; targets pass the same SSRF checks as web_fetch.
Exports are confined to the project workspace.
Treat page bodies as untrusted data (prompt-injection mindset).
No automatic bypass of captcha, Cloudflare, paywalls, or login walls.
Respect robots/ToS when the user cares about compliance - ask if unsure.

Studio module: navin_scraping
Web search/fetch: Configure Web Search
Native build: make native (see Performance)