Crawl APIPOST /v1/crawls

Give it a start page. Get the site.

The Crawl API follows links from a start page and fetches up to 10,000 pages, within the patterns, depth and rate you set. robots.txt is always respected.

What it does

Follow the links of a site up to 10,000 pages, always within robots.txt.

  • Patterns, depth and limits

    Up to 10,000 pages and 10 clicks deep. include and exclude glob patterns choose which paths to follow.

  • robots.txt, always

    robots.txt is read for every site and cannot be turned off. A Crawl-delay makes the crawl slower, never faster.

  • Polite by default

    One request per second per host unless you set rate_per_host (0.1–5). Tracking parameters are stripped and URLs deduplicated.

  • The same engine as scrape

    Every page goes through the Scraper API's tiers, block detection and billing, so a crawl behaves like many scrapes.

  • HTML or Markdown

    With format: "markdown" each page comes back as Markdown, ready for an index or an LLM.

  • Read as you go, or all at once

    Page through results while the crawl runs, download everything as one JSONL file, or get a signed crawl.finished webhook.

Example

One HTTP call

POST a start URL and the limits. The answer is the crawl object; poll it, read its pages or wait for the webhook.

Official SDKs are coming soon. Until then, any HTTP client works.

Crawl API docs
curl https://api.siftzo.com/v1/crawls \
  -H "X-API-Key: $SIFTZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/",
    "max_pages": 500,
    "include": ["/catalogue/**"],
    "format": "markdown"
  }'

# Later: read the pages, 50 at a time
curl "https://api.siftzo.com/v1/crawls/$ID/pages?limit=50" \
  -H "X-API-Key: $SIFTZO_API_KEY"

Reference

Key parameters

urlstringrequired
The start page.
max_pagesintegerdefault 100
1–10,000 pages fetched.
max_depthintegerdefault 3
0–10 clicks from the start page.
include, excludeglob patterns
Up to 20 each, matched against the path. exclude wins.
same_domainbooleandefault true
Stay on the start page's host. Add allow_subdomains for its subdomains.
rate_per_hostnumberdefault 1
0.1–5 requests per second per host.
render, country
As for the Scraper API, for every page.
formatjson | markdowndefault json
Markdown replaces the HTML body of every page.
webhook_urlstring
Gets a signed crawl.finished event when the crawl ends.

Every parameter and response field is in the Crawl API docs.

Credits

What it costs

Crawl API credits
RequestCredits
Crawl API pagesame as a scrape of that page; robots.txt fetches are free
HTTP request (render=never)1
Browser request (render=always, or auto when needed)10
Failed request (blocked, timeout, proxy error)0

Nothing is reserved for the crawl as a whole: each page is billed as it is fetched, and the crawl stops with credits_exhausted if your limit is reached.

See plans

Questions

Can I turn robots.txt off?

No. Pages that robots.txt disallows are skipped and counted in skipped_robots; they do not count towards max_pages. See robots.txt.

How long are results kept?

7 days from when the crawl ends. Download the JSONL file to keep them longer.

Can I stop a crawl?

Yes, with DELETE /v1/crawls/{id}. Pages already on their way finish and are billed; nothing else is fetched.

Does a crawl use my concurrency?

Yes. Every page takes a slot of your plan's concurrency while it is fetched, shared with your other requests, so a crawl never goes over your plan.

Try the Crawl API for free

Create an account, copy your key and send a request. No card needed for the Free plan.