Guides
Crawl API
Give Siftzo a start page and some limits. It follows links, respects robots.txt and stores every page for you to read while the crawl runs or download when it is done.
Start a crawl
POST /v1/crawls with a JSON body. Unknown fields are rejected with 400 invalid_request.
urlstringrequired- The start page. The same rules as for
POST /v1/scrape: a publichttporhttpsURL. max_pagesintegerdefault 100- 1–10,000. The most pages the crawl fetches. URLs that robots.txt disallows do not count.
max_depthintegerdefault 3- 0–10. How many links away from the start page the crawl goes.
0fetches only the start page. includestring[]- Path patterns a discovered link must match (any of them), at most 20. See patterns. The start page is always fetched.
excludestring[]- Path patterns a discovered link must not match, at most 20. Exclude wins over include.
same_domainbooleandefaulttrue- Stay on the start page's host. A leading
www.is ignored, soexample.comandwww.example.comcount as the same host. allow_subdomainsbooleandefaultfalse- With
same_domain, also follow links to subdomains of the start page's domain, for exampleshop.example.comfromexample.com. ignore_sitemapbooleandefaultfalse- Only follow links found on pages. By default the crawl also reads the start host's sitemaps (the ones its robots.txt names, or
/sitemap.xml) and adds the pages they list. renderauto | never | alwaysdefaultauto- How each page is fetched, as for a single scrape. See Rendering.
countrystring- Exit country for every page, as an ISO 3166-1 alpha-2 code.
rate_per_hostnumberdefault 1- 0.1–5 requests per second to each host. A
Crawl-delayin the site's robots.txt (up to 30 seconds) slows the crawl down further, never up. formatjson | markdowndefaultjsonmarkdownstores every page as Markdown inmarkdowninstead of the HTML body, converted the same way as a single scrape.include_htmlbooleandefaultfalse- Only with
format: "markdown": also keep each page'sbodyandbody_encoding. webhook_urlstring- An
httpsURL that gets a signedcrawl.finishedevent when the crawl has ended. See Webhooks.
curl https://api.siftzo.com/v1/crawls \
-H "X-API-Key: $SIFTZO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://books.toscrape.com/",
"max_pages": 500,
"include": ["/catalogue/**"],
"render": "never",
"rate_per_host": 2
}'The answer is 201 Created with a Location header and the crawl object, with status: "queued". Poll GET /v1/crawls/{id} for progress:
{
"id": "01K72B4W9Q6S3EZJ8M5N1RTVXA",
"status": "running",
"failure_reason": null,
"config": {
"url": "https://books.toscrape.com/",
"max_pages": 500,
"max_depth": 3,
"include": ["/catalogue/**"],
"exclude": [],
"same_domain": true,
"allow_subdomains": false,
"render": "never",
"country": null,
"rate_per_host": 2,
"format": "json",
"include_html": false,
"webhook_url": null
},
"counters": {
"discovered": 214,
"queued": 120,
"in_flight": 4,
"done": 88,
"failed": 1,
"skipped_robots": 0,
"credits": 88
},
"created_at": "2026-10-06T09:12:03.114Z",
"started_at": "2026-10-06T09:12:03.402Z",
"finished_at": null,
"pages_url": "/v1/crawls/01K72B4W9Q6S3EZJ8M5N1RTVXA/pages",
"download_url": "/v1/crawls/01K72B4W9Q6S3EZJ8M5N1RTVXA/download"
}status moves from queued to running and ends as one of:
done: nothing left to fetch, ormax_pagesreached.cancelled: you cancelled it.credits_exhausted: your monthly credits ran out. Pages fetched until then are kept.failed: not a single page could be fetched, for example because robots.txt disallows the start page.failure_reasonsays why.
Patterns
include and exclude take glob patterns that are matched against the whole path of a URL, which always starts with /. Matching is case-sensitive, and the query string is not part of the path.
| Pattern | Matches |
|---|---|
* | Any characters except /, so one path segment |
** | Any characters including / |
? | One character except / |
/blog/** matches every page under /blog/. /products/*/reviews matches /products/42/reviews but not /products/42/a/reviews. /**.pdf in exclude skips PDFs.
Read the results
GET /v1/crawls/{id}/pages returns pages in the order they were stored, also while the crawl runs. Pass the next value of one answer as cursor to the next call. limit is 1–100 (default 20).
curl "https://api.siftzo.com/v1/crawls/$ID/pages?limit=50&cursor=$NEXT" \
-H "X-API-Key: $SIFTZO_API_KEY"{
"id": "01K72B4W9Q6S3EZJ8M5N1RTVXA",
"status": "running",
"pages": [ {
"url": "https://books.toscrape.com/catalogue/page-2.html",
"final_url": "https://books.toscrape.com/catalogue/page-2.html",
"depth": 1,
"status": "done",
"status_code": 200,
"tier": "http",
"credits": 1,
"outcome": "success",
"links_found": 73,
"headers": { "content-type": ["text/html"] },
"body": "<!DOCTYPE html>…",
"body_encoding": "utf8",
"job_id": "01K72B5A2N7Y0C4H6FQ3D8KXWE",
"fetched_at": "2026-10-06T09:12:09.871Z"
} ],
"next": "4-12"
}- While the crawl runs, an empty
pageslist with the samenextmeans nothing new yet. Wait a few seconds and ask again. nextisnullonce the crawl has ended and you have read every page.- A failed page has
status: "failed", no body and anerrorwith the same codes as a single scrape. - Bodies that are not valid UTF-8 are base64-encoded (
body_encoding). - With
format: "markdown", every page hasmarkdown(a string, ornullwhen the page could not be converted) and nobodyorbody_encoding, unless you setinclude_htmlor the Markdown isnull. canonical_urlis set when the page's<link rel="canonical">names another URL: the page is a variant of that one.duplicate_ofis set when an earlier page of the crawl has the same visible text; it holds that page'surl. The copy is stored and billed like any page, but its links are not followed, so a crawl does not get lost in tracking parameters, session ids or a calendar that never ends. Pages with very little text are never compared.
To get everything at once, GET /v1/crawls/{id}/download streams all stored pages as one JSONL file, one page object per line:
curl -o crawl.jsonl "https://api.siftzo.com/v1/crawls/$ID/download" \
-H "X-API-Key: $SIFTZO_API_KEY"Crawls and their pages are kept for 7 days. After that, or for a crawl that belongs to another account, the answer is 404 not_found.
Rather than polling for the end of a crawl, pass a webhook_url. When the crawl ends, for any reason, Siftzo sends a crawl.finished event with its status and counters; see Webhooks.
Cancel
DELETE /v1/crawls/{id} stops the crawl and returns the crawl object. Pages that were already on their way finish, are billed and are stored. Cancelling a crawl that has ended changes nothing.
robots.txt and politeness
Siftzo fetches each site's robots.txt once per crawl and follows it. This cannot be turned off. Rules for the user agent siftzo apply if the file has them, otherwise the rules for *. Pages it disallows are counted in skipped_robots and never fetched.
- No robots.txt (a 4xx answer): everything is allowed.
- A server error, or a robots.txt that cannot be fetched at all: nothing on that host is crawled, since the site's wishes are unknown.
- Links marked
rel="nofollow"are not followed, and neither are any links on a page with<meta name="robots" content="nofollow">.
URLs are normalised before they are compared, so the same page is fetched once: the fragment is dropped, the host is lower-cased, default ports are removed, query parameters are sorted, and tracking parameters such as utm_*, gclid and fbclid are removed.
Credits and limits
Every page costs the same as a single scrape of it, on the tier that produced it, and only when it succeeded (see billing rules). Fetching robots.txt is free. Nothing is set aside for the crawl as a whole: each page reserves its credits when it is fetched, and the crawl stops with credits_exhausted when the month's credits run out.
A crawl uses your account's concurrency limit, shared with your other requests. While a large crawl runs, single scrapes may get 429 concurrency_limit more often. An account can have at most 10 crawls that have not ended; the 11th gets 429 crawl_limit.