Guides

Crawl API

Give Siftzo a start page and some limits. It follows links, respects robots.txt and stores every page for you to read while the crawl runs or download when it is done.

Start a crawl

POST /v1/crawls with a JSON body. Unknown fields are rejected with 400 invalid_request.

urlstringrequired
The start page. The same rules as for POST /v1/scrape: a public http or https URL.
max_pagesintegerdefault 100
1–10,000. The most pages the crawl fetches. URLs that robots.txt disallows do not count.
max_depthintegerdefault 3
0–10. How many links away from the start page the crawl goes. 0 fetches only the start page.
includestring[]
Path patterns a discovered link must match (any of them), at most 20. See patterns. The start page is always fetched.
excludestring[]
Path patterns a discovered link must not match, at most 20. Exclude wins over include.
same_domainbooleandefault true
Stay on the start page's host. A leading www. is ignored, so example.com and www.example.com count as the same host.
allow_subdomainsbooleandefault false
With same_domain, also follow links to subdomains of the start page's domain, for example shop.example.com from example.com.
ignore_sitemapbooleandefault false
Only follow links found on pages. By default the crawl also reads the start host's sitemaps (the ones its robots.txt names, or /sitemap.xml) and adds the pages they list.
renderauto | never | alwaysdefault auto
How each page is fetched, as for a single scrape. See Rendering.
countrystring
Exit country for every page, as an ISO 3166-1 alpha-2 code.
rate_per_hostnumberdefault 1
0.1–5 requests per second to each host. A Crawl-delay in the site's robots.txt (up to 30 seconds) slows the crawl down further, never up.
formatjson | markdowndefault json
markdown stores every page as Markdown in markdown instead of the HTML body, converted the same way as a single scrape.
include_htmlbooleandefault false
Only with format: "markdown": also keep each page's body and body_encoding.
webhook_urlstring
An https URL that gets a signed crawl.finished event when the crawl has ended. See Webhooks.
Example
curl https://api.siftzo.com/v1/crawls \
  -H "X-API-Key: $SIFTZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/",
    "max_pages": 500,
    "include": ["/catalogue/**"],
    "render": "never",
    "rate_per_host": 2
  }'

The answer is 201 Created with a Location header and the crawl object, with status: "queued". Poll GET /v1/crawls/{id} for progress:

200 OK
{
  "id": "01K72B4W9Q6S3EZJ8M5N1RTVXA",
  "status": "running",
  "failure_reason": null,
  "config": {
    "url": "https://books.toscrape.com/",
    "max_pages": 500,
    "max_depth": 3,
    "include": ["/catalogue/**"],
    "exclude": [],
    "same_domain": true,
    "allow_subdomains": false,
    "render": "never",
    "country": null,
    "rate_per_host": 2,
    "format": "json",
    "include_html": false,
    "webhook_url": null
  },
  "counters": {
    "discovered": 214,
    "queued": 120,
    "in_flight": 4,
    "done": 88,
    "failed": 1,
    "skipped_robots": 0,
    "credits": 88
  },
  "created_at": "2026-10-06T09:12:03.114Z",
  "started_at": "2026-10-06T09:12:03.402Z",
  "finished_at": null,
  "pages_url": "/v1/crawls/01K72B4W9Q6S3EZJ8M5N1RTVXA/pages",
  "download_url": "/v1/crawls/01K72B4W9Q6S3EZJ8M5N1RTVXA/download"
}

status moves from queued to running and ends as one of:

  • done: nothing left to fetch, or max_pages reached.
  • cancelled: you cancelled it.
  • credits_exhausted: your monthly credits ran out. Pages fetched until then are kept.
  • failed: not a single page could be fetched, for example because robots.txt disallows the start page. failure_reason says why.

Patterns

include and exclude take glob patterns that are matched against the whole path of a URL, which always starts with /. Matching is case-sensitive, and the query string is not part of the path.

Pattern syntax
PatternMatches
*Any characters except /, so one path segment
**Any characters including /
?One character except /

/blog/** matches every page under /blog/. /products/*/reviews matches /products/42/reviews but not /products/42/a/reviews. /**.pdf in exclude skips PDFs.

Read the results

GET /v1/crawls/{id}/pages returns pages in the order they were stored, also while the crawl runs. Pass the next value of one answer as cursor to the next call. limit is 1–100 (default 20).

Example
curl "https://api.siftzo.com/v1/crawls/$ID/pages?limit=50&cursor=$NEXT" \
  -H "X-API-Key: $SIFTZO_API_KEY"
200 OK
{
  "id": "01K72B4W9Q6S3EZJ8M5N1RTVXA",
  "status": "running",
  "pages": [ {
      "url": "https://books.toscrape.com/catalogue/page-2.html",
      "final_url": "https://books.toscrape.com/catalogue/page-2.html",
      "depth": 1,
      "status": "done",
      "status_code": 200,
      "tier": "http",
      "credits": 1,
      "outcome": "success",
      "links_found": 73,
      "headers": { "content-type": ["text/html"] },
      "body": "<!DOCTYPE html>…",
      "body_encoding": "utf8",
      "job_id": "01K72B5A2N7Y0C4H6FQ3D8KXWE",
      "fetched_at": "2026-10-06T09:12:09.871Z"
    } ],
  "next": "4-12"
}
  • While the crawl runs, an empty pages list with the same next means nothing new yet. Wait a few seconds and ask again.
  • next is null once the crawl has ended and you have read every page.
  • A failed page has status: "failed", no body and an error with the same codes as a single scrape.
  • Bodies that are not valid UTF-8 are base64-encoded (body_encoding).
  • With format: "markdown", every page has markdown (a string, or null when the page could not be converted) and no body or body_encoding, unless you set include_html or the Markdown is null.
  • canonical_url is set when the page's <link rel="canonical"> names another URL: the page is a variant of that one.
  • duplicate_of is set when an earlier page of the crawl has the same visible text; it holds that page's url. The copy is stored and billed like any page, but its links are not followed, so a crawl does not get lost in tracking parameters, session ids or a calendar that never ends. Pages with very little text are never compared.

To get everything at once, GET /v1/crawls/{id}/download streams all stored pages as one JSONL file, one page object per line:

Download
curl -o crawl.jsonl "https://api.siftzo.com/v1/crawls/$ID/download" \
  -H "X-API-Key: $SIFTZO_API_KEY"

Crawls and their pages are kept for 7 days. After that, or for a crawl that belongs to another account, the answer is 404 not_found.

Rather than polling for the end of a crawl, pass a webhook_url. When the crawl ends, for any reason, Siftzo sends a crawl.finished event with its status and counters; see Webhooks.

Cancel

DELETE /v1/crawls/{id} stops the crawl and returns the crawl object. Pages that were already on their way finish, are billed and are stored. Cancelling a crawl that has ended changes nothing.

robots.txt and politeness

Siftzo fetches each site's robots.txt once per crawl and follows it. This cannot be turned off. Rules for the user agent siftzo apply if the file has them, otherwise the rules for *. Pages it disallows are counted in skipped_robots and never fetched.

  • No robots.txt (a 4xx answer): everything is allowed.
  • A server error, or a robots.txt that cannot be fetched at all: nothing on that host is crawled, since the site's wishes are unknown.
  • Links marked rel="nofollow" are not followed, and neither are any links on a page with <meta name="robots" content="nofollow">.

URLs are normalised before they are compared, so the same page is fetched once: the fragment is dropped, the host is lower-cased, default ports are removed, query parameters are sorted, and tracking parameters such as utm_*, gclid and fbclid are removed.

Credits and limits

Every page costs the same as a single scrape of it, on the tier that produced it, and only when it succeeded (see billing rules). Fetching robots.txt is free. Nothing is set aside for the crawl as a whole: each page reserves its credits when it is fetched, and the crawl stops with credits_exhausted when the month's credits run out.

A crawl uses your account's concurrency limit, shared with your other requests. While a large crawl runs, single scrapes may get 429 concurrency_limit more often. An account can have at most 10 crawls that have not ended; the 11th gets 429 crawl_limit.