Give it a start page. Get the site.
The Crawl API follows links from a start page and fetches up to 10,000 pages, within the patterns, depth and rate you set. robots.txt is always respected.
What it does
Follow the links of a site up to 10,000 pages, always within robots.txt.
Patterns, depth and limits
Up to 10,000 pages and 10 clicks deep.
includeandexcludeglob patterns choose which paths to follow.robots.txt, always
robots.txt is read for every site and cannot be turned off. A Crawl-delay makes the crawl slower, never faster.
Polite by default
One request per second per host unless you set
rate_per_host(0.1–5). Tracking parameters are stripped and URLs deduplicated.The same engine as scrape
Every page goes through the Scraper API's tiers, block detection and billing, so a crawl behaves like many scrapes.
HTML or Markdown
With
format: "markdown"each page comes back as Markdown, ready for an index or an LLM.Read as you go, or all at once
Page through results while the crawl runs, download everything as one JSONL file, or get a signed crawl.finished webhook.
Example
One HTTP call
POST a start URL and the limits. The answer is the crawl object; poll it, read its pages or wait for the webhook.
Official SDKs are coming soon. Until then, any HTTP client works.
Crawl API docscurl https://api.siftzo.com/v1/crawls \
-H "X-API-Key: $SIFTZO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://books.toscrape.com/",
"max_pages": 500,
"include": ["/catalogue/**"],
"format": "markdown"
}'
# Later: read the pages, 50 at a time
curl "https://api.siftzo.com/v1/crawls/$ID/pages?limit=50" \
-H "X-API-Key: $SIFTZO_API_KEY"const headers = {
"X-API-Key": process.env.SIFTZO_API_KEY,
"Content-Type": "application/json",
};
const res = await fetch("https://api.siftzo.com/v1/crawls", {
method: "POST",
headers,
body: JSON.stringify({
url: "https://books.toscrape.com/",
max_pages: 500,
include: ["/catalogue/**"],
format: "markdown",
}),
});
const crawl = await res.json();
if (!res.ok) throw new Error(crawl.error.code);
// Later: read the pages, 50 at a time
let cursor = "";
do {
const page = await fetch(
`https://api.siftzo.com/v1/crawls/${crawl.id}/pages?limit=50&cursor=${cursor}`,
{ headers },
).then((r) => r.json());
for (const p of page.pages) console.log(p.status_code, p.url);
cursor = page.next;
} while (cursor);import os
import requests
headers = {"X-API-Key": os.environ["SIFTZO_API_KEY"]}
res = requests.post(
"https://api.siftzo.com/v1/crawls",
headers=headers,
json={
"url": "https://books.toscrape.com/",
"max_pages": 500,
"include": ["/catalogue/**"],
"format": "markdown",
},
timeout=30,
)
res.raise_for_status()
crawl = res.json()
# Later: read the pages, 50 at a time
cursor = ""
while True:
page = requests.get(
f"https://api.siftzo.com/v1/crawls/{crawl['id']}/pages",
headers=headers,
params={"limit": 50, "cursor": cursor},
timeout=30,
).json()
for p in page["pages"]:
print(p["status_code"], p["url"])
cursor = page["next"]
if not cursor:
breakReference
Key parameters
urlstringrequired- The start page.
max_pagesintegerdefault 100- 1–10,000 pages fetched.
max_depthintegerdefault 3- 0–10 clicks from the start page.
include, excludeglob patterns- Up to 20 each, matched against the path.
excludewins. same_domainbooleandefaulttrue- Stay on the start page's host. Add
allow_subdomainsfor its subdomains. rate_per_hostnumberdefault 1- 0.1–5 requests per second per host.
render, country- As for the Scraper API, for every page.
formatjson | markdowndefaultjson- Markdown replaces the HTML body of every page.
webhook_urlstring- Gets a signed
crawl.finishedevent when the crawl ends.
Every parameter and response field is in the Crawl API docs.
Credits
What it costs
| Request | Credits |
|---|---|
| Crawl API page | same as a scrape of that page; robots.txt fetches are free |
| HTTP request (render=never) | 1 |
| Browser request (render=always, or auto when needed) | 10 |
| Failed request (blocked, timeout, proxy error) | 0 |
Nothing is reserved for the crawl as a whole: each page is billed as it is fetched, and the crawl stops with credits_exhausted if your limit is reached.
Questions
Can I turn robots.txt off?
No. Pages that robots.txt disallows are skipped and counted in skipped_robots; they do not count towards max_pages. See robots.txt.
How long are results kept?
7 days from when the crawl ends. Download the JSONL file to keep them longer.
Can I stop a crawl?
Yes, with DELETE /v1/crawls/{id}. Pages already on their way finish and are billed; nothing else is fetched.
Does a crawl use my concurrency?
Yes. Every page takes a slot of your plan's concurrency while it is fetched, shared with your other requests, so a crawl never goes over your plan.
More from Siftzo
- Scraper APIAny public page as HTML or Markdown, with proxies, rendering and retries handled.
- SERP APIGoogle, Bing and DuckDuckGo results as structured JSON.
- Browser APIA fresh cloud Chrome for your Playwright or Puppeteer code, over CDP.
- Data APIProducts, articles and page metadata as records with fixed fields.
- AI extractionDescribe the data with a JSON Schema or a prompt and get JSON that matches.
- Proxy accessSiftzo's residential network in your own client, billed per MB.
Try the Crawl API for free
Create an account, copy your key and send a request. No card needed for the Free plan.