Guides
Batch API
Send up to 1,000 scrape requests in one call. Siftzo runs them as async jobs within your concurrency, and you read the results page by page or get a webhook when they are all done.
Create a batch
POST /v1/batch with a JSON body of at most 10 MB.
requestsobject[]required- 1–1,000 scrape requests, each with the same fields as POST /v1/scrape. Every item runs as an async job, so
asynccan be left out (trueis accepted).formatmust bejsonormarkdown. An item can have its ownwebhook_urlto getjob.finishedfor its job. webhook_urlstring- An
httpsURL that gets a signedbatch.finishedevent when every request has finished. See Webhooks. dedupebooleandefaultfalse- Run identical requests (the same URL with the same options) once. Their entries in
jobsshare one job ID, and you pay once.
curl https://api.siftzo.com/v1/batch \
-H "X-API-Key: $SIFTZO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"requests": [
{"url": "https://example.com/"},
{"url": "https://example.org/", "format": "markdown"},
{"url": "https://example.net/", "render": "always", "country": "de"}
],
"webhook_url": "https://hooks.example.com/siftzo"
}'Every item is checked before anything runs. If one is invalid, the whole batch is refused with 400 invalid_request (or 403 target_not_allowed), and the message starts with the item's position, such as requests[2]: . The answer to a valid batch is 202 Accepted with a Location header:
{
"batch_id": "01K72C2V8N4R7TQ1ZK6XMBEW3D",
"status": "queued",
"total": 3,
"jobs": [
{ "id": "01K72C2V9A1B2C3D4E5F6G7H8J", "url": "https://example.com/" },
{ "id": "01K72C2V9K9M8N7P6Q5R4S3T2V", "url": "https://example.org/" },
{ "id": "01K72C2V9W1X2Y3Z4A5B6C7D8E", "url": "https://example.net/" }
],
"status_url": "/v1/batch/01K72C2V8N4R7TQ1ZK6XMBEW3D"
}jobs has one entry per request, in the order you sent them. Each is an ordinary async job: GET /v1/jobs/{id} works for it, and shows queued until it is dispatched.
With the official SDKs (coming soon)
The official SDKs for TypeScript, Python and Go are coming soon and are not published yet. Until then, use the HTTP requests above from any client. Once they are out, a batch looks like this:
import { Siftzo } from "@siftzo/sdk";
const client = new Siftzo(process.env.SIFTZO_API_KEY);
const batch = await client.batch({
requests: [
{ url: "https://example.com/", format: "markdown" },
{ url: "https://example.org/", format: "markdown" },
],
webhook_url: "https://hooks.example.com/siftzo",
});
// Later, for example when batch.finished arrives:
const page = await client.getBatch(batch.batch_id, { limit: 100 });
console.log(page.status, page.counters, page.next);import os
import siftzo
client = siftzo.Siftzo(os.environ["SIFTZO_API_KEY"])
batch = client.batch(
[
{"url": "https://example.com/", "format": "markdown"},
{"url": "https://example.org/", "format": "markdown"},
],
webhook_url="https://hooks.example.com/siftzo",
)
# Later, for example when batch.finished arrives:
page = client.get_batch(batch["batch_id"], limit=100)
print(page["status"], page["counters"], page["next"])client := siftzo.NewClient(os.Getenv("SIFTZO_API_KEY"))
batch, err := client.Batch(ctx, &siftzo.BatchParams{
Requests: []siftzo.ScrapeParams{
{URL: "https://example.com/", Format: siftzo.FormatMarkdown},
{URL: "https://example.org/", Format: siftzo.FormatMarkdown},
},
WebhookURL: "https://hooks.example.com/siftzo",
})
if err != nil {
return err
}
// Later, for example when batch.finished arrives:
page, err := client.GetBatch(ctx, batch.BatchID, &siftzo.PageOptions{Limit: 100})
if err != nil {
return err
}
fmt.Println(page.Status, page.Counters.Done, page.Next)Read the results
GET /v1/batch/{id} returns the batch and one page of results, in request order. Pass the next value of one answer as cursor to the next call. limit is 1–100 (default 20). A page also ends early once it reaches 32 MB.
curl "https://api.siftzo.com/v1/batch/$ID?limit=100&cursor=$NEXT" \
-H "X-API-Key: $SIFTZO_API_KEY"{
"batch_id": "01K72C2V8N4R7TQ1ZK6XMBEW3D",
"status": "running",
"counters": { "total": 1000, "queued": 900, "running": 5, "done": 90, "failed": 5, "credits": 95 },
"created_at": "2026-10-06T12:00:00Z",
"finished_at": null,
"webhook_url": null,
"results": [
{
"index": 0,
"id": "01K72C2V9A1B2C3D4E5F6G7H8J",
"url": "https://example.com/",
"status": "done",
"result": { "id": "01K72C2V9A1B2C3D4E5F6G7H8J", "status": "done", "target": { "…": "…" } }
},
{
"index": 1,
"id": "01K72C2V9K9M8N7P6Q5R4S3T2V",
"url": "https://example.org/",
"status": "queued",
"result": null
}
],
"next": "20"
}statusmoves fromqueuedtorunningand ends asdone,cancelledorcredits_exhausted.finished_atis set when it ends.counterssum up all requests: how many are queued, running, done and failed, and the credits used so far.- Each result has a
statusofqueued,running,done,failedorexpired. For finished jobs,resultis the same JSON that GET /v1/jobs/{id} returns. nextisnullafter the last entry. Pages always cover every request, so read them again for entries that were still running.
Results are kept for 24 hours after each job finishes; after that the entry shows expired. The batch itself is kept for 7 days. A batch that does not exist or belongs to another account gives 404 not_found.
Get a webhook
With a top-level webhook_url, Siftzo sends one batch.finished event with the final status and counters when the batch has ended, however it ended. An item with its own webhook_url also sends job.finished for that job, with the result inline and a batch_id. See Webhooks for payloads and signatures.
Cancel
DELETE /v1/batch/{id} cancels the batch and returns the batch object with empty results and next: null. Jobs that were not dispatched yet become failed with the error code cancelled and cost nothing. Jobs that were already running finish and are billed.
Deduplication
With dedupe: true, requests that are identical (the same URL and the same options) run once. Every one of them still has its own entry in jobs and in the results, all with the same job ID. Without it, every request runs and is billed, even when it repeats another.
Credits and concurrency
Nothing is reserved when you create a batch. Each job reserves its maximum price when it is dispatched and is settled like any async job, so you pay for what runs (see billing rules). An organization that is already out of credits gets 402 credits_exhausted when it creates a batch. If credits run out while a batch runs, it stops with credits_exhausted and the remaining jobs fail with the code credits_exhausted.
A batch never runs more jobs at once than your organization's concurrency limit, which it shares with your other requests. While a large batch runs, single scrapes may get 429 concurrency_limit more often. An organization can have at most 20 batches that have not finished; the next one gets 429 batch_limit.