Guides
Webhooks
Instead of polling, give Siftzo a URL. When an async job, a batch or a crawl finishes, Siftzo sends a signed POST request to it with the result.
Set it up
Add webhook_url to any request that runs in the background:
POST /v1/scrapeandPOST /v1/serpwithasync: truesendjob.finished. Sync requests do not take awebhook_url.POST /v1/batchsendsbatch.finished, and each item with its ownwebhook_urlsendsjob.finished. See Batch API.POST /v1/crawlssendscrawl.finished. See Crawl API.
curl https://api.siftzo.com/v1/scrape \
-H "X-API-Key: $SIFTZO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/",
"async": true,
"webhook_url": "https://hooks.example.com/siftzo"
}'The URL must use https, be at most 2,048 characters and point to a public host, on port 443 or a port of 1024 or higher. Private, loopback and internal addresses are refused with 400 invalid_request, the same as for scrape targets.
Every delivery is signed with your organization's signing secret. Find it, copy it and rotate it on the Webhooks page of the dashboard, which also lists recent deliveries.
The request
Siftzo sends POST <webhook_url> with a JSON body and these headers:
Content-Typeapplication/jsonUser-AgentSiftzo-Webhooks/1.0Siftzo-Event- The event type, for example
job.finished. Siftzo-Event-Id- The same as
event_idin the body. It stays the same across retries and redeliveries, so use it to skip events you have already handled. Siftzo-Signaturet=<unix seconds>,v1=<hex signature>. See Verify the signature.
Events
job.finished
An async scrape or SERP request has finished.
{
"event_id": "evt_01K72C0H5D8W2PZ6M9QX4TBNRA",
"event": "job.finished",
"created_at": "2026-10-06T12:00:00Z",
"id": "01K72BZQ3F6T1VXN8RJ2KHCWDE",
"product": "scrape",
"status": "done",
"result_url": "https://api.siftzo.com/v1/jobs/01K72BZQ3F6T1VXN8RJ2KHCWDE",
"result": {
"id": "01K72BZQ3F6T1VXN8RJ2KHCWDE",
"status": "done",
"target": { "url": "https://example.com/", "status_code": 200, "…": "…" },
"tier": "http",
"attempts": 1,
"credits": 1,
"duration_ms": 812
}
}productscrape | serp- For
serp,result_urlis/v1/serp/{id}andresultis the SERP result. statusdone | failed | expiredexpiredmeans the result was no longer stored when the delivery was made;resultis thennull.batch_idstring- Only for jobs that are part of a batch.
resultobject | null- Exactly what
GET result_urlreturns, when it is at most 256 KiB. Larger results arenull: fetchresult_urlwith your API key within 24 hours.
batch.finished
Every request of a batch has finished. status is done, cancelled or credits_exhausted.
{
"event_id": "evt_01K72D1M7B3Q9XC5N2TWAHZK8E",
"event": "batch.finished",
"created_at": "2026-10-06T12:14:09Z",
"id": "01K72C2V8N4R7TQ1ZK6XMBEW3D",
"status": "done",
"counters": { "total": 1000, "queued": 0, "running": 0, "done": 990, "failed": 10, "credits": 1180 },
"result_url": "https://api.siftzo.com/v1/batch/01K72C2V8N4R7TQ1ZK6XMBEW3D"
}crawl.finished
A crawl has ended. status is done, cancelled, failed or credits_exhausted.
{
"event_id": "evt_01K72E4A2Y6H8KD3P5RVWQ0ZNC",
"event": "crawl.finished",
"created_at": "2026-10-06T12:31:47Z",
"id": "01K72B4W9Q6S3EZJ8M5N1RTVXA",
"status": "done",
"counters": {
"discovered": 120,
"queued": 0,
"in_flight": 0,
"done": 100,
"failed": 2,
"skipped_robots": 3,
"credits": 140
},
"result_url": "https://api.siftzo.com/v1/crawls/01K72B4W9Q6S3EZJ8M5N1RTVXA/pages"
}Verify the signature
Anyone can send a request to your URL, so check Siftzo-Signature before you trust a delivery. Pass the raw request body exactly as you received it: a body that was parsed and serialised again will not match.
Verify it yourself
Any language with HMAC-SHA256 can check a delivery:
- Read the raw request body as bytes, before any JSON parsing.
- Split the
Siftzo-Signatureheader on commas. Take the value oft(a unix timestamp in seconds) and every value ofv1; there can be more than one. - Compute an HMAC-SHA256 over
t, a period and the raw body (<t>.<body>). The key is your whole signing secret as UTF-8, including thewhsec_prefix. Write the result as lowercase hex. - Compare it with each
v1using a constant-time comparison. Accept the delivery if any of them matches. - Reject it if
tis more than 5 minutes away from your current time. This stops old deliveries from being replayed.tis new on every attempt.
To test your code: with the example secret whsec_MfKQ9r2Lx7Vb4Nc8Zp1Ty6Hd3Wg0Js5Ea_Ru-Ok2Xi7, t=1759750000 and the body {"event_id":"evt_test","event":"job.finished"}, the v1 signature is 930120d61855fb2a08c8d4b09d0e289fec73f0521a8f9048da4ead148815dc61.
import { createHmac, timingSafeEqual } from "node:crypto";
/** Throws unless the delivery was signed with secret within the last 5 minutes. */
export function verifySiftzo(rawBody, header, secret, toleranceSec = 300) {
const parts = (header ?? "").split(",").map((p) => p.trim().split("="));
const t = parts.find(([k]) => k === "t")?.[1];
const sigs = parts.filter(([k]) => k === "v1").map(([, v]) => v);
if (!t || sigs.length === 0) throw new Error("malformed signature header");
if (Math.abs(Date.now() / 1000 - Number(t)) > toleranceSec) throw new Error("too old");
const expected = createHmac("sha256", secret).update(`${t}.${rawBody}`).digest();
const ok = sigs.some((s) => {
const got = Buffer.from(s, "hex");
return got.length === expected.length && timingSafeEqual(got, expected);
});
if (!ok) throw new Error("invalid signature");
return JSON.parse(rawBody);
}import hashlib
import hmac
import json
import time
def verify_siftzo(raw_body: bytes, header: str, secret: str, tolerance: int = 300):
"""Raises ValueError unless the delivery was signed with secret recently."""
parts = [p.strip().split("=", 1) for p in (header or "").split(",") if "=" in p]
t = next((v for k, v in parts if k == "t"), None)
sigs = [v for k, v in parts if k == "v1"]
if t is None or not sigs:
raise ValueError("malformed signature header")
if abs(time.time() - int(t)) > tolerance:
raise ValueError("too old")
signed = t.encode() + b"." + raw_body
expected = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
if not any(hmac.compare_digest(expected, s) for s in sigs):
raise ValueError("invalid signature")
return json.loads(raw_body)With the official SDKs (coming soon)
The official SDKs for TypeScript, Python and Go are coming soon and are not published yet. Until then, use the code above. Once they are out, verification is one call:
import { verifyWebhook } from "@siftzo/sdk";
// A handler that gets a web Request (Next.js, Hono, Bun, Deno, Workers).
export async function POST(req) {
const body = await req.text(); // the raw body, before any JSON parsing
let event;
try {
event = await verifyWebhook(
body,
req.headers.get("Siftzo-Signature"),
process.env.SIFTZO_WEBHOOK_SECRET,
);
} catch {
return new Response("invalid signature", { status: 400 });
}
// Deliveries can repeat: skip event_ids you have handled before.
console.log(event.event, event.id, event.status);
return new Response(null, { status: 204 });
}import os
import siftzo
from flask import Flask, request
app = Flask(__name__)
@app.post("/siftzo")
def siftzo_webhook():
try:
event = siftzo.verify_webhook(
request.get_data(), # the raw body, before any JSON parsing
request.headers.get("Siftzo-Signature"),
os.environ["SIFTZO_WEBHOOK_SECRET"],
)
except siftzo.WebhookVerificationError:
return "invalid signature", 400
# Deliveries can repeat: skip event_ids you have handled before.
print(event["event"], event["id"], event["status"])
return "", 204func handleSiftzo(w http.ResponseWriter, r *http.Request) {
payload, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 4<<20))
if err != nil {
http.Error(w, "bad body", http.StatusBadRequest)
return
}
// 0 uses the default tolerance of 5 minutes.
secret := os.Getenv("SIFTZO_WEBHOOK_SECRET")
if err := siftzo.VerifyWebhook(payload, r.Header.Get("Siftzo-Signature"), secret, 0); err != nil {
http.Error(w, "invalid signature", http.StatusBadRequest)
return
}
var event struct {
EventID string `json:"event_id"`
Event string `json:"event"`
ID string `json:"id"`
Status string `json:"status"`
}
json.Unmarshal(payload, &event)
// Deliveries can repeat: skip event_ids you have handled before.
w.WriteHeader(http.StatusNoContent)
}Rotating the secret
When you rotate the secret in the dashboard, the previous one keeps signing for 24 hours. During that time the header carries two signatures, the new one first: t=…,v1=<new>,v1=<old>. An endpoint that still has the old secret keeps working, so you can roll out the new secret without dropping deliveries.
Delivery and retries
- Any
2xxanswer within 10 seconds counts as delivered. Answer quickly and do the work afterwards, for example in a queue. - Anything else is retried: other status codes, timeouts, connection errors and also
3xx, because redirects are not followed. - Retries back off exponentially: 30 seconds, 1 minute, 2 minutes, 4 minutes and so on, up to 2 hours between attempts, with some jitter. Siftzo keeps trying for 24 hours after the first attempt.
- Answer
410 Goneto stop retries for a delivery right away. - Delivery is at least once, and events can arrive in any order. Deduplicate on
event_id(or theSiftzo-Event-Idheader).
Deliveries are kept for 7 days. The Webhooks page in the dashboard shows each delivery with its attempts and the last answer from your endpoint, and lets you redeliver one. A redelivery has the same event_id and starts a new 24 hour retry window.
Deliveries leave through the same proxy network as scrapes, so they come from changing IP addresses. Do not allow-list IPs; check the signature instead.