Guides

Webhooks

Instead of polling, give Siftzo a URL. When an async job, a batch or a crawl finishes, Siftzo sends a signed POST request to it with the result.

Set it up

Add webhook_url to any request that runs in the background:

  • POST /v1/scrape and POST /v1/serp with async: true send job.finished. Sync requests do not take a webhook_url.
  • POST /v1/batch sends batch.finished, and each item with its own webhook_url sends job.finished. See Batch API.
  • POST /v1/crawls sends crawl.finished. See Crawl API.
Example
curl https://api.siftzo.com/v1/scrape \
  -H "X-API-Key: $SIFTZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/",
    "async": true,
    "webhook_url": "https://hooks.example.com/siftzo"
  }'

The URL must use https, be at most 2,048 characters and point to a public host, on port 443 or a port of 1024 or higher. Private, loopback and internal addresses are refused with 400 invalid_request, the same as for scrape targets.

Every delivery is signed with your organization's signing secret. Find it, copy it and rotate it on the Webhooks page of the dashboard, which also lists recent deliveries.

The request

Siftzo sends POST <webhook_url> with a JSON body and these headers:

Content-Type
application/json
User-Agent
Siftzo-Webhooks/1.0
Siftzo-Event
The event type, for example job.finished.
Siftzo-Event-Id
The same as event_id in the body. It stays the same across retries and redeliveries, so use it to skip events you have already handled.
Siftzo-Signature
t=<unix seconds>,v1=<hex signature>. See Verify the signature.

Events

job.finished

An async scrape or SERP request has finished.

job.finished
{
  "event_id": "evt_01K72C0H5D8W2PZ6M9QX4TBNRA",
  "event": "job.finished",
  "created_at": "2026-10-06T12:00:00Z",
  "id": "01K72BZQ3F6T1VXN8RJ2KHCWDE",
  "product": "scrape",
  "status": "done",
  "result_url": "https://api.siftzo.com/v1/jobs/01K72BZQ3F6T1VXN8RJ2KHCWDE",
  "result": {
    "id": "01K72BZQ3F6T1VXN8RJ2KHCWDE",
    "status": "done",
    "target": { "url": "https://example.com/", "status_code": 200, "…": "…" },
    "tier": "http",
    "attempts": 1,
    "credits": 1,
    "duration_ms": 812
  }
}
productscrape | serp
For serp, result_url is /v1/serp/{id} and result is the SERP result.
statusdone | failed | expired
expired means the result was no longer stored when the delivery was made; result is then null.
batch_idstring
Only for jobs that are part of a batch.
resultobject | null
Exactly what GET result_url returns, when it is at most 256 KiB. Larger results are null: fetch result_url with your API key within 24 hours.

batch.finished

Every request of a batch has finished. status is done, cancelled or credits_exhausted.

batch.finished
{
  "event_id": "evt_01K72D1M7B3Q9XC5N2TWAHZK8E",
  "event": "batch.finished",
  "created_at": "2026-10-06T12:14:09Z",
  "id": "01K72C2V8N4R7TQ1ZK6XMBEW3D",
  "status": "done",
  "counters": { "total": 1000, "queued": 0, "running": 0, "done": 990, "failed": 10, "credits": 1180 },
  "result_url": "https://api.siftzo.com/v1/batch/01K72C2V8N4R7TQ1ZK6XMBEW3D"
}

crawl.finished

A crawl has ended. status is done, cancelled, failed or credits_exhausted.

crawl.finished
{
  "event_id": "evt_01K72E4A2Y6H8KD3P5RVWQ0ZNC",
  "event": "crawl.finished",
  "created_at": "2026-10-06T12:31:47Z",
  "id": "01K72B4W9Q6S3EZJ8M5N1RTVXA",
  "status": "done",
  "counters": {
    "discovered": 120,
    "queued": 0,
    "in_flight": 0,
    "done": 100,
    "failed": 2,
    "skipped_robots": 3,
    "credits": 140
  },
  "result_url": "https://api.siftzo.com/v1/crawls/01K72B4W9Q6S3EZJ8M5N1RTVXA/pages"
}

Verify the signature

Anyone can send a request to your URL, so check Siftzo-Signature before you trust a delivery. Pass the raw request body exactly as you received it: a body that was parsed and serialised again will not match.

Verify it yourself

Any language with HMAC-SHA256 can check a delivery:

  1. Read the raw request body as bytes, before any JSON parsing.
  2. Split the Siftzo-Signature header on commas. Take the value of t (a unix timestamp in seconds) and every value of v1; there can be more than one.
  3. Compute an HMAC-SHA256 over t, a period and the raw body (<t>.<body>). The key is your whole signing secret as UTF-8, including the whsec_ prefix. Write the result as lowercase hex.
  4. Compare it with each v1 using a constant-time comparison. Accept the delivery if any of them matches.
  5. Reject it if t is more than 5 minutes away from your current time. This stops old deliveries from being replayed. t is new on every attempt.

To test your code: with the example secret whsec_MfKQ9r2Lx7Vb4Nc8Zp1Ty6Hd3Wg0Js5Ea_Ru-Ok2Xi7, t=1759750000 and the body {"event_id":"evt_test","event":"job.finished"}, the v1 signature is 930120d61855fb2a08c8d4b09d0e289fec73f0521a8f9048da4ead148815dc61.

import { createHmac, timingSafeEqual } from "node:crypto";

/** Throws unless the delivery was signed with secret within the last 5 minutes. */
export function verifySiftzo(rawBody, header, secret, toleranceSec = 300) {
  const parts = (header ?? "").split(",").map((p) => p.trim().split("="));
  const t = parts.find(([k]) => k === "t")?.[1];
  const sigs = parts.filter(([k]) => k === "v1").map(([, v]) => v);
  if (!t || sigs.length === 0) throw new Error("malformed signature header");
  if (Math.abs(Date.now() / 1000 - Number(t)) > toleranceSec) throw new Error("too old");

  const expected = createHmac("sha256", secret).update(`${t}.${rawBody}`).digest();
  const ok = sigs.some((s) => {
    const got = Buffer.from(s, "hex");
    return got.length === expected.length && timingSafeEqual(got, expected);
  });
  if (!ok) throw new Error("invalid signature");
  return JSON.parse(rawBody);
}

With the official SDKs (coming soon)

The official SDKs for TypeScript, Python and Go are coming soon and are not published yet. Until then, use the code above. Once they are out, verification is one call:

import { verifyWebhook } from "@siftzo/sdk";

// A handler that gets a web Request (Next.js, Hono, Bun, Deno, Workers).
export async function POST(req) {
  const body = await req.text(); // the raw body, before any JSON parsing
  let event;
  try {
    event = await verifyWebhook(
      body,
      req.headers.get("Siftzo-Signature"),
      process.env.SIFTZO_WEBHOOK_SECRET,
    );
  } catch {
    return new Response("invalid signature", { status: 400 });
  }

  // Deliveries can repeat: skip event_ids you have handled before.
  console.log(event.event, event.id, event.status);
  return new Response(null, { status: 204 });
}

Rotating the secret

When you rotate the secret in the dashboard, the previous one keeps signing for 24 hours. During that time the header carries two signatures, the new one first: t=…,v1=<new>,v1=<old>. An endpoint that still has the old secret keeps working, so you can roll out the new secret without dropping deliveries.

Delivery and retries

  • Any 2xx answer within 10 seconds counts as delivered. Answer quickly and do the work afterwards, for example in a queue.
  • Anything else is retried: other status codes, timeouts, connection errors and also 3xx, because redirects are not followed.
  • Retries back off exponentially: 30 seconds, 1 minute, 2 minutes, 4 minutes and so on, up to 2 hours between attempts, with some jitter. Siftzo keeps trying for 24 hours after the first attempt.
  • Answer 410 Gone to stop retries for a delivery right away.
  • Delivery is at least once, and events can arrive in any order. Deduplicate on event_id (or the Siftzo-Event-Id header).

Deliveries are kept for 7 days. The Webhooks page in the dashboard shows each delivery with its attempts and the last answer from your endpoint, and lets you redeliver one. A redelivery has the same event_id and starts a new 24 hour retry window.

Deliveries leave through the same proxy network as scrapes, so they come from changing IP addresses. Do not allow-list IPs; check the signature instead.