Async jobs
A synchronous scrape holds the connection open until the result is ready. That is simple, but it ties up a connection and a worker on your side for as long as the target takes. With async: true, POST /v1/scrape answers 202 Accepted at once with a job ID, and the work runs in the background. The same works for POST /v1/serp. Batches and crawls are always asynchronous.
You then have two ways to get the result: poll for it, or receive a webhook. Results of async jobs are kept for 24 hours either way.
Polling
Polling means asking GET /v1/jobs/{id} until status changes from queued or running to done or failed.
curl https://api.siftzo.com/v1/jobs/$JOB_ID \
-H "X-API-Key: $SIFTZO_API_KEY"Polling is the right choice when:
- Your code cannot receive inbound HTTPS, for example a script on a laptop or a job behind a firewall.
- You run a handful of jobs and wait for them in the same process.
- You want the simplest thing that works while you prototype.
Poll with a sensible interval, a few seconds at first and longer for crawls, and stop when the job ends. Polling once a second for hundreds of jobs mostly produces "still running" answers and adds load on both sides without telling you anything new.
Webhooks
With a webhook, you pass a webhook_url and Siftzo sends a signed POST to it when the work is done. There are three events:
job.finishedfor an async scrape or SERP request, including batch items that have their ownwebhook_url.batch.finishedwhen every request of a batch has finished.crawl.finishedwhen a crawl ends, for any reason.
curl https://api.siftzo.com/v1/scrape \
-H "X-API-Key: $SIFTZO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/",
"async": true,
"webhook_url": "https://hooks.example.com/siftzo"
}'The job.finished body includes the result itself when it is at most 256 KiB; larger results are null and you fetch result_url with your API key. Webhooks fit pipelines with many jobs, long crawls and any system that already reacts to events. The URL must use HTTPS and point to a public host.
Verifying the signature
Anyone can send a request to your webhook URL, so check the Siftzo-Signature header before trusting a delivery. It looks like t=<unix seconds>,v1=<hex>. The v1 value is an HMAC-SHA256 over the string <t>.<raw body>, keyed with your whole signing secret, including the whsec_ prefix.
import { createHmac, timingSafeEqual } from "node:crypto";
const TOLERANCE_SECONDS = 300;
// rawBody: the request body as a Buffer, before any JSON parsing.
export function verifySiftzoSignature(rawBody, header, secret) {
let t = null;
const signatures = [];
for (const part of header.split(",")) {
const i = part.indexOf("=");
const key = part.slice(0, i).trim();
const value = part.slice(i + 1).trim();
if (key === "t") t = value;
if (key === "v1") signatures.push(value);
}
if (t === null || signatures.length === 0) return false;
if (Math.abs(Date.now() / 1000 - Number(t)) > TOLERANCE_SECONDS) return false;
const expected = createHmac("sha256", secret)
.update(t + ".")
.update(rawBody)
.digest();
return signatures.some((sig) => {
const given = Buffer.from(sig, "hex");
return given.length === expected.length && timingSafeEqual(given, expected);
});
}Three details that matter:
- Use the raw body. A body that was parsed and serialised again will not match.
- Reject timestamps more than 5 minutes from your clock.
tis new on every attempt, so this blocks replays without rejecting retries. - Accept any matching
v1. For 24 hours after you rotate the secret, the header carries two signatures, new first, so endpoints with the old secret keep working while you roll out the new one.
Official SDKs with a built-in verifier are coming soon. Until then, the code above is all you need; the webhooks docs include a test vector to check it against.
Delivery, retries and idempotency
- Only a 2xx answer within 10 seconds counts as delivered. Acknowledge quickly and do the work afterwards, for example from a queue.
- Redirects are not followed; a 3xx is retried like any other failure, so point
webhook_urlat the final URL. - Retries back off from 30 seconds, doubling up to 2 hours between attempts, and continue for 24 hours after the first attempt.
- Answer
410 Goneto stop retries for a delivery immediately. - Delivery is at least once and events can arrive in any order. Deduplicate on
event_id, which is also sent as theSiftzo-Event-Idheader and stays the same across retries.
A sound setup combines both approaches: webhooks for speed, and an occasional poll of anything still open after a deadline, in case your endpoint was down for longer than the retry window.
A minimal handler
In practice, a webhook handler does four things in order: read the raw body, verify the signature, record the event_id and return 200. Everything else, such as fetching a large result from result_url, parsing pages or writing to your database, happens afterwards in a worker. If verification fails, return 401 and log it; a legitimate delivery will be retried. Deliveries come from changing IP addresses, so do not rely on an IP allow-list; the signature is the check.
Try it on your own data
1,000 free credits every month, no card needed. Failed requests are never billed.
Questions
How long are async job results kept?
24 hours. Fetch them with GET /v1/jobs/{id} within that time, whether you poll or receive a webhook.
What counts as a successful webhook delivery?
A 2xx response within 10 seconds. Anything else, including redirects and timeouts, is retried with backoff for 24 hours. A 410 Gone stops retries.
Can the same webhook arrive twice?
Yes. Delivery is at least once. Use event_id, or the Siftzo-Event-Id header, to skip events you have already handled.
Which key do I use to verify signatures?
Your whole signing secret as UTF-8, including the whsec_ prefix, as the HMAC-SHA256 key over the timestamp, a period and the raw body.