Guides
Feeds
A feed runs the same job on a schedule and delivers the results: prices every morning, a search every hour, a site crawl every week. Siftzo stores every run for 7 days.
Create a feed
POST /v1/feeds with a JSON body. Feeds are managed with the same API keys as everything else, or in the dashboard under Feeds.
namestringrequired- A name for you, shown in the dashboard.
jobobjectrequiredtype,urlsandparams. See jobs.scheduleobjectrequired{"cron": "0 6 * * *"}. See schedule.destinationobjectrequired- Where results go:
webhook,s3ordownload. See destinations. statusactive | pauseddefaultactive- Create the feed paused to set it up before it runs.
curl https://api.siftzo.com/v1/feeds \
-H "X-API-Key: $SIFTZO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Daily prices",
"job": {
"type": "data",
"urls": ["https://shop.example/p/1", "https://shop.example/p/2"],
"params": { "source": "product" }
},
"schedule": { "cron": "0 6 * * *" },
"destination": { "type": "webhook", "url": "https://hooks.example.com/siftzo" }
}'const res = await fetch("https://api.siftzo.com/v1/feeds", {
method: "POST",
headers: {
"X-API-Key": process.env.SIFTZO_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({
name: "Daily prices",
job: {
type: "data",
urls: ["https://shop.example/p/1", "https://shop.example/p/2"],
params: { source: "product" },
},
schedule: { cron: "0 6 * * *" },
destination: { type: "webhook", url: "https://hooks.example.com/siftzo" },
}),
});
const feed = await res.json();
if (!res.ok) throw new Error(feed.error.code);
// Shown once: store it to verify deliveries.
console.log(feed.id, feed.next_run_at, feed.webhook_secret);import os
import requests
res = requests.post(
"https://api.siftzo.com/v1/feeds",
headers={"X-API-Key": os.environ["SIFTZO_API_KEY"]},
json={
"name": "Daily prices",
"job": {
"type": "data",
"urls": ["https://shop.example/p/1", "https://shop.example/p/2"],
"params": {"source": "product"},
},
"schedule": {"cron": "0 6 * * *"},
"destination": {"type": "webhook", "url": "https://hooks.example.com/siftzo"},
},
timeout=30,
)
feed = res.json()
res.raise_for_status()
# Shown once: store it to verify deliveries.
print(feed["id"], feed["next_run_at"], feed["webhook_secret"])package main
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"os"
)
func main() {
payload, _ := json.Marshal(map[string]any{
"name": "Daily prices",
"job": map[string]any{
"type": "data",
"urls": []string{"https://shop.example/p/1", "https://shop.example/p/2"},
"params": map[string]any{"source": "product"},
},
"schedule": map[string]string{"cron": "0 6 * * *"},
"destination": map[string]string{"type": "webhook", "url": "https://hooks.example.com/siftzo"},
})
req, _ := http.NewRequest("POST", "https://api.siftzo.com/v1/feeds", bytes.NewReader(payload))
req.Header.Set("X-API-Key", os.Getenv("SIFTZO_API_KEY"))
req.Header.Set("Content-Type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer res.Body.Close()
var feed struct {
ID string `json:"id"`
NextRunAt string `json:"next_run_at"`
WebhookSecret string `json:"webhook_secret"`
}
json.NewDecoder(res.Body).Decode(&feed)
fmt.Println(res.StatusCode, feed.ID, feed.NextRunAt)
}The answer is 201 Created with the feed. For a webhook destination it includes webhook_secret, once. Store it to verify deliveries.
{
"id": "feed_01k74g2b8n3qz5w7x1c9v4m6ta",
"name": "Daily prices",
"status": "active",
"pause_reason": null,
"job": {
"type": "data",
"urls": ["https://shop.example/p/1", "https://shop.example/p/2"],
"params": { "source": "product" }
},
"schedule": { "cron": "0 6 * * *" },
"destination": { "type": "webhook", "url": "https://hooks.example.com/siftzo" },
"next_run_at": "2026-10-07T06:00:00Z",
"last_run": null,
"created_at": "2026-10-06T09:30:12Z",
"updated_at": "2026-10-06T09:30:12Z",
"webhook_secret": "whsec_3q8Zk…"
}Jobs
job.type picks the endpoint each run calls, and job.params are that endpoint's own request fields, except url, async and format.
| type | urls | params |
|---|---|---|
scrape | 1–100, required | Scrape fields: render, country, session, wait_for, screenshot, iframes, timeout_ms, method, headers, body |
data | 1–100, required | source (required), render, country, timeout_ms, llm_fallback. See Data API |
extract | 1–100, required | schema and/or prompt, render, country, wait_for, timeout_ms. See AI extraction |
serp | Not used | SERP fields, with engine and q required |
crawl | Not used | Crawl fields, with url required |
Schedule
schedule.cron is a standard 5-field cron expression (minute, hour, day of month, month, day of week), always in UTC. The minute must be a single number, so a feed runs at most once an hour. next_run_at shows when it runs next.
| cron | Runs |
|---|---|
0 * * * * | Every hour, on the hour |
0 6 * * * | Every day at 06:00 UTC |
30 7 * * 1-5 | Weekdays at 07:30 UTC |
0 6 * * 1 | Mondays at 06:00 UTC |
*/15 * * * * | Refused: more often than hourly (400 invalid_request) |
POST /v1/feeds/{id}/run starts a run right away, outside the schedule. It is refused with 409 run_in_progress while a run is going and 429 too_soon within 15 minutes of the last start.
Destinations
Webhook
{ "type": "webhook", "url": "https://hooks.example.com/siftzo" }The URL must be https on a public host. When a run ends, Siftzo sends a POST with a JSON summary. results holds the run's results inline when they are 1 MB or less as JSONL; otherwise it is left out and you fetch download_url with your API key.
{
"event_id": "evt_run_01k76a0f4e2r8t6y3u1i9o5p7s",
"event": "feed.run.completed",
"type": "feed.run.completed",
"feed_id": "feed_01k74g2b8n3qz5w7x1c9v4m6ta",
"run_id": "run_01k76a0f4e2r8t6y3u1i9o5p7s",
"status": "done",
"records": 2,
"failed_records": 0,
"credits": 2,
"started_at": "2026-10-07T06:00:00Z",
"finished_at": "2026-10-07T06:00:41Z",
"download_url": "https://api.siftzo.com/v1/feeds/feed_01k74g2b8n3qz5w7x1c9v4m6ta/runs/run_01k76a0f4e2r8t6y3u1i9o5p7s/download",
"expires_at": "2026-10-14T06:00:41Z",
"results": [
{ "id": "01K76A0F9…", "status": "done", "source": "product", "found": true, "record": { … } },
{ "id": "01K76A0FB…", "status": "done", "source": "product", "found": true, "record": { … } }
]
}A 2xx answer within 10 seconds counts as delivered. Otherwise Siftzo tries 3 times in all, and the run's delivery.status ends as failed with the reason in delivery.error. Use Siftzo-Event-Id (evt_ plus the run ID) to ignore a delivery you have already handled.
Verify signatures
Feed deliveries are signed exactly like other webhooks, with the same headers:
Siftzo-Event: feed.run.completedSiftzo-Event-Id:evt_plus the run IDSiftzo-Signature: t=<unix seconds>,v1=<hex>, wherev1is HMAC-SHA256 over<t>.<raw body>
The one difference is the key: each feed signs with its own webhook_secret (whsec_…, shown once), not your account's webhook secret. Check it with the code in Verify the signature, passing the feed's secret. Changing the webhook URL with PATCH creates a new secret, returned once in the answer.
Webhook deliveries and S3 uploads leave through the same proxy network as scrapes, so they come from changing IP addresses. Do not allow-list IPs; check the signature instead.
S3-compatible bucket
{
"type": "s3",
"endpoint": "https://s3.example.com",
"region": "auto",
"bucket": "my-bucket",
"prefix": "siftzo/",
"access_key_id": "…",
"secret_access_key": "…"
}Any S3-compatible storage over https works. Each run is written as one object, <prefix><feed_id>/<run_id>.jsonl. Credentials are write-only: answers show access_key_id as … plus its last 4 characters and never the secret. Give the key permission to write to the prefix and nothing else.
Download only
{"type": "download"} pushes nothing. Fetch the results from the run's download_url within 7 days.
Runs and results
{
"id": "run_01k76a0f4e2r8t6y3u1i9o5p7s",
"feed_id": "feed_01k74g2b8n3qz5w7x1c9v4m6ta",
"trigger": "schedule",
"status": "done",
"started_at": "2026-10-07T06:00:00Z",
"finished_at": "2026-10-07T06:00:41Z",
"records": 2,
"failed_records": 0,
"credits": 2,
"bytes": 5821,
"delivery": { "status": "delivered", "attempts": 1, "error": null },
"error": null,
"download_url": "/v1/feeds/feed_01k74g2b8n3qz5w7x1c9v4m6ta/runs/run_01k76a0f4e2r8t6y3u1i9o5p7s/download",
"expires_at": "2026-10-14T06:00:41Z"
}statusisrunningwhile the run goes, thendone,partial(for example when the credits ran out during the run) orfailed.delivery.statusispending,delivered,failedorskipped.- Results are one JSON object per line, exactly as the API call would have answered. A crawl has one line per page.
- Every run is stored for 7 days, whatever the destination. After that,
download_urlisnulland the download answers404.
curl -o run.jsonl "https://api.siftzo.com/v1/feeds/$FEED/runs/$RUN/download" \
-H "X-API-Key: $SIFTZO_API_KEY"A feed is paused automatically after 5 failed runs in a row; pause_reason says why. Fix the cause, then resume it with PATCH {"status": "active"}.
Endpoints
| Endpoint | Body | Answer |
|---|---|---|
POST /v1/feeds | name, job, schedule, destination, status | 201 with the feed, plus webhook_secret for a webhook |
GET /v1/feeds | – | {feeds: [...]}, newest first |
GET /v1/feeds/{id} | – | The feed |
PATCH /v1/feeds/{id} | Any of name, status, job, schedule, destination | The feed, plus webhook_secret when the webhook URL changed |
DELETE /v1/feeds/{id} | – | 204 |
POST /v1/feeds/{id}/run | – | 202 with the new run (trigger manual) |
GET /v1/feeds/{id}/runs?limit= | – | {runs: [...]}, newest first. limit 1–100, default 20 |
GET /v1/feeds/{id}/runs/{run_id} | – | The run |
GET /v1/feeds/{id}/runs/{run_id}/download | – | JSONL (application/x-ndjson), 404 after 7 days |
Errors
| HTTP | Code | When |
|---|---|---|
| 400 | invalid_request | A field is missing or invalid, or the cron runs more than hourly. |
| 404 | not_found | No such feed or run in your account, or a download older than 7 days. |
| 409 | feed_limit | The account already has 20 feeds. |
| 409 | run_in_progress | Run now while a run of the feed is still going. |
| 429 | too_soon | Run now less than 15 minutes after the last run started. |
| 503 | unavailable | Feeds are briefly unavailable. Retry shortly. |
Errors inside a run, for one URL or for the whole run, are recorded in the results and in the run's error, with the codes on Errors.
Credits
Feeds have no fee of their own. Every request in a run is billed exactly like the same API call, from the account's monthly credits, and shows up in usage and logs with the key ID feed_<id>. When the credits run out during a run, it stops early with status: "partial".
Example: a data feed with 200 product URLs that run daily over HTTP costs 200 × 1 = 200 credits per run, about 6,000 a month. If the pages need a browser, it is 200 × 10 = 2,000 credits per run.
Limits
- 20 feeds per account.
- 1–100 URLs per feed for
scrape,dataandextract. - At most one run an hour on schedule, and Run now at most every 15 minutes.
- One run of a feed at a time.
- Results are kept for 7 days. Inline webhook results up to 1 MB.