Guides

Feeds

A feed runs the same job on a schedule and delivers the results: prices every morning, a search every hour, a site crawl every week. Siftzo stores every run for 7 days.

Create a feed

POST /v1/feeds with a JSON body. Feeds are managed with the same API keys as everything else, or in the dashboard under Feeds.

namestringrequired
A name for you, shown in the dashboard.
jobobjectrequired
type, urls and params. See jobs.
scheduleobjectrequired
{"cron": "0 6 * * *"}. See schedule.
destinationobjectrequired
Where results go: webhook, s3 or download. See destinations.
statusactive | pauseddefault active
Create the feed paused to set it up before it runs.
curl https://api.siftzo.com/v1/feeds \
  -H "X-API-Key: $SIFTZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Daily prices",
    "job": {
      "type": "data",
      "urls": ["https://shop.example/p/1", "https://shop.example/p/2"],
      "params": { "source": "product" }
    },
    "schedule": { "cron": "0 6 * * *" },
    "destination": { "type": "webhook", "url": "https://hooks.example.com/siftzo" }
  }'

The answer is 201 Created with the feed. For a webhook destination it includes webhook_secret, once. Store it to verify deliveries.

201 Created
{
  "id": "feed_01k74g2b8n3qz5w7x1c9v4m6ta",
  "name": "Daily prices",
  "status": "active",
  "pause_reason": null,
  "job": {
    "type": "data",
    "urls": ["https://shop.example/p/1", "https://shop.example/p/2"],
    "params": { "source": "product" }
  },
  "schedule": { "cron": "0 6 * * *" },
  "destination": { "type": "webhook", "url": "https://hooks.example.com/siftzo" },
  "next_run_at": "2026-10-07T06:00:00Z",
  "last_run": null,
  "created_at": "2026-10-06T09:30:12Z",
  "updated_at": "2026-10-06T09:30:12Z",
  "webhook_secret": "whsec_3q8Zk…"
}

Jobs

job.type picks the endpoint each run calls, and job.params are that endpoint's own request fields, except url, async and format.

Job types
typeurlsparams
scrape1–100, requiredScrape fields: render, country, session, wait_for, screenshot, iframes, timeout_ms, method, headers, body
data1–100, requiredsource (required), render, country, timeout_ms, llm_fallback. See Data API
extract1–100, requiredschema and/or prompt, render, country, wait_for, timeout_ms. See AI extraction
serpNot usedSERP fields, with engine and q required
crawlNot usedCrawl fields, with url required

Schedule

schedule.cron is a standard 5-field cron expression (minute, hour, day of month, month, day of week), always in UTC. The minute must be a single number, so a feed runs at most once an hour. next_run_at shows when it runs next.

Cron examples
cronRuns
0 * * * *Every hour, on the hour
0 6 * * *Every day at 06:00 UTC
30 7 * * 1-5Weekdays at 07:30 UTC
0 6 * * 1Mondays at 06:00 UTC
*/15 * * * *Refused: more often than hourly (400 invalid_request)

POST /v1/feeds/{id}/run starts a run right away, outside the schedule. It is refused with 409 run_in_progress while a run is going and 429 too_soon within 15 minutes of the last start.

Destinations

Webhook

destination
{ "type": "webhook", "url": "https://hooks.example.com/siftzo" }

The URL must be https on a public host. When a run ends, Siftzo sends a POST with a JSON summary. results holds the run's results inline when they are 1 MB or less as JSONL; otherwise it is left out and you fetch download_url with your API key.

POST to your URL
{
  "event_id": "evt_run_01k76a0f4e2r8t6y3u1i9o5p7s",
  "event": "feed.run.completed",
  "type": "feed.run.completed",
  "feed_id": "feed_01k74g2b8n3qz5w7x1c9v4m6ta",
  "run_id": "run_01k76a0f4e2r8t6y3u1i9o5p7s",
  "status": "done",
  "records": 2,
  "failed_records": 0,
  "credits": 2,
  "started_at": "2026-10-07T06:00:00Z",
  "finished_at": "2026-10-07T06:00:41Z",
  "download_url": "https://api.siftzo.com/v1/feeds/feed_01k74g2b8n3qz5w7x1c9v4m6ta/runs/run_01k76a0f4e2r8t6y3u1i9o5p7s/download",
  "expires_at": "2026-10-14T06:00:41Z",
  "results": [
    { "id": "01K76A0F9…", "status": "done", "source": "product", "found": true, "record": { … } },
    { "id": "01K76A0FB…", "status": "done", "source": "product", "found": true, "record": { … } }
  ]
}

A 2xx answer within 10 seconds counts as delivered. Otherwise Siftzo tries 3 times in all, and the run's delivery.status ends as failed with the reason in delivery.error. Use Siftzo-Event-Id (evt_ plus the run ID) to ignore a delivery you have already handled.

Verify signatures

Feed deliveries are signed exactly like other webhooks, with the same headers:

  • Siftzo-Event: feed.run.completed
  • Siftzo-Event-Id: evt_ plus the run ID
  • Siftzo-Signature: t=<unix seconds>,v1=<hex>, where v1 is HMAC-SHA256 over <t>.<raw body>

The one difference is the key: each feed signs with its own webhook_secret (whsec_…, shown once), not your account's webhook secret. Check it with the code in Verify the signature, passing the feed's secret. Changing the webhook URL with PATCH creates a new secret, returned once in the answer.

Webhook deliveries and S3 uploads leave through the same proxy network as scrapes, so they come from changing IP addresses. Do not allow-list IPs; check the signature instead.

S3-compatible bucket

destination
{
  "type": "s3",
  "endpoint": "https://s3.example.com",
  "region": "auto",
  "bucket": "my-bucket",
  "prefix": "siftzo/",
  "access_key_id": "…",
  "secret_access_key": "…"
}

Any S3-compatible storage over https works. Each run is written as one object, <prefix><feed_id>/<run_id>.jsonl. Credentials are write-only: answers show access_key_id as … plus its last 4 characters and never the secret. Give the key permission to write to the prefix and nothing else.

Download only

{"type": "download"} pushes nothing. Fetch the results from the run's download_url within 7 days.

Runs and results

GET /v1/feeds/{id}/runs/{run_id}
{
  "id": "run_01k76a0f4e2r8t6y3u1i9o5p7s",
  "feed_id": "feed_01k74g2b8n3qz5w7x1c9v4m6ta",
  "trigger": "schedule",
  "status": "done",
  "started_at": "2026-10-07T06:00:00Z",
  "finished_at": "2026-10-07T06:00:41Z",
  "records": 2,
  "failed_records": 0,
  "credits": 2,
  "bytes": 5821,
  "delivery": { "status": "delivered", "attempts": 1, "error": null },
  "error": null,
  "download_url": "/v1/feeds/feed_01k74g2b8n3qz5w7x1c9v4m6ta/runs/run_01k76a0f4e2r8t6y3u1i9o5p7s/download",
  "expires_at": "2026-10-14T06:00:41Z"
}
  • status is running while the run goes, then done, partial (for example when the credits ran out during the run) or failed.
  • delivery.status is pending, delivered, failed or skipped.
  • Results are one JSON object per line, exactly as the API call would have answered. A crawl has one line per page.
  • Every run is stored for 7 days, whatever the destination. After that, download_url is null and the download answers 404.
Download a run
curl -o run.jsonl "https://api.siftzo.com/v1/feeds/$FEED/runs/$RUN/download" \
  -H "X-API-Key: $SIFTZO_API_KEY"

A feed is paused automatically after 5 failed runs in a row; pause_reason says why. Fix the cause, then resume it with PATCH {"status": "active"}.

Endpoints

Feed endpoints
EndpointBodyAnswer
POST /v1/feedsname, job, schedule, destination, status201 with the feed, plus webhook_secret for a webhook
GET /v1/feeds–{feeds: [...]}, newest first
GET /v1/feeds/{id}–The feed
PATCH /v1/feeds/{id}Any of name, status, job, schedule, destinationThe feed, plus webhook_secret when the webhook URL changed
DELETE /v1/feeds/{id}–204
POST /v1/feeds/{id}/run–202 with the new run (trigger manual)
GET /v1/feeds/{id}/runs?limit=–{runs: [...]}, newest first. limit 1–100, default 20
GET /v1/feeds/{id}/runs/{run_id}–The run
GET /v1/feeds/{id}/runs/{run_id}/download–JSONL (application/x-ndjson), 404 after 7 days

Errors

Feed errors
HTTPCodeWhen
400invalid_requestA field is missing or invalid, or the cron runs more than hourly.
404not_foundNo such feed or run in your account, or a download older than 7 days.
409feed_limitThe account already has 20 feeds.
409run_in_progressRun now while a run of the feed is still going.
429too_soonRun now less than 15 minutes after the last run started.
503unavailableFeeds are briefly unavailable. Retry shortly.

Errors inside a run, for one URL or for the whole run, are recorded in the results and in the run's error, with the codes on Errors.

Credits

Feeds have no fee of their own. Every request in a run is billed exactly like the same API call, from the account's monthly credits, and shows up in usage and logs with the key ID feed_<id>. When the credits run out during a run, it stops early with status: "partial".

Example: a data feed with 200 product URLs that run daily over HTTP costs 200 × 1 = 200 credits per run, about 6,000 a month. If the pages need a browser, it is 200 × 10 = 2,000 credits per run.

Limits

  • 20 feeds per account.
  • 1–100 URLs per feed for scrape, data and extract.
  • At most one run an hour on schedule, and Run now at most every 15 minutes.
  • One run of a feed at a time.
  • Results are kept for 7 days. Inline webhook results up to 1 MB.