Use case

Price monitoring with a web data API

Price monitoring is a loop: find the product pages, read price and availability, store a snapshot, compare and act. Siftzo handles the fetching and the parsing so the loop stays small.

Updated

What a price monitor actually needs

Most price monitors start as a script that downloads a page and pulls a number out of the HTML with a CSS selector. That works until the shop changes its template, starts blocking repeated requests from one address, or renders the price with JavaScript. Then the script returns nothing, or worse, the wrong number.

A dependable monitor needs four things:

  • A list of product URLs that stays current as catalogues change.
  • A fetch that gets through on the first or second try, from the right country, so you see the price a local visitor sees.
  • A record with fixed fields: price, currency and availability in the same shape for every shop.
  • A way to run hundreds or thousands of checks and know when they are finished.

Siftzo covers each of these with one API: the Crawl API for catalogue discovery, the Scraper API for reliable fetching, the Data API for normalized product records, and Batch with webhooks for scheduling at scale.

Step 1: discover the catalogue with a crawl

If you already have a list of product URLs, skip ahead. If not, start a crawl from a category page and limit it to the paths that hold products. include and exclude take glob patterns, max_depth keeps the crawl close to the start page, and rate_per_host sets how many requests per second go to the shop (0.1–5, default 1).

curl https://api.siftzo.com/v1/crawls \
  -H "X-API-Key: $SIFTZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://shop.example/category/shoes",
    "max_pages": 2000,
    "max_depth": 3,
    "include": ["/category/**", "/p/**"],
    "exclude": ["/**/reviews"],
    "rate_per_host": 1,
    "webhook_url": "https://hooks.example.com/siftzo"
  }'

robots.txt is always respected, so paths the shop has disallowed are skipped and never fetched. When the crawl is done you get a crawl.finished webhook, and you can download every page as JSONL from /v1/crawls/{id}/download. Results are kept for 7 days. Filter the URLs that match your product pattern and save them as your watch list. Re-run the crawl now and then to pick up new products and drop removed ones.

Step 2: read price and availability as a record

Most shops already publish structured data for search engines. The Data API reads it (JSON-LD, microdata and Open Graph, then the HTML itself) and returns a product record with fixed fields, whatever the template looks like.

curl -G https://api.siftzo.com/v1/data/product \
  -H "X-API-Key: $SIFTZO_API_KEY" \
  --data-urlencode "url=https://shop.example/p/1" \
  -d country=de

Each entry in offers has a price as a string (such as "119.95", so no precision is lost), a currency as an ISO 4217 code, and an availability that is always one of in_stock, out_of_stock, preorder, backorder, discontinued, limited or unknown. That normalization is what lets you compare shops side by side. The record also carries sku, gtin and mpn when the page states them, which helps you match the same product across shops.

The parser field tells you where the record came from. When a page has no structured data, found is false. Pass llm_fallback: true to have AI extraction read the page with a fixed product schema instead; it is priced as extraction, so turn it on only for the shops that need it.

Step 3: run checks in bulk and get notified

For large watch lists, send pages in groups with POST /v1/batch. One batch holds up to 1,000 scrape requests, runs them as async jobs within your concurrency, and with dedupe: true fetches identical requests only once. Give it a webhook_url and you get one batch.finished event when every job is done instead of polling.

Batch runs scrapes, so you get the page back, not a product record. Two patterns work well:

  1. Call the Data API per URL from your own worker, a few requests at a time within your plan's concurrency. Simple, and every response is already a record.
  2. Use Batch to fetch the pages, parse the structured data yourself, and send the HTML of the odd page with no structured data to POST /v1/extract with your own JSON Schema. Extraction from HTML you already have fetches nothing, so no scrape credits are charged for it.

Webhooks are signed with HMAC-SHA256 in the Siftzo-Signature header, delivered at least once and retried for up to 24 hours. Verify the signature and store event_id so a retry does not trigger a second alert. See webhooks versus polling for the trade-offs.

Step 4: store snapshots and compare

Store each check as a row: URL, time, price, currency, availability and the request ID. Keep the history rather than overwriting it; trends and the date of the last change are often more useful than the current price. Compare prices as decimals, not floats, and only compare offers in the same currency.

Billing follows outcomes. Only responses with a 2xx, 3xx, 404 or 410 status are billed; blocks, timeouts and proxy errors cost nothing. A 404 or 410 is itself useful: it usually means the product was removed. See pricing for credits per request type.

Doing it responsibly

Check as often as prices actually change, not as often as you can. Keep rate_per_host low on crawls, read only public product pages, and stay out of logged-in areas such as accounts and baskets. The responsible use page covers the rest.

Try it on your own data

1,000 free credits every month, no card needed. Failed requests are never billed.

Questions

How do I see the price a visitor in a specific country sees?

Set country to an ISO 3166-1 alpha-2 code on the scrape or Data API request. The request then exits from that country, and the shop typically shows its local price and currency.

What happens when a shop has no structured data?

The Data API returns found: false. You can pass llm_fallback: true to let AI extraction read the page with a fixed product schema, or call POST /v1/extract with your own schema.

Am I charged when a shop blocks a request?

No. Only 2xx, 3xx, 404 and 410 responses are billed. Blocks, timeouts and proxy errors cost nothing, and Siftzo retries up to three attempts before giving up.

Can Siftzo run the checks on a schedule for me?

Yes. A feed runs the same requests on a schedule, at most once an hour, and delivers each run to a webhook or your own storage bucket. Results are kept for 7 days.