Guide

What is a web scraping API?

A web scraping API takes a URL and returns the page. Everything between those two steps, from picking an IP address to deciding whether the answer was a real page or a block, is handled for you.

Updated

The short answer

Fetching a single web page is easy. Fetching thousands of them a day, from sites that change their markup, rate limit unknown visitors and build content with JavaScript, is not. A web scraping API is a service that does the fetching for you: you send an HTTP request with the URL you want, and you get back the HTML, Markdown or structured data of that page.

The value is not in the HTTP request itself but in the operational work around it. Without an API, your team runs the proxies, keeps browser fingerprints current, operates a fleet of headless browsers, writes retry logic and learns to tell a real page from a challenge page. With an API, those become parameters on a request.

curl https://api.siftzo.com/v1/scrape \
  -H "X-API-Key: $SIFTZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/products/42", "country": "de" }'

The answer is JSON with the target's status code, headers and body, which tier produced it, how many attempts it took and what it cost. The quickstart walks through a first request.

What a scraping API handles for you

Proxies

Requests from a single server IP get rate limited or refused quickly. A scraping API sends traffic through a proxy network so each attempt can leave from a different address, and lets you choose the country a request comes from. Siftzo uses residential proxies for every request; the trade-offs are covered in residential vs datacenter proxies.

Fingerprints

Servers look at more than the IP address. The TLS handshake and HTTP/2 settings of a typical scripting library look nothing like a browser, and that difference alone can trigger a block. Siftzo's HTTP tier uses the TLS and HTTP/2 fingerprint of Chrome, so even a plain HTTP fetch looks like a browser on the wire.

Rendering

Many pages ship an empty shell and fill it with JavaScript. A scraping API can run a real browser for those pages, wait for the content and return the rendered HTML. See scraping JavaScript-rendered pages for how to tell when that is needed.

Retries and block detection

A page that returns 200 can still be a challenge page. A useful API inspects status codes, headers and body for the signatures of anti-bot systems, treats those responses as blocked, and retries with a new IP or a heavier tier. In Siftzo each request gets up to three attempts within its timeout_ms, and with render: "auto" a blocked HTTP attempt moves to the browser.

Paying only for successful requests

The billing model tells you a lot about a scraping API. If blocked attempts are billed, the provider earns more when its own anti-blocking fails. Siftzo bills only responses that count as a result: status codes 2xx, 3xx, 404 and 410. Blocks, timeouts and proxy errors cost nothing, and neither do the extra attempts it took to get a page.

A browser render costs more than a plain HTTP fetch, because it uses far more compute. You pay for the tier that produced the result. Current prices are on the pricing page, and the exact billing rules are in the rate limits and credits docs.

Build or buy?

Building your own scraper is reasonable when you fetch a handful of friendly sites, the pages are static HTML and you already run infrastructure. A cron job and an HTTP client go a long way.

The calculation changes when any of these is true:

  • You need many target sites, and each one breaks in its own way when it changes its protection.
  • You need results from specific countries.
  • Some pages need a browser, and running browsers at scale means memory limits, crash recovery and sandboxing.
  • Your engineers spend more time on blocks and retries than on using the data.

Buying moves those costs to a per-request price. The trade-off is less control over low-level details such as exactly which IP is used. Siftzo keeps some of that control available: the Browser API lets you drive a remote Chrome with your own Playwright or Puppeteer code, and sticky sessions keep one IP across requests.

Beyond raw HTML

Most scraping projects do not want HTML; they want prices, titles or search results. Modern scraping APIs therefore offer higher-level endpoints next to the raw fetch:

  • Search result parsing that returns organic results as JSON.
  • Structured records for products, articles and page metadata, read from the structured data pages already publish.
  • AI extraction that fills a JSON Schema you define, for pages without structured data.
  • Site crawling that follows links, respects robots.txt and stores every page.

Markdown output (format: "markdown") is useful when the page is going into a language model or a search index, since it removes scripts, styles and navigation while keeping headings, lists, tables and links.

Using it responsibly

A scraping API makes fetching easier; it does not change what you are allowed to fetch. Collect public data, respect rate limits and site terms, and avoid personal data. Siftzo is hosted in the EU and publishes its rules in the responsible use docs. For crawling, read robots.txt and responsible crawling.

Try it on your own data

1,000 free credits every month, no card needed. Failed requests are never billed.

Questions

Is a web scraping API the same as a proxy?

No. A proxy only forwards your traffic through another IP address. A scraping API also handles fingerprints, browser rendering, retries and block detection, and returns the page or structured data.

Do I pay for blocked requests?

Not with Siftzo. Only responses with status 2xx, 3xx, 404 or 410 are billed. Blocks, timeouts and proxy errors cost nothing, and extra attempts are free.

Can I use my existing scraping code?

Yes. You can call the REST API from any language that speaks HTTP, or connect existing Playwright or Puppeteer scripts to the Browser API over CDP. Official SDKs are coming soon.