Guide

Handling anti-bot challenges responsibly

Sites use anti-bot systems to protect themselves from abuse. A good scraper recognises when it has been challenged, backs off or changes approach, and never pretends to be a human solving a puzzle.

Updated

What a challenge page is

When a site's protection is unsure about a visitor, it can answer with something other than the requested page. Common forms are:

  • Interstitials: a short "checking your browser" page that runs a script and then reloads to the real page if the check passes.
  • Interactive challenges: a checkbox, an image puzzle or another captcha that asks a human to do something.
  • Access-denied pages: a plain refusal, often with a reference ID.
  • Rate limiting: a 429 Too Many Requests because one IP has sent too much.

Several of these come back with status 200, so a scraper that only checks the status code will happily store a challenge page as if it were the product page it asked for.

Detecting a block

Siftzo looks at the status code, the response headers and the body of every attempt and compares them with the signatures of common anti-bot systems and generic captcha pages. A response that matches counts as blocked. So does a 429 from the target, since it means the exit IP is being rate limited.

Not every error is a block. A 500 from the site itself, or a 403 without an anti-bot signature, is the site's real answer. It is returned to you as a result with its status code, so you can decide what to do, and it is not billed either.

A blocked page is never returned as a result. If every attempt is blocked, you get an error instead:

const res = await fetch("https://api.siftzo.com/v1/scrape", {
  method: "POST",
  headers: {
    "X-API-Key": process.env.SIFTZO_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com/item/42" }),
});
const body = await res.json();

if (res.status === 424 && body.error?.code === "target_blocked") {
  // Every attempt was blocked. Nothing was billed.
  // Slow down, or try again later instead of retrying at once.
} else if (res.ok) {
  console.log(body.tier, body.attempts, body.target.status_code);
}

The full list of codes is in the errors reference.

Retrying with a new IP and escalating

A block is often specific to one IP address or one kind of client. Siftzo gives each request up to three attempts within its timeout_ms, and without a sticky session each attempt leaves from a new residential IP.

With render: "auto", a blocked HTTP attempt escalates to a real Chrome. Interstitials that resolve on their own, without anyone clicking, get a short time to finish in the browser. If Chrome is blocked too, a later attempt can use a second browser engine with a different fingerprint. Siftzo also remembers per host how well each tier works, so hosts that keep blocking HTTP start directly on the browser. The details are in the rendering docs and in scraping JavaScript-rendered pages.

No captcha solving

An interactive challenge is a site asking whether a human is present. Siftzo does not answer that question falsely: it never clicks captcha checkboxes, never uses captcha solving services and never pays people to solve puzzles. If an interactive challenge is shown, the attempt counts as blocked, and the next attempt, if any is left, tries a different IP or engine as described above.

This has a practical consequence. Some pages sit behind interactive challenges for every automated visitor, and Siftzo will not get them. That is a deliberate limit, not a temporary gap.

If a target you care about is always challenged, consider whether the site offers the data another way: an official API, a public data export, a feed or a sitemap. Asking the site owner for access is often faster than any technical workaround.

Blocked attempts are not billed

Only responses with status 2xx, 3xx, 404 or 410 are billed. Blocked attempts, timeouts and proxy errors cost nothing, and a request that ends in target_blocked costs nothing at all. When a request succeeds after an escalation, you pay for the tier that produced the result. Prices per tier are on the pricing page.

Scraping responsibly

Anti-bot systems exist because some traffic does harm. Staying on the right side of that line is mostly common sense:

  • Collect public data only. Do not scrape pages behind a login you are not entitled to use, and avoid personal data.
  • Respect rate limits. If a site starts returning 429 or blocks, slow down instead of adding IPs. For crawls, see robots.txt and responsible crawling.
  • Read and respect the site's terms of use.
  • Cache what you have fetched rather than fetching it again.

Siftzo's rules are in the acceptable use policy and the responsible use docs. This guide is general information, not legal advice; whether a specific project is allowed depends on the site, the data and the law that applies to you.

Try it on your own data

1,000 free credits every month, no card needed. Failed requests are never billed.

Questions

Does Siftzo solve captchas?

No. Siftzo never clicks captcha checkboxes and does not use captcha solving services. If an interactive challenge is shown, the attempt counts as blocked.

Am I charged when a request is blocked?

No. Blocked attempts, timeouts and proxy errors are not billed. Only responses with status 2xx, 3xx, 404 or 410 are.

Is a 403 from a site always a block?

No. A 403 without an anti-bot signature is the site's own answer. Siftzo returns it as a result with its status code and does not bill it.