Guide

Headless browser scraping

A headless browser runs a page the way a visitor’s browser does: it executes JavaScript, loads data from APIs and builds the final DOM. You need one less often than you might think, but when you do, nothing else will work.

Updated

When you need a browser

An HTTP client gets the HTML the server sends. For most content sites, shops and documentation, that HTML already contains the text and the structured data you want. A browser adds nothing but cost.

You need a browser when:

  • The server sends an empty application shell and the content is loaded by JavaScript.
  • Content appears only after a script runs, for example prices fetched from a separate API after the page loads.
  • You need a screenshot of the page.
  • You need to interact: click a tab, fill a search box, scroll to load more, or go through several steps.
  • A site's protection refuses clients that do not run JavaScript.

The guide on scraping JavaScript-rendered pages shows how to check which case a page falls into.

What a browser costs

A browser render is much heavier than an HTTP request. It downloads scripts, runs them, holds a full DOM in memory and often makes dozens of extra requests. That shows up as slower responses and higher prices per page.

Siftzo's browser tier keeps this down by not loading images, fonts and media during a scrape (unless you ask for a screenshot), and by running each task in a fresh Chrome profile. Still, a browser render costs more than an HTTP fetch; see the pricing page. With render: "auto", Siftzo starts on HTTP and only uses the browser when the HTTP attempt is blocked or the host is known to need it, so you do not pay for a browser on pages that do not need one.

wait_for and screenshots

The hardest part of browser scraping is knowing when the page is ready. The load event can fire long before a script has fetched and rendered the data. Fixed sleeps are either too short or waste time.

wait_for takes a CSS selector. The browser waits until an element matching it is on the page, within the request timeout, and then captures the HTML. Pick a selector for the content you actually need, such as the price element or the list of results, rather than a wrapper that exists from the start.

curl https://api.siftzo.com/v1/scrape \
  -H "X-API-Key: $SIFTZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/app/catalog",
    "render": "always",
    "wait_for": ".product-card",
    "screenshot": true
  }'

With screenshot: true, the response includes a base64-encoded PNG of the viewport next to the HTML. Because images are skipped, screenshots show layout and text but not pictures. Both wait_for and screenshot force the browser tier, also with render: "auto", and cannot be combined with render: "never". The rendering docs cover timeouts and retries.

Driving a remote browser over CDP

A single scrape request covers load, wait and capture. When you need clicks, form input or several pages in one flow, you need to control the browser yourself. The Siftzo Browser API gives you a Chrome in the cloud that you connect to over the Chrome DevTools Protocol, so your existing Playwright or Puppeteer code keeps working:

import { chromium } from "playwright";

const browser = await chromium.connectOverCDP(
  "wss://api.siftzo.com/v1/browser?country=de&timeout=300",
  { headers: { "X-API-Key": process.env.SIFTZO_API_KEY } },
);
const context = browser.contexts()[0] ?? (await browser.newContext());
const page = await context.newPage();
await page.goto("https://example.com/search");
await page.fill("input[name=q]", "trail shoes");
await page.press("input[name=q]", "Enter");
await page.waitForSelector(".result");
console.log(await page.locator(".result h2").allTextContents());
await browser.close();

The connection takes country, session for a sticky IP, and timeout between 60 and 900 seconds. A session with no CDP messages in either direction for 60 seconds is closed. Puppeteer connects the same way with puppeteer.connect({ browserWSEndpoint }).

Unlike a scrape request, the Browser API loads images, fonts and media and does not retry or switch IPs on a block. Block resources you do not need with page.route to keep pages fast. Sessions are billed by time, not by page; see the Browser API docs.

Isolation and security

Running untrusted pages in a browser is a security question as much as a scraping one. Every Browser API connection starts a new Chrome with an empty profile, and closing the connection ends it. Nothing is kept between sessions: no cookies, no local storage, no cache.

The browser runs with Chrome's sandbox on, and a few things are blocked:

  • Only http, https, data: and about:blank can be opened.
  • Downloads and local file uploads are blocked.
  • All traffic goes through Siftzo's residential proxies; per-context proxies are refused.

If you need to keep a login or a cart across steps, do those steps within one connection, or send the cookies you need yourself.

Each open session also uses one slot of your plan's concurrency while it runs, including idle time, so close the browser as soon as the flow is done. The close code Siftzo sends when it ends a session, for example after the timeout or 60 seconds of inactivity, tells you why.

Try it on your own data

1,000 free credits every month, no card needed. Failed requests are never billed.

Questions

Do I need a headless browser to scrape most sites?

No. Most pages include their content in the HTML the server sends, and an HTTP fetch is faster and cheaper. Use a browser when content is built by JavaScript, when you need screenshots or interaction, or when a site refuses clients without JavaScript.

Can I use Playwright or Puppeteer with Siftzo?

Yes. Connect them to wss://api.siftzo.com/v1/browser over CDP. Each connection gets a fresh, sandboxed Chrome that sends its traffic through residential proxies.

Can I get a screenshot of the whole page?

Yes. Set screenshot: true and screenshot_full_page: true. The browser loads the page's images and fonts for a screenshot, and captures up to 10,000 pixels of height. Without screenshot_full_page you get the visible part of a desktop browser window.

How long can a Browser API session run?

Between 60 and 900 seconds, set with the timeout parameter (default 300). A session that is idle for 60 seconds is closed.