How to tell if a page needs JavaScript
Fetch the page without a browser and look for the data you want in the raw HTML. If it is there, you do not need JavaScript, even if the page uses a modern front-end framework. Many such sites render on the server first.
curl -s "https://api.siftzo.com/v1/scrape?url=https://example.com/p/42&render=never&format=raw" \
-H "X-API-Key: $SIFTZO_API_KEY" | grep -c "product-price"Signs that a page builds its content in the browser:
- The body is mostly an empty container such as
<div id="root"></div>and a list of script tags. - The text you see in your browser is missing from the raw HTML.
- Viewing the page with JavaScript disabled shows a loading spinner or a blank screen.
Before you reach for a browser, check two shortcuts. Data is often embedded in the HTML as JSON for the front-end to pick up, either as JSON-LD or in a script tag holding the initial state. And the page may load its data from a JSON endpoint that you could fetch directly. Both are cheaper and more stable than rendering.
render=auto, never and always
Siftzo has two tiers. The HTTP tier fetches the page with the TLS and HTTP/2 fingerprint of Chrome but does not run JavaScript. The browser tier runs Chrome in a fresh profile, executes scripts and can wait for elements. The render parameter chooses between them:
auto(default) starts on the tier that usually works for the host, which is HTTP unless the host has been blocking it. If an HTTP attempt is blocked, the next attempt uses the browser.neverstays on HTTP, also for retries.alwaysuses only the browser.
Note what auto does not do: it escalates on blocks, not on missing content. A page that returns an empty shell with status 200 is a valid response, and auto will return it. For pages you know are built with JavaScript, use render: "always" or add wait_for, which forces the browser.
Waiting for the content with wait_for
A browser that captures the page as soon as it loads can still miss data that a script fetches a moment later. wait_for takes a CSS selector and makes the browser wait until a matching element exists before it captures the HTML, within the request timeout.
import requests, os
res = requests.post(
"https://api.siftzo.com/v1/scrape",
headers={"X-API-Key": os.environ["SIFTZO_API_KEY"]},
json={
"url": "https://example.com/app/listings",
"render": "always",
"wait_for": "[data-testid=listing-card]",
"timeout_ms": 45000,
},
)
result = res.json()
print(result["tier"], result["attempts"], len(result["target"]["body"]))Choose a selector for the data itself, not for a container that exists from the start. If the selector never appears, the request runs until its timeout, so give browser requests a timeout_ms of at least 15 seconds. The rendering docs list the per-attempt limits.
Per-host learning
With auto, Siftzo keeps a running success rate per host and tier. When HTTP succeeds less than half the time on a host, new requests to that host start directly on the browser and skip the HTTP attempt that would probably fail. That saves time on every request.
Protection changes, so the decision is not permanent. Every 20th request to such a host tries HTTP again, and if it starts working, the host moves back to the cheaper tier. The memory expires after 7 days without traffic. You get the benefit without maintaining a list of which domains need a browser.
The tier field in every response tells you which tier produced the result, so you can see where your requests end up. The attempts field shows how many tries it took; a host that regularly needs two or three attempts is a sign to check your request rate or set render explicitly.
The cost difference
A browser render uses far more compute than an HTTP fetch, and it is priced accordingly; the pricing page has the current numbers. You pay for the tier that produced the result, and only when it succeeded. Blocked HTTP attempts before an escalation are free.
To keep costs down:
- Use the default
autounless you know a page needs JavaScript. - Use
neverfor targets you know work over HTTP, such as sitemaps, feeds and server-rendered pages. - Look for embedded JSON before rendering. The structured data guide shows how much is often already there.
When you need interaction rather than a single render, see headless browser scraping and the Browser API.
Try it on your own data
1,000 free credits every month, no card needed. Failed requests are never billed.
Questions
Does render=auto detect empty JavaScript shells?
No. auto escalates to the browser when an attempt is blocked or the host is known to need it. A page that returns an empty shell with status 200 is returned as is. Use render=always or wait_for for pages you know need JavaScript.
Can I combine wait_for with render=never?
No. wait_for and screenshot need the browser, so they force the browser tier with auto and are rejected with never.
How do I know which tier produced a result?
Every response has a tier field, http or browser, and an attempts field with the number of attempts it took.