Why raw HTML is the wrong input
A typical web page is mostly markup, scripts, navigation and footer links. Feeding that into a model wastes context, adds noise to embeddings and makes retrieval pick up menu text instead of content. For retrieval-augmented generation (RAG), agents and fine-tuning datasets you want the main content, with its headings, lists, tables and links intact, and nothing else.
Siftzo gives you three shapes of output for this: Markdown for text you will chunk and embed, article records for content with metadata, and JSON that matches your own schema for facts you will query.
Single pages as Markdown
Add format: "markdown" to a Scraper API request. Siftzo converts the final HTML, after any JavaScript rendering, and keeps the main or article element when the page has one. Navigation, header, footer, sidebars, scripts, forms and iframes are dropped; headings, lists, tables, code blocks, images, links and button text stay, with links made absolute. Add iframes: true to keep the content of embedded iframes as well.
curl -G https://api.siftzo.com/v1/scrape \
-H "X-API-Key: $SIFTZO_API_KEY" \
--data-urlencode "url=https://docs.example.com/getting-started" \
-d format=raw_markdownraw_markdown returns only the Markdown as the response body, which is handy for agents and command-line tools. markdown returns it inside the usual JSON result in target.markdown, along with status code and headers. Both cost the same as a JSON scrape.
Whole sites for a knowledge base
To index a documentation site, help centre or blog, start a crawl with format: "markdown". Limit it with include globs such as /docs/**, set max_pages (up to 10,000) and max_depth, and add a webhook_url to hear when it finishes.
A practical ingestion workflow:
- Start the crawl and wait for the
crawl.finishedwebhook. - Download all pages as JSONL from
/v1/crawls/{id}/download, one page per line. - Split each page's Markdown on headings, so every chunk carries its own context, and keep the page URL with each chunk for citations.
- Embed the chunks and store them in your vector index.
- Re-crawl on a schedule and replace the chunks of pages that changed.
Crawls always respect robots.txt and pace themselves per host. Results are kept for 7 days.
Structured facts with AI extraction
Some pipelines need fields rather than prose: a product's specifications, an event's date and venue, a policy's effective date. Send a JSON Schema to POST /v1/extract and Siftzo fetches the page and returns JSON that matches it.
curl https://api.siftzo.com/v1/extract \
-H "X-API-Key: $SIFTZO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://events.example.org/conference",
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"start_date": {"type": "string"},
"topics": {"type": "array", "items": {"type": "string"}}
},
"required": ["name"]
},
"prompt": "Dates as YYYY-MM-DD. Use null when a value is missing."
}'The result is in data. Extraction is synchronous only. If you already have the HTML, send it as html instead of url; nothing is fetched then. See AI extraction for schema rules.
Live pages for agents and tools
Agents that browse on a user's behalf need a page now, not from last week's crawl. A synchronous scrape with raw_markdown is the simplest tool to give them: one URL in, readable text out. Set timeout_ms (5,000–90,000) to match how long your agent can wait, and let render: "auto" decide when a page needs a real browser. It starts with a plain HTTP request that uses a real browser fingerprint and escalates to a browser only when the page is blocked or needs JavaScript, so most pages stay fast.
For pages that load content late, wait_for waits for a selector before the HTML is converted. For flows that need clicks, scrolling or form input, the Browser API gives your agent a remote browser over a WebSocket connection that common automation libraries already speak.
Failed attempts are not billed, and Siftzo tries up to three times, so an agent does not need its own retry loop for blocks. Put a short timeout and a clear fallback in the tool description, so the model knows what to do when a page cannot be read.
Articles with metadata
For news, blog posts and papers, /v1/data/article returns a record with title, authors, publisher, language, publication and modification dates, the body as Markdown and a word count. Dates let your retrieval prefer recent sources and let answers cite when something was published. Records come from the structured data on the page and readable text, so they are deterministic and cost the same as a scrape in most cases; llm_fallback covers pages that have none.
Doing it responsibly
Respect the terms of the sites you collect from, especially where they restrict reuse for training, and honour opt-outs in robots.txt. Keep source URLs with your data so you can cite and remove content on request. See responsible use.
Try it on your own data
1,000 free credits every month, no card needed. Failed requests are never billed.
Questions
What is the difference between markdown and raw_markdown?
markdown returns the Markdown inside the JSON result in target.markdown, with status and headers. raw_markdown returns only the Markdown as the response body. raw_markdown is not available for async requests.
Does Markdown conversion work on JavaScript-heavy pages?
Yes. With render set to auto, Siftzo escalates to a real browser when needed and converts the final rendered HTML to Markdown.
Can I use AI extraction without fetching the page through Siftzo?
Yes. Send the HTML you already have in the html field instead of url. Nothing is fetched and no scrape credits are charged; you pay for the extraction only.
Is there an SDK for my language?
Official SDKs are coming soon. The API is plain HTTPS with JSON, so any HTTP client works today.