Use case

Web data collection for academic research

Researchers need web data that is collected politely, documented precisely and reproducible later. Siftzo's crawl and data tools are built around those constraints rather than around volume.

Updated

What research data collection needs

Research projects use the web as a source for corpora of news, policy documents, public statements, product descriptions or online discourse. The requirements differ from commercial scraping:

  • A documented method: which pages, from where, when, and with which settings.
  • Polite collection that respects site owners' wishes.
  • Clean text that can go straight into analysis tools.
  • Files that can be archived and shared with reviewers.

The Crawl API and the Data API cover these with little code, so more of the project's time goes into research rather than infrastructure.

Plan the collection before you run it

A web corpus is a sample, and the sampling decisions belong in the research design. Before the first request, write down:

  • The population: which sites and which sections of them, and why those and not others.
  • The period: when you collect, and whether you collect once or repeatedly.
  • The limits: how many pages per site, how deep from the start page, and which page types you exclude.
  • What you will do with pages that fail, redirect or are disallowed.

These map one to one onto crawl parameters, which makes the method easy to describe and repeat. Run a small pilot crawl first with a low max_pages and read a sample of the output by hand. Pilots catch the usual problems early: an include pattern that is too broad, archive pages that multiply the page count, or a section that mostly holds navigation.

Funding is often fixed, so give the project its own team API key with a credit cap. Only successful fetches and 404 or 410 answers are billed, and URLs skipped because of robots.txt are free.

Step 1: a polite, bounded crawl

Define the corpus as a crawl configuration: a start URL, max_pages (up to 10,000), max_depth (0–10) and include and exclude globs for the sections you study.

curl https://api.siftzo.com/v1/crawls \
  -H "X-API-Key: $SIFTZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.example.org/politics/",
    "max_pages": 5000,
    "max_depth": 4,
    "include": ["/politics/**"],
    "exclude": ["/**/comments", "/**.pdf"],
    "rate_per_host": 0.5,
    "format": "markdown"
  }'

Politeness is built in. robots.txt is always respected and cannot be turned off; disallowed URLs are recorded as skipped_robots and never fetched. A Crawl-delay in robots.txt only ever slows the crawl. rate_per_host caps requests per second to each host between 0.1 and 5; for research, a value at or below the default of 1 is a sensible choice. The guide to robots.txt and responsible crawling explains the rules in more detail.

Step 2: reproducible downloads

When the crawl finishes, download the whole result as JSONL, one page per line, from /v1/crawls/{id}/download. Each line holds the URL, final URL after redirects, depth, status code and the content in the format you chose.

curl -o corpus.jsonl "https://api.siftzo.com/v1/crawls/$ID/download" \
  -H "X-API-Key: $SIFTZO_API_KEY"

For reproducibility, archive three things together:

  1. The crawl configuration you sent, exactly as sent.
  2. The crawl object with its counters and start and finish times.
  3. The JSONL file, with a checksum.

Results are kept on Siftzo for 7 days, so download them promptly. The web changes; your archived file is the record of what the pages said when you collected them. Report the collection dates, the configuration and the robots.txt-skipped count in your methods section.

Step 3: clean text and article metadata

With format: "markdown", each page holds its main content as Markdown: headings, paragraphs, lists and tables, without navigation, footers and scripts. That is usually ready for tokenisation, topic modelling or close reading.

For news and blog corpora, the /v1/data/article source adds structure: title, authors, publisher, language, section, publication and modification dates as RFC 3339 timestamps, the body as Markdown and a word count. The parser field records which method produced the record (structured data or readable text), which you can report and filter on. Records from the deterministic parsers are repeatable: the same page gives the same record.

If you use llm_fallback or AI extraction for pages without structured data, mark those records in your data, since model output is not deterministic in the same way.

Ethics and responsible use

Follow your institution's ethics process, especially when a corpus contains text written by individuals. Collect only what the research question needs, avoid personal data where you can, and store it securely when you cannot. Respect site terms. Siftzo is hosted in the EU. See responsible use and pricing; this is general information, not legal advice.

Try it on your own data

1,000 free credits every month, no card needed. Failed requests are never billed.

Questions

Can I disable robots.txt for research purposes?

No. Crawls always respect robots.txt and it cannot be disabled. Disallowed URLs are counted in skipped_robots and never fetched, which you can report in your methods.

How do I make a crawl reproducible?

Archive the exact crawl configuration, the crawl object with its counters and timestamps, and the downloaded JSONL file with a checksum. Results stay on Siftzo for 7 days.

Are article records deterministic?

Records from the structured data and readability parsers are: the same page gives the same record. The parser field says which one was used. Records from the AI fallback are marked with fallback: llm.