Use case

Aggregating public real-estate and classifieds listings

Listings aggregation collects public offers from many sites into one consistent dataset. Done well, it captures the listing, not the people behind it, and treats every source with care.

Updated

What to collect, and what to leave out

A listing has two kinds of information. The offer itself: type, price, size, rooms, location at the level the listing shows it, features, photos and the date it was posted. And the person behind it: a private seller's name, phone number, email address or profile.

Aggregate the first and leave the second out. The offer is what your users compare; a private seller's personal data is personal data under rules such as the GDPR, and collecting it creates obligations and risks you do not need. Link to the original listing so people contact the seller through the source site.

Work only with listings that are public, meaning visible without logging in, and read each source's terms before you start. Some sites forbid aggregation outright; others offer feeds or partnerships, which are usually the better route. This is general information, not legal advice.

Step 1: discover listing pages with a crawl

Listing sites usually have search or category pages that link to individual listings. Start a crawl from a category page and restrict it to the paths you need.

curl https://api.siftzo.com/v1/crawls \
  -H "X-API-Key: $SIFTZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://listings.example/for-sale/apartments",
    "max_pages": 1000,
    "max_depth": 2,
    "include": ["/for-sale/**", "/listing/**"],
    "exclude": ["/account/**", "/messages/**"],
    "rate_per_host": 0.5
  }'

robots.txt is always respected and cannot be turned off; disallowed URLs are counted in skipped_robots and never fetched. A Crawl-delay in the site's robots.txt only ever slows the crawl further. A rate_per_host of 0.5 means one request every two seconds, which is a reasonable starting point for smaller sites.

Step 2: extract listing fields with a schema

Listing pages vary a lot between sites, and few publish complete structured data. AI extraction lets you define the fields once as a JSON Schema and get the same shape from every source.

  • Define only offer fields: title, price, currency, area_m2, rooms, location, posted_at, features.
  • Use the prompt to say what you do not want, for example "Do not extract names, phone numbers or email addresses."
  • Ask for units explicitly so prices and areas can be compared without guesswork.

If you crawl first, the pages are already fetched. Send the stored HTML to POST /v1/extract in the html field so nothing is fetched twice. Where a site does publish product-like structured data, the Data API may return a record directly at the cost of a scrape.

A data model that compares well

Aggregation is only useful if a listing from one site can be compared with a listing from another. Decide on a common model before collecting at scale:

  • Prices as decimals with an explicit currency, and a field that says what the price covers, such as sale price or monthly rent.
  • Areas in one unit, converted at extraction time rather than in reports.
  • Location at the precision the listing itself shows publicly, such as a district or postal area, not geocoded down to a door.
  • A status (active, inactive), first-seen and last-seen dates, and the source URL.

Some listing pages only fill in their details with JavaScript. With render: "auto", Siftzo tries a fast HTTP request first and escalates to a real browser when a page needs it, so you do not have to sort sources by technology yourself. If a detail only appears after loading, add a wait_for selector. Validate every extracted record against your model and set aside records that fail, rather than storing half-filled rows.

Step 3: keep the dataset fresh

Listings change and disappear quickly. Re-check known listing URLs with POST /v1/batch (up to 1,000 per batch, with dedupe and a batch.finished webhook) and re-run discovery crawls less often to find new ones. A 404 or 410 answer usually means the listing was removed; mark it inactive rather than deleting the history.

Deduplicate across sources on the offer fields, such as price, size and location, since the same property is often listed in several places. Keep the source URL and the time of each check with every record. Only 2xx, 3xx, 404 and 410 responses are billed; see pricing.

Checklist for responsible aggregation

  • Public pages only; never log in to reach listings.
  • Read and follow each site's terms; prefer official feeds where offered.
  • No personal data of private sellers, and no contact details.
  • Low request rates and robots.txt respected on every crawl.
  • Link back to the original listing.

Try it on your own data

1,000 free credits every month, no card needed. Failed requests are never billed.

Questions

Can I collect seller contact details from listings?

Do not collect personal data of private sellers. Aggregate the offer itself and link to the original listing so people contact the seller through the source site. Siftzo's acceptable use policy prohibits harvesting personal data.

Can I turn off robots.txt for a crawl?

No. Crawls always respect robots.txt and it cannot be disabled. A Crawl-delay in robots.txt can only slow a crawl down.

How do I handle listings that have been removed?

Re-check known URLs regularly. A 404 or 410 usually means the listing is gone; mark it inactive and keep the history. Both status codes are billed because the request succeeded.