Use case

Market research from public web data

Desk research on a market used to mean weeks of opening tabs. With a crawl, a few searches and structured records, you can collect the public picture in a form you can count, filter and compare.

Updated

Start with the questions, not the data

Web data is plentiful and most of it is noise for any given question. Before writing a request, write down what you want to know. Typical market research questions that public pages answer well:

  • Who shows up when buyers search for this category, and in which markets?
  • What do competing products cost, and how are ranges structured?
  • Which features, claims and terms appear on competitors' pages?
  • What are companies publishing about, and how often?

Each question maps to a Siftzo product: search results come from the SERP API, site content from the Crawl API, and products and articles as fixed records from the Data API.

Step 1: find the players with search

Run the queries a buyer would type, per market. country and language localise the result page, so the same query in several markets shows you who competes where.

curl https://api.siftzo.com/v1/serp \
  -H "X-API-Key: $SIFTZO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"engine": "bing", "q": "project management software", "country": "gb", "language": "en"}'

Collect the domains from results.organic and results.ads. Ads tell you who pays to appear; organic results tell you who the engine considers relevant. related_searches and people_also_ask are a good source of the vocabulary buyers use, which you can feed back in as new queries.

Keep the query list in a file with the engine, country and language for each entry, and store every response with its date. When you repeat the research later, you can run the same list again and see which players have entered or left the market.

Step 2: crawl the relevant parts of each site

For each competitor, crawl only the sections that answer your questions, such as product, pricing and blog pages. Use include globs like /products/** and /blog/**, set a modest max_pages, and ask for format: "markdown". Markdown keeps the main content without navigation, footers and scripts, which makes the pages far easier to read, search and summarise.

Crawls respect robots.txt automatically, and rate_per_host (default 1 request per second) keeps the load on each site low. Read pages while the crawl runs from /v1/crawls/{id}/pages, or download everything as JSONL when it is done. Results are kept for 7 days, so move what you need into your own storage.

Step 3: turn pages into records

Unstructured text is fine for reading; analysis needs fields.

Products and prices

Send product URLs to /v1/data/product to get name, brand, category, offers with normalized price and availability, and rating. Put the records in a spreadsheet or database and you can compare price ranges and positioning directly.

Articles and announcements

/v1/data/article returns title, authors, publication and modification dates, the body as Markdown and a word count. Dates let you measure how often a competitor publishes and what about.

Anything else

For fields no standard covers, such as plan names, feature lists or integration counts, define a JSON Schema and use AI extraction. The result matches the schema, so every competitor ends up in the same columns.

Step 4: analyse and keep it current

With records in place, the analysis is ordinary data work. Group products by brand and category to see how ranges are built. Plot prices per market to see where a competitor positions itself. Count articles per month and topic to see where their attention goes. Compare which domains appear across many of your SERP queries; those are the players buyers actually meet.

Markets move, so decide up front which parts of the picture need refreshing. Search results and prices change often and are worth re-checking regularly. Positioning, feature pages and company descriptions change slowly, and a quarterly crawl is usually enough. Store every run with its date rather than overwriting the last one; the differences between runs are often the most interesting finding.

Keep the cost predictable by limiting crawls with max_pages and tight include globs, and by giving the project its own team API key with a credit cap.

Doing it responsibly

Market research works on public, company-level information. Keep to public pages, respect the terms of the sites you visit, and do not collect personal data you do not need; the Data API has no sources for people by design. Cite sources in your findings and keep the request IDs, so every number can be traced back to a page and a date. See responsible use for Siftzo's rules and pricing for what each request type costs.

Try it on your own data

1,000 free credits every month, no card needed. Failed requests are never billed.

Questions

Can I research several countries at once?

Yes. Run the same SERP query with different country and language values, and set country on scrapes and Data API requests to see each market's local pages and prices.

Why use Markdown instead of HTML?

Markdown keeps headings, lists, tables and links from the main content and drops navigation, scripts and styling. It is much smaller and easier to read, search or pass to a language model.

How long are crawl results stored?

Crawls and their pages are kept for 7 days. Download the JSONL file or page through the results and store what you need in your own system.