robots.txt basics
robots.txt is a plain text file at the root of a host, for example https://example.com/robots.txt. It was used informally for decades and is now described in RFC 9309, the Robots Exclusion Protocol. The file is made of groups. Each group starts with one or more User-agent lines and continues with Allow and Disallow rules that match URL paths.
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
User-agent: siftzo
Disallow: /internal/
Crawl-delay: 2A crawler picks the group that matches its own user agent most specifically, and falls back to the * group. Within a group, the most specific matching rule wins, so the longest matching path decides; when an Allow and a Disallow are equally specific, Allow wins. Rules are prefixes, and * and $ can be used as wildcards and end anchors.
robots.txt is a request, not access control. It does not protect anything, and it is not a licence to collect everything it does not mention. It is, however, the clearest signal a site gives about crawling.
How the Siftzo Crawl API applies robots.txt
The Siftzo Crawl API fetches each site's robots.txt once per crawl and follows it. This cannot be turned off. Disallowed URLs are counted in skipped_robots and never fetched, and they do not count towards max_pages. Fetching robots.txt itself is free.
- Rules in a group for the user agent
siftzoapply if the file has one; otherwise the*group applies. - A 4xx answer, meaning there is no robots.txt, allows everything, as RFC 9309 specifies.
- A 5xx answer, a fetch that fails or a file that cannot be parsed disallows everything on that host. When the site's wishes are unknown, the crawl assumes the strictest answer.
- Only the first 500 KiB of the file is read, in line with the RFC's minimum.
If robots.txt disallows the start page, the crawl ends with status failed and a failure_reason that says why.
Crawl rate and Crawl-delay
Respecting robots.txt is only half of responsible crawling. The other half is not overloading the site. A crawl that requests hundreds of pages per second from a small shop can slow it down for real visitors.
The Crawl API limits requests per host with rate_per_host, between 0.1 and 5 requests per second, with a default of 1. A Crawl-delay in robots.txt, up to 30 seconds, slows the crawl further. It is never used to speed a crawl up: whichever is slower, your rate or the site's delay, wins.
curl https://api.siftzo.com/v1/crawls \
-H "X-API-Key: $SIFTZO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/docs/",
"max_pages": 300,
"max_depth": 3,
"include": ["/docs/**"],
"rate_per_host": 0.5,
"format": "markdown"
}'Start low. Most crawls are not urgent, and half a request per second still fetches thousands of pages in a few hours.
Watch the crawl's failed counter while it runs. A rising number of failures on one host often means the site is struggling or rate limiting you, and the right response is a lower rate or a pause, not more parallel crawls.
Following links: nofollow, canonicalisation and deduplication
A crawler is defined as much by which links it skips as by which it follows. The Crawl API does not follow links marked rel="nofollow", and follows no links at all from a page with <meta name="robots" content="nofollow">.
The same page often appears under many URLs: with a fragment, a different letter case in the host, an explicit default port, query parameters in another order, or tracking parameters attached. Before comparing URLs, the Crawl API normalises them: it drops the fragment, lower-cases the host, removes default ports, sorts query parameters and strips tracking parameters such as utm_*. Each page is then fetched once. That saves your credits and the site's bandwidth.
You can narrow a crawl further with include and exclude path patterns, and with same_domain, which is on by default. The Crawl API docs describe the pattern syntax.
Beyond robots.txt
Some habits that matter as much as the protocol:
- Only crawl what you need. Use path filters and a sensible page limit.
- Re-crawl on a schedule that matches how often the content changes, not as often as you can.
- Prefer public pages, and leave personal data alone.
- Read the site's terms, and stop if the site asks you to.
Siftzo's own rules are in the responsible use docs. This guide is general information, not legal advice. Whether a crawl is allowed depends on the site, the data and the law that applies to you, and robots.txt is only one input to that question.
Try it on your own data
1,000 free credits every month, no card needed. Failed requests are never billed.
Questions
Can I turn off robots.txt in the Siftzo Crawl API?
No. The Crawl API always fetches and follows robots.txt. Disallowed URLs are counted in skipped_robots and never fetched.
What happens if a site has no robots.txt?
A 4xx answer for robots.txt means everything is allowed. A server error, a failed fetch or an unparseable file means nothing on that host is crawled.
Does Crawl-delay make my crawl faster or slower?
Only slower. A Crawl-delay of up to 30 seconds slows the crawl below your rate_per_host. It never raises the rate.
Does the Scraper API check robots.txt for single requests?
The robots.txt handling described here is part of the Crawl API, which discovers and follows links on its own. For single requests you choose the URLs, so check the site's robots.txt and terms yourself.