Crawler policy

Our collector identifies itself with a descriptive User-Agent containing a contact address. It is not anonymous and does not attempt to appear to be a browser.

How it behaves

• A fixed delay between requests, defaulting to 2.5 seconds per source.
• A hard per-run request budget, so a pagination bug cannot become a flood.
• Backoff on 429 and 5xx responses, and it honours Retry-After.
• It collects only published record fields, and never attempts authentication.

If you operate an agency portal

If our collection is causing you a problem, or you would rather supply the same data as a scheduled export, please get in touch - a bulk file is less work for your servers than any crawler and we would prefer it.