Crawler
PXke Algorand is an independent newspaper covering the Algorand ecosystem. Our crawler fetches public web pages so that we can find and check the sources we write about.
What the crawler fetches and why
The crawler follows links from sources we track, such as project sites, documentation, developer blogs, forums and official announcements. It reads the pages to decide whether they are relevant to Algorand, to quote them, and to check that a link still works. It does not log in, submit forms, or post anything on the sites it visits.
How the crawler identifies itself
Every request carries this User-Agent:
algorand-platform-newspaper/1.0 (+https://algorand.pxke.me/bot)
Requests come from these IP ranges:
5.135.131.229/322001:41d0:303:15e5::1/128
IP ranges (JSON): /bot/ips.json
Requests are also signed with Web Bot Auth (RFC 9421 HTTP Message Signatures, the IETF web-bot-auth drafts). The public key is published at https://algorand.pxke.me/.well-known/http-message-signatures-directory, and each signed request names our origin in its Signature-Agent header.
Pacing and limits
- robots.txt is fetched once per host and cached. Any URL that robots.txt disallows for our user agent is skipped before it is fetched.
- At most 50 pages per domain in any rolling seven-day window.
- Each URL is fetched at most once every 30 days.
- Bulk passes over one site pause 0.3 seconds between requests.
- Each request times out after 12 seconds. A response body is capped at 20 MB; larger bodies are cut off.
- Redirects are followed one hop at a time, and each hop is checked again. Private and internal addresses are never fetched.
Robots.txt and opting out
We honor robots.txt, including Crawl-delay for our user agent (algorand-platform-newspaper) or *. If you want us to stop, add a disallow rule for algorand-platform-newspaper, or write to us and we will remove your site from our crawl list.
Contact
Questions about the crawler or a request to stop it: our contact page.