world-index

world-index is a research crawler. It reads public news feeds and sitemaps, and some of the articles they list. It keeps headlines, times and counts to study what news outlets cover.

How it behaves

• It checks robots.txt before every request, using the name world-index. It keeps each answer for a day.
• It sends one request at a time to a site, at least five seconds apart. If robots.txt asks for a longer crawl delay, it waits that long, up to 60 seconds.
• For feeds it sends conditional requests (ETag and If-Modified-Since), so an unchanged feed costs one short reply. Article pages are fetched once, without them.
• Its User-Agent is world-index/0.1 (+https://worldindex.dev/). Requests to the SEC use the format the SEC asks for instead.

To stop it

Add this to your robots.txt. It takes effect within a day.

User-agent: world-index
Disallow: /

Contact

Write to bot@worldindex.dev. This page is at https://worldindex.dev/.