About this crawler

It fetches publicly available facts so that quiz questions can be checked against a source instead of being made up.

It only fetches from wikipedia.org and sherdog.com. Any other host is refused by the code — it cannot be pointed anywhere else.

Rates differ by site. Against sherdog.com: at least 5 seconds before every fetch, including the first one, capped at 10 fetches per run.

Against wikipedia.org: one fetch at a time, never in parallel, with no fixed wait between them, capped at 200 per run.

Everything goes through a single queue — never two fetches at once, even across sites. A full run today fetches six pages in total.

The only headers we set are a User-Agent — the name above — and, for Wikipedia's API, a plain Accept: application/json header.

It never sends cookies, never a Referer, never an Accept-Language header, and never any header meant to make it look like a browser. Every request is stateless. Only https:// is used; anything else is rejected before it reaches the network.

Retries happen only on network failure — DNS, connection, or timeout — up to 3 attempts total.

A request is never retried because of an HTTP status: a 403 or a 429 is never retried. A retry counts against the same cap as any other fetch, since it's a new request to someone else's server, not a free one.

When a response includes a Retry-After header, that wait is honoured — up to 300 seconds — before the next fetch to that host.

This crawler does not read robots.txt today. The reason: it can only reach two sites, both chosen by hand, fetching six pages per run.

If you want us to stop, block the name above, or write to hello@gek.agency — we will remove your site. We never go around a block.

We don't store sites' pages in bulk, and we don't redistribute them. What's stored is individual facts — numbers, names, dates — with a reference to the page they came from.

One exception: six Wikipedia pages are kept unchanged as test fixtures in our private code repository, so the same text can be re-run without asking Wikipedia again.

Block the name above by User-Agent and we'll get a 403 back — we never retry that (see above). We never go around a block: no browser User-Agent, no proxy, no IP rotation. Hard line.

A guard now sits on top of the crawler itself — today, the map-builder is the only thing actually fetching. One 403 or 429 from a host is enough: it trips an alarm and blocks that host for the rest of that run. The alarm lands in our internal ops log, read as a matter of routine.

The block doesn't persist — the next run starts with a clean slate, so requests can still come back later, until you tell us. One exception: a high failure rate against wikipedia.org (over 10% in a batch) trips the alarm without blocking — Wikipedia gets flagged, not cut off.

Want us gone entirely: write to hello@gek.agency and we'll remove your site.