DevoteBot
The crawler behind Devote Democracy's civic data: who runs it, what it reads, how often, and how to reach us or turn it away.
Who we are
Devote Democracy is a non-partisan civic app. It shows people who represents them, what their parliaments and councils are doing, and how public money reaches political parties. DevoteBot reads a short, hand-picked list of public pages for that record. It is run by Devote's data pipeline, not by people browsing.
How to recognise it
Every request carries:
User-Agent: DevoteBot/1.0 (+https://devotedemocracy.com/bot) From: [email protected]
Each request is also signed with Web Bot Auth: HTTP Message Signatures (RFC 9421) with an Ed25519 key, carried in the Signature-Agent, Signature-Input and Signature headers. Each signature covers the host and expires after a minute. Our public keys are published at:
https://www.devotedemocracy.com/.well-known/http-message-signatures-directory
That directory is itself signed, and lists one key, with this key ID (its JWK thumbprint, RFC 7638), which appears as keyid in every signature:
okxQsNcSL3MjBqBFoLIV3rk5S6C1c11rPBXENtL6Klg
We never present ourselves as a browser. We never solve, bypass or get round a bot challenge, and we never use proxies or other people's networks. If a site challenges us, we stop and ask its owner.
What it reads
Only public pages and files, under open licences (the Open Government Licence and the Open Parliament Licence). We always credit the source and link back to it. Every source behind Devote's data, with its licence and credit, is listed on our data sources page.
| Publisher | What | How often |
|---|---|---|
| The Electoral Commission | Candidate spending data (UK general election 2024; Scottish Parliament and Senedd 2021 and 2026; Northern Ireland Assembly 2022), election data files, concluded investigations, and the public funding of parties | Monthly to yearly, per item |
| House of Commons | Short money allocations, and the Commons Library briefings on Short money and political financing | Monthly to quarterly |
| House of Lords Library | The briefing on Cranborne money | Yearly |
| UK Parliament | Written statements on Short money (API) | Monthly |
How it behaves
- It runs once a week, and reads an item only when that item is due. Most items are read monthly or less often.
- It leaves at least 5 seconds between requests to the same site (more if your robots.txt sets a
Crawl-delay), and makes at most 60 requests a week across every site. - It obeys robots.txt (RFC 9309) under the product token
DevoteBot, falling back to*. - When a site refuses it, it may use a copy the Internet Archive already holds. That copy is marked as archived and dated, and is checked by a person before use. We never ask the Archive to capture a page for us.
Opting out, or letting it in
To stop DevoteBot reading your site, add this to your robots.txt. It is honoured from the next run:
User-agent: DevoteBot Disallow: /
If your site challenges automated requests and you are happy for DevoteBot to read the pages above, either of these works. Neither opens your site to anyone else:
- allow Cloudflare's verified bots (how);
- add a Skip rule for requests from DevoteBot carrying our signature (how).
If you publish the same data through a feed or an API, tell us and we'll switch to it.
Contact
[email protected]. We reply to every message about DevoteBot, and we stop at once if you ask.