Donation Whistle

By Will Thong, Tue 28 November 2023, in category Programming


My Donation Whistle app

Politicians should only use the power they’re given by the people in the interests of the people. If donors influence politicians when they vote on laws or nominate peers to the House of Lords, then they are doing their jobs badly.

Thankfully, the Labour Government’s 2000 reforms forced political parties to disclose their significant donors to the Electoral Commission. The Commission publishes these donation records online. This gives journalists the chance to investigate which donors might be paying politicians for influence, rather than supporting a cause they believe in as most do.

But even with these sensible measures in place, the system is let down by poor implementation. The most obvious problem: the Electoral Commission simply publishes whatever donor name is reported by the political party. If parties are inconsistent about a donor’s name, either internally or amongst themselves, that donor will be listed under different names. For instance, the Conservative Party and Reform UK donor Christopher Harborne is listed under seven.1 The Electoral Commission’s system hides how much a donor has given, making it harder for the public to learn when they have undue influence.2

A related problem: it is hard to understand the data at a glance. You could download the donation records to build Excel (or similar) charts whenever the data change, but that’s a lot of work. And the Electoral Commission publishes some graphs on this page, but they’re hard to get to and uninformative. They’re hard to get to because they’re not directly linked from the donation register page or the homepage (it seems to take five clicks). They’re also uninformative. By default they include Public Funds (government cash distributed to opposition parties), which is very confusing to anyone who doesn’t know what short money is. They only split out donors by category (individuals, companies, trade unions etc) - you cannot see any donor’s total giving. They don’t compare trends over time - there is no way to see which year parties raised the most since records began.3

This causes real problems. For example, journalists, even at respected institutions like the Times, understand the data so poorly that they just repeat whatever they’re told. In September, the Times described John Caudwell, the Phones 4U founder, as “The biggest donor to the Conservative Party before the last election”. This is a completely empty boast. If you count ‘before’ as one month, he’s the fourth largest, or if you count it as three months, he’s the eighth largest.4 If the Electoral Commission is meant to be providing a window into our democracy, it’s so fogged up so as to be useless.

One solution: Westminster Accounts

Others have improved public understanding of political finance through solid data journalism. Tortoise and Sky News should be commended for Westminster Accounts, a project which combined the Electoral Commission’s donations database with the House of Commons’ Register of Financial Interests and its Register of All-Party Parliamentary Groups (APPGs).

This wide scope allows the Westminster Accounts to tell a compelling, well-visualised story about individual lawmakers. However, it also creates three problems.

First, it only covers the most recent Parliament. I guess this is because converting the terribly-formatted House of Commons records into usable data is far too much work. Covering 25 years rather than four would be impractical.

Second, the reporting thresholds are higher for donations to MPs (£1,500) than to central political parties (£7,500). In fairness, the Westminster Accounts do not directly compare or aggregate the two, so this is a stylistic choice. However, in my view it is potentially confusing.

Third, some donations are made both to parties and to individual members (eg where a donor donates to a party to cover an MP’s member of staff’s salary). This leads the Westminster Accounts to list some duplicate donations. This can’t be solved under the current rules, because they don’t require political parties to share the purpose of donations. As far as I can tell, it is impossible to accurately combine the MP and APPG with the Electoral Commission data.

Another challenge with the Westminster Accounts is open access. To know how much money each donor is giving, we need to connect donations listed under different donor names (as discussed above). This work has to be done manually (and with clever AI machine learning techniques” in the Westminster Accounts’ case). But, while the Westminster Accounts’ authors share the way they have interpreted the MP and APPG Registers’ data in a Google Sheets document, they have not published the way they have combined the names of political party donors. As the authors rightly point out at the end of each MP’s Westminster Accounts page:

the amount of money flowing into the political parties is on a completely different scale to the sums MPs and APPGs are collecting. In particular, the Labour and Conservative parties have each – alone – taken in two and four times more money than all MPs and all APPGs put together.

This fact is either not reflected in their bubble chart, or the chart is not to scale, as the MPs’ purple bubble is larger than Labour’s. But it is true that party donations dwarf parliamentary donations. If anything, it’s much more important that we know and can iterate on this aggregation work.

My solution: Donation Whistle

I have made Donation Whistle, a webapp based on the Python web microframework Flask. It is completely open source, so anyone can audit the source code on my GitHub. Here’s a video tour of its features:

Automatic import

The app starts by requesting all of the Electoral Commission’s donation records from its website. It then throws most of them away. Why?

First, it removes donations to local parties like Conservative Associations or Constituency Labour Parties. The reporting limits for these are lower than for central parties (£500 rather than £7,500), so counting them as part of of how much parties receive would be unfair. This brings the number of donations from around 80,000 to 27,000 (at the time of writing).

Next, it removes duplicate records which exist because parties double-report donations made in the run-up to an election.5 There are only about 1000 of these.

Finally, it removes donations which parties forfeited (because they couldn’t work out who the donor was) or returned (eg because the donor turned out not to be eligible to donate). This removes a further 200 or so donations.

Having cleaned the data, Donation Whistle puts the records into a SQLite database. It uses the date the donation was received by the political party or, if that’s missing, the date that the donation was accepted by the political party. This choice explains some minor differences between Donation Whistle’s figures and the Westminster Accounts’.6

Using Donation Whistle

We then see the list of donations on Donation Whistle’s homepage. The most important information about each donation is presented in a table. The table can be searched, filtered and sorted by donor name, donation date and amount. The data can be exported in CSV format.

By default, Donation Whistle filters out two sorts of payments received by political parties. It excludes short money (public cash distributed to opposition parties) because it is an legally obligatory payment as opposed to a donation and because it isn’t paid to the party in government, so including it would be an unfair comparison. It also removes donations which are returned to the donor, as these do not benefit the party.

Clicking ‘Recipients’, you can see an interactive, birds-eye view of how political donations have changed since records began in 2001. Neither the Electoral Commission nor the Westminster Accounts show this. You can zoom in to see smaller timescales, you can add parties to or remove them from the graph and you can downloaded any view you have generated as an image.

Donation Whistle allows admin users to add aliases to group together donations which have been recorded under different donor names. You can also export your alias settings or import them from other users. This way, you can always audit the choices made by anybody running a copy of Donation Whistle.

Whether you choose to make your own or import somebody else’s sample aliases (such as the ones included in the GitHub repository, Donation Whistle needs these aliases to generate the graphs on the ‘Donors’ page (biggest donors to all parties) and on each individual ‘Recipient’ page (biggest donors to each party). As with the Recipients graph, these graphs are also interactive, so you can zoom and export however you like.

Under the hood

I’ll now explain some technical choices I made when building Donation Whistle.

I chose the SQLite database because, at 26,000 records in its largest table, the dataset is relatively small. It is likely to grow only at a roughly linear rate. I therefore valued the simplicity and small resource footprint of SQLite.

Similarly, I chose flask_caching to provide a lightweight filesystem cache. Visualisation and API calls are much faster thanks to the cache. I didn’t feel I needed anything more complex than the filesystem cache, though, because (as mentioned above) the database size is so small. A filesystem cache which is mostly being read rather than written to allows for data persistence, therefore speedy access, but without the constant memory overhead of a more fully-featured cache. That said, I ended up needing a Redis container anyway for background tasks, so I may switch to a Redis cache in a later version.

I needed to build a very basic background task system. All I needed was to offload the data import process so the user would not be stuck staring at a loading bar for the five minutes it might take to import all the data. I chose RQ (Redis Queue), again for its simplicity.

The core technologies I built off were Flask (therefore Python), HTML, JavaScript and, for deployment, Docker. Being less familiar with JavaScript than I am with Python, the most challenging part of the project was ensuring that the filters selected by the user in the Flask form on the homepage are synced to the UI so as to avoid user confusion.

Next steps

Donation Whistle is not finished! As an open source project, of course I’d be delighted if anyone wanted to contribute code. In particular, I’d love to add these features:

  1. In case you are interested: Christopher Charles Sherriff Harborne, Mr Christopher Harborne, Mr Christopher C S Harborne, Christopher C S Harborne, Mr Christopher C Harborne, Christopher C Harbone and Christopher C Harborne 

  2. This is probably not the Commission’s fault. Better legislation would oblige it to do this work. 

  3. 2019, it turns out. 

  4. It would be crazy to set the cutoff this late, by the way - the £1.4m JCB gave to the Conservatives was obviously a part of the election campaign, even if the donation record on CCHQ’s systems says it arrived on 16 December 2019. 

  5. During the election period before the poll date Political Parties and Non-party campaigners are required to submit regular pre-poll donation and loan reports for all items received (even if they are not accepted). These same items are required to be reported again in either the Political Party’s normal quarterly reporting or in the Non-party campaigner’s post-poll report.” (Electoral Commission donation records

  6. The Westminster Accounts use the date the donation was reported to the Electoral Commission rather than the date the donation was received by the party. I don’t think this is the right choice. For instance, take JCB’s £1.4m to the Conservative Party, given on 16 December 2019 but reported on 30 January 2020. It seems unlikely this donation went into the Tories’ operating costs for 2020 onwards as opposed to funding its December 2019 General Election campaign.