Benchmarked against humans
Whitehall’s Consult: an AI tool measured against the humans it assists
UK Government · Consult · Public sector · United Kingdom
The UK government’s in-house AI tool sorted more than 50,000 consultation responses into themes in about two hours for £240, then had experts check the result. Its agreement with those experts beat the experts’ agreement with each other. It is still in beta, and the headline savings remain projections.
50,000+
responses themed in ~2 hours, for £240
0.79–0.82
F1 score against two expert groups
0.74
F1 score between the two expert groups
AI proposes the themes; experts check, refine and decide.
The record
- In its first live use, Consult themed more than 2,000 responses to a Scottish Government consultation on non-surgical cosmetic procedures; officials also reviewed every response, and the tool scored an F1 of 0.76 against their judgement. (DSIT, May 2025)
- For the Independent Water Commission’s review, Consult sorted more than 50,000 responses into themes in around two hours for £240, followed by 22 hours of expert checking; it agreed with one or both expert groups almost 83% of the time, while the two groups agreed with each other 55% of the time. (DSIT, Oct 2025)
- As of September 2026 the tool remained in beta, tested on a small number of live consultations with wider rollout planned for 2027; the government projects savings of up to 75,000 days of analysis, about £20m a year, across roughly 500 annual consultations. (i.AI, Sep 2026; DSIT, May 2025)
A tedious, high-volume, checkable task
The UK government runs around 500 public consultations a year, and analysing free-text responses is slow manual work: officials read each one and code it to themes. The Incubator for Artificial Intelligence (i.AI), a government unit, built Consult as part of its “Humphrey” suite of civil-service tools to propose themes and assign responses to them, with officials reviewing the output through a dashboard (DSIT, May 2025).
It is a well-chosen problem. The output is an input to human policy judgement, not a decision; errors are visible and correctable; and there is a natural benchmark, because the same task has always been done by people. For its first live use, on a Scottish Government consultation of more than 2,000 responses, officials reviewed every response by hand alongside the tool, so that the two could be compared directly.
Measured against human disagreement
The evaluation design is the instructive part. Rather than assert accuracy, i.AI scored Consult against expert reviewers using a standard measure, the F1 score: 0.76 on the Scottish consultation, where reviewers left the tool’s themes unchanged for 60% of responses and took a median of 23 seconds to check each one (i.AI, Sep 2026). Differences had a negligible effect on how themes were ranked overall (DSIT, May 2025).
On the Independent Water Commission’s review, Consult themed more than 50,000 responses in about two hours for £240, followed by 22 hours of expert checking (DSIT, Oct 2025). Its F1 scores against two expert groups were 0.79 and 0.82; between the two human groups the score was 0.74. On this measure the tool was closer to each team of experts than the teams were to each other.
Beta, not victory
The government has been more cautious in deployment than in its press releases. As of September 2026 Consult was still in beta, being tested on a small number of live consultations, with wider rollout planned for 2027 and new requests on a waiting list (i.AI, Sep 2026). The widely quoted saving — 75,000 days of analysis and about £20m a year — is a projection across all consultations, not a measured result (DSIT, May 2025).
Public consent is also unfinished business. In deliberative sessions with 144 adults run by Nesta with i.AI, 85% saw human oversight as a necessary safeguard, and more than three-quarters remained concerned about model manipulation even after hearing the planned mitigations; some were uneasy about relying on a model from a US company (Nesta, Aug 2025). For a democratic process, accuracy is necessary for legitimacy, not sufficient.
The lesson
It worked because the government chose a task with a human benchmark, published its accuracy against that benchmark — including how much humans disagree with one another — and kept experts reviewing the output before it informed policy. Measure AI against the real human baseline, not against perfection.
How we’re reading this
The evaluations are run and published by the government itself, on a handful of consultations; we have not seen independent replication. F1 agreement measures consistency with expert coding, not whether the resulting policy was better. The £20m-a-year saving is a projection, not a realised figure, and the tool is still in beta. Concerns about bias, manipulation and dependence on commercial models remain, as Nesta’s public dialogue found.
Sources
- 01Government-built Humphrey AI tool reviews responses to consultation for first time in bid to save millions
GOV.UK (DSIT) · May 14, 2025
- 02Ground-breaking use of AI saves taxpayers’ money and delivers greater government efficiency
GOV.UK (DSIT) · Oct 16, 2025
- 03Consult: processing consultation responses to support policymaking
i.AI (UK Government Web Archive) · Sep 2, 2026
- 04
Compiled from public filings, court records, company statements and reputable press. Figures are attributed to their sources; allegations are labeled as such. Not legal or investment advice.