Recap: Towards Trustworthy Harnesses — our first High Assurance AI Programme workshop

On Monday 27 July, we hosted Towards Trustworthy Harnesses: An Engineering Discipline for Agentic AI — our first workshop as part of our High Assurance AI Programme. Held at Jesus College in Cambridge, the workshop brought together CommonAI members, academics and other domain experts to discuss how we can develop harnesses to make agentic AI safer and more trustworthy.

Throughout the day, there were a number of talks, with speakers bringing forward their perspectives and ideas of what these harnesses could look like, alongside discussions of specific pain points in the deployment of agentic AI, and the need for collaboration to overcome these challenges. A recurring theme was that a harness is an engineered system in its own right — deliberately placing checks, gates, and points of human oversight around a model, rather than simply wrapping it.

Many attendees discussed the increasing importance of formal verification as a safeguard for agentic AI. One speaker focused on the huge increase in the prevalence of AI-generated code in recent years. Due to the massively increased rate of production, traditional software-assurance methods are not able to scale. AI-generated code must be treated as inherently untrustworthy, with risks including inaccurate training data as well as data poisoning. One of the largest issues is the specification gap: if you cannot accurately and precisely specify what your code should do, the best case is ambiguity — you get something that isn’t what you wanted. The worst case is that an AI agent works around the specification, exploits it, knows that it’s wrong, and lies about it, for example, a model that claims to have run a formal verification step when it hasn’t.

This speaker’s perspective was that we should spend much more time specifying what a system should do than on how to build it. They argued that we should build systems in such a way that lets us reason about them afterwards by using formal methods (i.e., mathematical techniques used to specify and prove how a system behaves) which give concrete guarantees. Two complementary slogans were presented, capturing a theme that was present in many of the talks:

Another speaker agreed that AI-driven formalisation was the path to assurance, and they brought forward a concrete example of how AI can be used to accelerate formal methods. They argued that meta-optimisation is the key to enabling AI to get real leverage over formal methods. Meta-optimisation, sometimes called self-improving workflows, involves a model orchestrating its own optimisation process. It borrows its guiding logic from reinforcement learning: the model is treated as an agent, but with a very complex action space and a reward that’s often implicit. The trade-off observed when using this technique is that more autonomy gives better performance, but potentially weaker guarantees. Navigating this trade-off is a core area of research.

An alternative approach presented was domain-specialised, local, small language models. Using the Qwen family of models as an example, a much smaller 7B-parameter model reaches ~86% of the accuracy of the 72B model; however, narrowing the model’s domain can close or even reverse this gap on specialised tasks, showing the promise of smaller, task-focused models. The advantages of such models include:

• Improved inference economics: for example, through lower energy usage
• Easier to host: important for meeting data sovereignty requirements
• Systemic diversity: one of the greatest risks of many sectors relying on the same handful of frontier models is that they inherit the same systemic failures, creating an ecosystem monoculture that could lead to catastrophic consequences
• Easier to interpret: with fewer parameters and a narrower set of concepts that need to be represented, these models are more tractable to interpret

Interpretability was discussed in more depth during a presentation on concept-based learning. This approach is inspired by how humans tend to explain things using high-level concepts, so the model is trained to reason through concepts that are meaningful to people. For example, a model trained to detect photos of cats will reason through concepts like whiskers, fur, and a tail rather than through pixels alone. This can be done by introducing a concept-encoding layer before the model’s label predictor, which allows you to gain an insight into how the label predictor works and enables you to see how it goes wrong, making it much easier to audit, debug and improve. Ultimately, this can achieve high accuracy while substantially reducing the traditional trade-off with interpretability. Moreover, as concepts are explicit, a domain expert can intervene at inference time (i.e., while the model is making a live prediction, rather than during training) for high-stakes or low-confidence decisions. The model’s explanation, shown through the identified concepts, creates an interface for causal debugging. Together, this control and transparency build trust in the system.

There was also a discussion of how concept-based models can benefit from formal verification. Interpretability alone isn’t enough, so the model’s reasoning can be turned into formally checkable objects to create a more trustworthy system which relies on the following pillars:

1. Explainability: beyond accuracy, explanations should be offered in human concepts
2. Interaction: keep domain experts in the loop, able to intervene and correct
3. Correctness: using formal methods to ensure outputs are right

The talks concluded by drawing together the day’s central thread: trust is a precondition, something that has to be embedded from the start, not added later. Achieving trustworthy AI requires a wider range of expertise than any single organisation holds. A key opportunity exists in closing the loop between industry requirements and academia, connecting UK startups working on different parts of the stack, and thinking whole-stack, i.e., considering how the entire stack from hardware to software contributes to the trustworthiness of a system. No single person, group or company can solve this alone, which is why CommonAI exists: to bring people together, each contributing their expertise to help solve some of the biggest challenges in AI.