Reasoning with
Machines Lab
@ University of Oxford

A benchmark that tells real navigation apart from stochastic search when agents work over document collections.

LLM self-explanations are usually dismissed as unreliable. Measured the right way, they predict model behavior.

A benchmark for whether web agents can be socially engineered into abandoning the user's task. Today's agents fall for it.

A preregistered randomized study in Nature Medicine on how reliably LLMs serve as medical assistants for the general public.

A construct-validity audit of widely-used LLM benchmarks: what they claim to measure versus what they capture.

How LLM judges degrade across languages, modalities, and domains, and where the failure modes sit.

Direct Preference Optimization reduces toxicity. We trace where it acts, neuron by neuron.

Ask an LLM "what would change your answer?" and it looks like introspection. It is often confabulation.

A survey of multimodal ML in clinical practice, from data-fusion strategies through to deployment.

A benchmark that obfuscates orthography to strip memorised knowledge out of reasoning problems, showing how much "reasoning" was recall.