An empirical research group at the Oxford Internet Institute. We study LLM evaluation, safety, reasoning, and the agentic systems built from them.
Standing direction
Reasoning systems that scientists, clinicians, and the public can trust — measured by what they actually do, not what they claim.
Each direction is a standing condition the lab is working toward. Together they describe the field we want AI evaluation, safety, and reasoning to become.
We work on the science of LLM evaluation: what benchmarks measure, where they mislead, and how to build ones that hold up.
Toward
A field where every benchmark publishes its construct validity, and where evaluation is treated as an experimental science.
We work on bias, toxicity, and agentic misalignment, and on the technical and governance tools that address them.
Toward
A practice of measuring real harms — bias, toxicity, agentic misalignment — at the neuron and the deployment, before they reach the public.
Agentic systems for scientific work. We focus on keeping them reliable, transparent, and grounded in the domain.
Toward
Scientific agents that synthesise knowledge reliably enough that a researcher can act on them — and transparently enough that they can audit them.
Empirical studies of how people use AI in high-stakes settings: healthcare, law, and policy.
Toward
Decisions made with AI in healthcare, law, and policy that are studied empirically — not assumed safe because the model is impressive.
“There is hope in honest error; none in the icy perfection of the mere stylist.”— J. D. Sedding, after the Glasgow Four
Four directions · One tableTen papers from the past eighteen months. Nature Medicine, ICML (with a Spotlight), ICLR, NeurIPS, EMNLP, Information Fusion. Each plate is one finished cycle — built, peer-reviewed, published in the open.
A benchmark that tells real navigation apart from stochastic search when agents work over document collections.
Ł Borchmann, J Van Landeghem, M Turski, S Padarha, RO Kearns, A Mahdi, et al.
LLM self-explanations are usually dismissed as unreliable. Measured the right way, they predict model behavior.
H Mayne, JS Kang, D Gould, K Ramchandran, A Mahdi, NY Siegel
A benchmark for whether web agents can be socially engineered into abandoning the user's task. Today's agents fall for it.
K Korgul, Y Yang, A Drohomirecki, P Błaszczyk, W Howard, L Aichberger, C Russell, P H S Torr, A Mahdi, A Bibi
A preregistered randomized study in Nature Medicine on how reliably LLMs serve as medical assistants for the general public.
AM Bean, RE Payne, G Parsons, HR Kirk, J Ciro, R Mosquera-Gómez, S Hincapié, AS Ekanayaka, L Tarassenko, L Rocher, A Mahdi
A construct-validity audit of widely-used LLM benchmarks: what they claim to measure versus what they capture.
AM Bean, RO Kearns, A Romanou, FS Hafner, H Mayne, J Batzner, et al.
How LLM judges degrade across languages, modalities, and domains, and where the failure modes sit.
S Padarha, E Semenova, B Vidgen, A Mahdi, S A Hale
Direct Preference Optimization reduces toxicity. We trace where it acts, neuron by neuron.
Y Yang, F Sondej, H Mayne, A Lee, A Mahdi
Ask an LLM "what would change your answer?" and it looks like introspection. It is often confabulation.
H Mayne, RO Kearns, Y Yang, AM Bean, E Delaney, C Russell, A Mahdi
A survey of multimodal ML in clinical practice, from data-fusion strategies through to deployment.
F Krones, U Marikkar, G Parsons, A Szmul, A Mahdi
A benchmark that obfuscates orthography to strip memorised knowledge out of reasoning problems, showing how much "reasoning" was recall.
J Khouja, K Korgul, S Hellsten, L Yang, V Neacsu, H Mayne, RO Kearns, A Bean, A Mahdi
A Principal Investigator, ten DPhil students, two MSc students, a visiting fellow, and a research affiliate. Each is introduced by what they are currently working on — the focus is the headline; the role is the caption.
Master of the Pavilion · Principal Investigator
Oxford Internet Institute, University of Oxford
Adam leads OxRML. The group studies how language models reason, how people work with them, and how agentic systems behave on real scientific and decision-making tasks. He won the Oxford Teaching Excellence Award in 2025.
Currently practising
Coaching the lab's research cycles across evaluation, safety, agentic AI, and human–AI interaction — measuring what we ship before we ship it.
01 / 14DPhil Student
Multimodal evaluation across imaging and clinical text.
02 / 14DPhil Student
LLMs applied to food-security data and policy questions.
03 / 14DPhil Student
LLM evaluations that capture how people actually use models.
04 / 14DPhil Student
Post-training for LLM and agentic alignment, at the neuron level.
05 / 14DPhil Student
Interpretability and safety-relevant LLM evaluations.
06 / 14DPhil Student
Knowledge-graph methods for metascience and research synthesis.
07 / 14DPhil Student
Healthcare AI evaluation grounded in clinical workflow.
08 / 14DPhil Student
Agentic-AI safety and web-agent persuasion attacks.
09 / 14DPhil Student
The science of evals — measuring reasoning honestly.
10 / 14DPhil Student
Agentic systems for science, with safety and eval rigour.
11 / 14MSc Student
Studies of human–LLM interaction and LLM evaluation.
12 / 14MSc Student
LLM evaluation and reasoning benchmarks.
13 / 14Visiting Policy Fellow
LLMs applied to financial time series, at the policy boundary.
14 / 14Research Affiliate
Public-health AI and LLM evaluations grounded in epidemiology.
The reading-room ledger — papers accepted, conferences attended, honours noted. The most recent entry takes the rose tag. The ledger is kept in the open so collaborators always know what is current.
Three OxRML papers accepted at ICML 2026 — including a Spotlight
OxRML presenting at ICLR 2026
New paper in Nature Medicine on LLMs as medical assistants
Ryan Othniel Kearns wins MSc Thesis Prize
OxRML at NeurIPS 2025
OxRML at EMNLP 2025
Prof. Adam Mahdi wins Oxford Teaching Excellence Award 2025
New review paper in Information Fusion
Winners of the 2024 PhysioNet Challenge
9 entries · ledger held in the open
We work with foundations, governments, hyperscalers, and global corporates who want AI evaluation, safety, and reasoning treated with the same care as the systems they ship. Three formats — three cadences — one table.
On-site sessions for product and ML teams on evaluation, safety, and agent reliability.
Half-day to multi-week formats. For teams shipping LLM products in healthcare, finance, retail, and government.
Cadence
Half-day to multi-week. On-site in your office or in Oxford; bespoke to the team and the question.
We work with engineering partners to turn lab work into tools other teams can run.
Evaluation harnesses, safety dashboards, agentic-research platforms. We build them with partners we trust, carrying the research methods through to the code.
Cadence
Quarterly cycles with a partner studio. Joint roadmaps, shared evals, shipped tooling.
Applied research collaborations with foundations, governments, and large companies.
Multi-year programmes: shared roadmaps, sponsored DPhil studentships, named labs.
Cadence
Multi-year. Named programmes, dedicated DPhil studentships, shared scientific direction.
The invitation
If your team is shipping AI into a high-stakes setting — healthcare, finance, public infrastructure — there is a chair at the table.
Write directly. A brief note about the problem you are working on, the stakes, and the question you want answered. We'll reply within the week.
Plate VII · Honour Roll
Universities, journals, and conferences where the lab's cycles have been hosted, peer-reviewed, and published.
Plate VIII · The Visitors' Book
New papers, open positions, partnership opportunities, and what we have been reading.