An empirical research group at the Oxford Internet Institute. We study LLM evaluation, safety, reasoning, and the agentic systems built from them.
Posture
We are interested in the measurement of intelligence the way a metrologist is interested in the kilogram: patiently, in the open, at the level of mechanism.
A single research practice composed of four interlocking studies, each one a partition that defines a region of the lab. Together they describe the field we want AI evaluation, safety, agentic AI, and human–AI interaction to become.
No. 01 · Instrument
Evaluation
We work on the science of LLM evaluation: what benchmarks measure, where they mislead, and how to build ones that hold up.
What this partition defines
The science of how a benchmark is built, what it claims to measure, and what it actually captures.
No. 02 · Instrument
Safety
We work on bias, toxicity, and agentic misalignment, and on the technical and governance tools that address them.
What this partition defines
A field that measures real harms (bias, toxicity, agentic misalignment) at the neuron and the deployment.
No. 03 · Instrument
Agentic
Agentic systems for scientific work. We focus on keeping them reliable, transparent, and grounded in the domain.
What this partition defines
Agentic systems that synthesise knowledge with enough rigour for a working scientist to act on them.
No. 04 · Instrument
Human-AI
Empirical studies of how people use AI in high-stakes settings: healthcare, law, and policy.
What this partition defines
High-stakes decisions made with AI, studied empirically rather than assumed safe because the model is impressive.
Each instrument is, in the Bouroullec sense, a micro-architecture: modest by itself, complete in concert with the others.
A reservoir of deliberate experiments. Each paper began as a question and resolved into an instrument the field can use. The most recent sits at the top of the plate.
Pl. 03.01 · Featured drawing
NOWICML (Spotlight) · May 2026
A benchmark that distinguishes document-collection navigation from stochastic search in multimodal agents.
Ł Borchmann, J Van Landeghem, M Turski, S Padarha, RO Kearns, A Mahdi, et al.
A positive result on LLM self-explanation faithfulness, usable as a behavioural predictor when measured carefully.
H Mayne, JS Kang, D Gould, K Ramchandran, A Mahdi, NY Siegel
ICML
May 2026
A web-agent benchmark for resistance to social engineering. Frontier agents fall for the persuasion attack.
K Korgul, Y Yang, A Drohomirecki, P Błaszczyk, W Howard, L Aichberger, C Russell, P H S Torr, A Mahdi, A Bibi
ICML
May 2025
A preregistered, randomised Nature Medicine study on the public’s use of LLMs as medical assistants.
AM Bean, RE Payne, G Parsons, HR Kirk, J Ciro, R Mosquera-Gómez, S Hincapié, AS Ekanayaka, L Tarassenko, L Rocher, A Mahdi
Nature Medicine
February 2026
A construct-validity audit of the benchmarks the field treats as ground truth.
AM Bean, RO Kearns, A Romanou, FS Hafner, H Mayne, J Batzner, et al.
NeurIPS Datasets and Benchmarks
November 2025
How LLM-as-a-judge degrades, sometimes catastrophically, across language, modality, and domain.
S Padarha, E Semenova, B Vidgen, A Mahdi, S A Hale
NeurIPS LLM Lifecycle Workshop
November 2025
A neuron-level account of how Direct Preference Optimization reduces toxicity.
Y Yang, F Sondej, H Mayne, A Lee, A Mahdi
EMNLP
November 2025
Counterfactual self-explanations are fluent but systematically mis-locate the model's decision boundary.
H Mayne, RO Kearns, Y Yang, AM Bean, E Delaney, C Russell, A Mahdi
EMNLP
September 2025
A survey mapping the multimodal healthcare ML landscape: fusion strategies, deployment realities.
F Krones, U Marikkar, G Parsons, A Szmul, A Mahdi
Information Fusion
February 2025
A reasoning benchmark that disentangles inference from memorisation by templatised orthographic obfuscation.
J Khouja, K Korgul, S Hellsten, L Yang, V Neacsu, H Mayne, RO Kearns, A Bean, A Mahdi
ICLR
April 2026
10 drawings · catalogued in order of recency
A field of individual practitioners. Each is introduced by what they are currently practising and what they are working toward; roles sit underneath, as a matter of record.
Tile 00 · Principal investigator
Oxford Internet Institute, University of Oxford
Adam leads OxRML. The group studies how language models reason, how people work with them, and how agentic systems behave on real scientific and decision-making tasks. He won the Oxford Teaching Excellence Award in 2025.
Currently practising
Coaching the lab’s research cycles across evaluation, safety, agentic AI, and human–AI interaction.
Working toward
A field where AI evaluation is treated as a science, with the same care as the systems it studies.
Tile 01
DPhil Student
Currently
Multimodal evaluation across imaging and clinical text.
Toward
Digital-health systems measured before they are deployed.
Tile 02
DPhil Student
Currently
LLMs applied to food-security data and policy questions.
Toward
Decision tools for food systems that hold up under audit.
Tile 03
DPhil Student
Currently
LLM evals that capture how people actually use models.
Toward
A standard for evaluating LLMs in real human contexts of use.
Tile 04
DPhil Student
Currently
Post-training for LLM and agentic alignment, at the neuron level.
Toward
Alignment interventions whose mechanism we can explain.
Tile 05
DPhil Student
Currently
Interpretability and safety-relevant LLM evaluations.
Toward
Interpretability methods that practitioners can trust under shift.
Tile 06
DPhil Student
Currently
Knowledge graphs for metascience and research synthesis.
Toward
Synthesis tools scientists treat as collaborators, not search engines.
Tile 07
DPhil Student
Currently
Healthcare AI evaluation grounded in clinical workflow.
Toward
Digital-health products that earn the trust of clinicians and patients.
Tile 08
DPhil Student
Currently
Agentic-AI safety, including web-agent persuasion attacks.
Toward
Web agents that resist social engineering as a default behaviour.
Tile 09
DPhil Student
Currently
The science of evals: measuring reasoning honestly.
Toward
A field where every benchmark publishes its construct validity.
Tile 10
DPhil Student
Currently
Agentic systems for science, with safety and eval rigour.
Toward
Scientific agents auditable enough to act on in real research.
Tile 11
MSc Student
Currently
Studies of human–LLM interaction and LLM evaluation.
Toward
Interaction patterns that improve, rather than substitute, human judgement.
Tile 12
MSc Student
Currently
LLM evaluation and reasoning benchmarks.
Toward
Reasoning evals that separate genuine inference from recall.
Tile 13
Visiting Policy Fellow
Currently
LLMs applied to financial time series, at the policy boundary.
Toward
Honest characterisation of LLM utility in high-stakes financial settings.
Tile 14
Research Affiliate
Currently
Public-health AI and LLM evaluations grounded in epidemiology.
Toward
Public-health AI evaluated like a health intervention.
What we saw at the board this season: papers accepted, conferences reached, awards noted. Kept in order, kept in the open.
Three OxRML papers accepted at ICML 2026 — including a Spotlight
OxRML presenting at ICLR 2026
New paper in Nature Medicine on LLMs as medical assistants
Ryan Othniel Kearns wins MSc Thesis Prize
OxRML at NeurIPS 2025
OxRML at EMNLP 2025
Prof. Adam Mahdi wins Oxford Teaching Excellence Award 2025
New review paper in Information Fusion
Winners of the 2024 PhysioNet Challenge
9 entries · kept in order of event
Three formats for partnering with the lab. Each one a real commitment: we are picky about partners and slow to ramp, because shared roadmaps need shared standards.
Format · 01
On-site sessions for product and ML teams on evaluation, safety, and agent reliability.
Half-day to multi-week formats. For teams shipping LLM products in healthcare, finance, retail, and government.
Format · 02
We work with engineering partners to turn lab work into tools other teams can run.
Evaluation harnesses, safety dashboards, agentic-research platforms. We build them with partners we trust, carrying the research methods through to the code.
Format · 03
Applied research collaborations with foundations, governments, and large companies.
Multi-year programmes: shared roadmaps, sponsored DPhil studentships, named labs.
We work with foundations, governments, and corporates who want AI evaluation, safety, and reasoning treated with the same care as the systems they ship.
We reply to every serious enquiry
Universities, journals, and conferences where the lab’s work has been hosted, peer-reviewed, and published.
Pl. 08 · The lab newsletter
New papers, open positions, partnership opportunities, and what we have been reading.
