@ Oxford University
Led by Prof. Adam Mahdi, our lab advances the science of AI evaluation, benchmarking, safety and security. Through rigorous empirical research, we study how LLMs and agentic systems reason, interact with humans and drive scientific discovery.
Mission
We advance the science of AI evaluation, safety, and reasoning through rigorous empirical research, in the open and over the long term.
Four areas of focused work: AI evaluation, AI safety and security, agentic AI for science, and human–AI interaction.
Theme 01
Evaluation
We develop the science of LLM evaluation, setting the standard for rigorous assessment and identifying hidden risks before they matter.
Focus
How benchmarks are built, what they claim to measure, and what they actually capture.
Theme 02
Safety
From bias and toxicity to agentic misalignment, we study the full spectrum of AI risk and develop the technical and governance tools to address it.
Focus
The full spectrum of AI risk — bias, toxicity, agentic misalignment — studied at the neuron and at deployment.
Theme 03
Agentic
We build agentic systems that automate scientific knowledge synthesis and discovery, with a focus on agents that are reliable, transparent and domain-grounded.
Focus
Agentic systems that synthesise scientific knowledge reliably enough to act on.
Theme 04
Human-AI
We run large-scale empirical studies on how people use AI for high stakes decisions, from healthcare and law to policy and beyond.
Focus
How people use AI in high-stakes decisions — healthcare, law, policy — studied empirically.
Each theme runs on a multi-year horizon and feeds the others.
Recent peer-reviewed work across evaluation, safety, agentic AI, and human–AI interaction. The most recent paper sits at the top.
Featured paper
NOWICML (Spotlight) · May 2026
A benchmark that distinguishes document-collection navigation from stochastic search in multimodal agents.
Ł Borchmann, J Van Landeghem, M Turski, S Padarha, RO Kearns, A Mahdi, et al.
A positive result on LLM self-explanation faithfulness, usable as a behavioural predictor when the measurement is set up right.
H Mayne, JS Kang, D Gould, K Ramchandran, A Mahdi, NY Siegel
ICML
May 2026
A web-agent benchmark for resistance to social engineering. Frontier agents fall for the persuasion attack.
K Korgul, Y Yang, A Drohomirecki, P Błaszczyk, W Howard, L Aichberger, C Russell, P H S Torr, A Mahdi, A Bibi
ICML
May 2025
A preregistered, randomised Nature Medicine study on the public’s use of LLMs as medical assistants.
AM Bean, RE Payne, G Parsons, HR Kirk, J Ciro, R Mosquera-Gómez, S Hincapié, AS Ekanayaka, L Tarassenko, L Rocher, A Mahdi
Nature Medicine
February 2026
A construct-validity audit of the benchmarks the field treats as ground truth.
AM Bean, RO Kearns, A Romanou, FS Hafner, H Mayne, J Batzner, et al.
NeurIPS Datasets and Benchmarks
November 2025
How LLM-as-a-judge degrades, sometimes catastrophically, across language, modality, and domain.
S Padarha, E Semenova, B Vidgen, A Mahdi, S A Hale
NeurIPS LLM Lifecycle Workshop
November 2025
A neuron-level account of how Direct Preference Optimization reduces toxicity.
Y Yang, F Sondej, H Mayne, A Lee, A Mahdi
EMNLP
November 2025
Counterfactual self-explanations are fluent but systematically mis-locate the model's decision boundary.
H Mayne, RO Kearns, Y Yang, AM Bean, E Delaney, C Russell, A Mahdi
EMNLP
September 2025
A survey mapping the multimodal healthcare ML landscape: fusion strategies, deployment realities.
F Krones, U Marikkar, G Parsons, A Szmul, A Mahdi
Information Fusion
February 2025
A reasoning benchmark that disentangles inference from memorisation by templatised orthographic obfuscation.
J Khouja, K Korgul, S Hellsten, L Yang, V Neacsu, H Mayne, RO Kearns, A Bean, A Mahdi
ICLR
April 2026
10 papers · most recent first
DPhil and MSc students, fellows, and affiliates. Each is introduced by what they are currently working on and what they are working toward.
Principal Investigator
Oxford Internet Institute, University of Oxford
Adam leads OxRML. The group studies how language models reason, how people work with them, and how agentic systems behave on real scientific and decision-making tasks.
Currently
Leading the lab’s research across evaluation, safety, agentic AI, and human–AI interaction.
Working toward
Advancing AI evaluation as a rigorous science, with the same care as the systems it studies.
No. 01
DPhil Student
Currently
Multimodal evaluation across imaging and clinical text.
Toward
Digital-health systems measured before they are deployed.
No. 02
DPhil Student
Currently
LLMs applied to food-security data and policy questions.
Toward
Decision tools for food systems that hold up under audit.
No. 03
DPhil Student
Currently
LLM evals that capture how people use models in practice.
Toward
A standard for evaluating LLMs in real human contexts of use.
No. 04
DPhil Student
Currently
Post-training for LLM and agentic alignment, at the neuron level.
Toward
Alignment interventions whose mechanism we can explain.
No. 05
DPhil Student
Currently
Interpretability and safety-relevant LLM evaluations.
Toward
Interpretability methods that practitioners can trust under shift.
No. 06
DPhil Student
Currently
Knowledge graphs for metascience and research synthesis.
Toward
Synthesis tools scientists treat as collaborators, not search engines.
No. 07
DPhil Student
Currently
Healthcare AI evaluation grounded in clinical workflow.
Toward
Digital-health products that earn the trust of clinicians and patients.
No. 08
DPhil Student
Currently
Agentic-AI safety, including web-agent persuasion attacks.
Toward
Web agents that resist social engineering as a default behaviour.
No. 09
DPhil Student
Currently
The science of evals: measuring reasoning honestly.
Toward
A field where every benchmark publishes its construct validity.
No. 10
DPhil Student
Currently
Agentic systems for science, with safety and eval rigour.
Toward
Scientific agents auditable enough to act on in real research.
No. 11
MSc Student
Currently
Studies of human–LLM interaction and LLM evaluation.
Toward
Interaction patterns that improve, rather than substitute, human judgement.
No. 12
MSc Student
Currently
LLM evaluation and reasoning benchmarks.
Toward
Reasoning evals that separate genuine inference from recall.
No. 13
Visiting Policy Fellow
Currently
LLMs applied to financial time series, at the policy boundary.
Toward
Honest characterisation of LLM utility in high-stakes financial settings.
No. 14
Research Affiliate
Currently
Public-health AI and LLM evaluations grounded in epidemiology.
Toward
Public-health AI evaluated like a health intervention.
No. 15
Entrepreneur in Residence
Currently
Research-to-product translation
Toward
Steady work on research-to-product translation.
Papers accepted, conferences attended, and awards received. Most recent first.
Papers accepted at ICML 2026!
OxRML at ICLR 2026
Ryan Othniel Kearns Wins MSc Thesis Prize
New Paper in Nature Medicine!
OxRML @ NeurIPS 2025
OxRML @ EMNLP 2025
Prof. Adam Mahdi Wins Teaching Excellence Award 2025
New Paper in Information Fusion!
OxRML @ NeurIPS 2024
Winners of 2024 PhysioNet Challenge
OxRML @ ICML 2024
11 entries · most recent first
Three formats for partnering with the lab. Each is a real commitment — we are selective about partners and ramp slowly, because shared roadmaps need shared standards.
Format · 01
On-site sessions for product and ML teams on evaluation, safety, and agent reliability.
Half-day to multi-week formats. For teams shipping LLM products in healthcare, finance, retail, and government.
Format · 02
We work with engineering partners to turn lab work into tools other teams can run.
Evaluation harnesses, safety dashboards, agentic-research platforms. We build them with partners we trust, carrying the research methods through to the code.
Format · 03
Applied research collaborations with foundations, governments, and large companies.
Multi-year programmes: shared roadmaps, sponsored DPhil studentships, named labs.
We work with foundations, governments, and companies who want AI evaluation, safety, and reasoning treated as a science.
We reply to every serious enquiry.
Universities, journals, and conferences where the lab’s work has been hosted, peer-reviewed, and published.
§ 08 · AI News
New papers, open positions, partnership opportunities, and what we have been reading.
