Led by Prof. Adam Mahdi, the Reasoning with Machines Lab advances the science of AI evaluation, benchmarking, safety and security. Through rigorous empirical research, we study how LLMs and agentic systems reason, interact with humans, and drive scientific discovery. We work with industry partners deploying AI where reliability matters — teams and enterprises partner with the lab.
Adam leads OxRML. The group studies how language models reason, how people work with them, and how agentic systems behave on real scientific and decision-making tasks.
Public science is the product. Workshops and partnerships exist so the lab can keep producing it.
Read, cite, fork, and ship anything we publish. Built for researchers, students, hobbyist evaluators, and engineers who want the source. No license gymnastics, no signups, no quota.
arxiv, openreview & nature links direct — read, cite, fork, ship.
On-site sessions for product and ML teams on evaluation, safety, and agent reliability.
booked by teams at healthcare, finance, retail & gov shops shipping LLM products into high-stakes settings.
Applied research collaborations with foundations, governments, and large companies.
we work with partners who care about getting AI right. small number of slots per year.
How it works. Individual researchers read the work, build with it, bring it to their team. When the team needs to scale a benchmark, run an evaluation cohort, or audit an agent at production load, that's when teams book a workshop. When the workshop turns into a three-year roadmap, it becomes a partnership. We built it in this order on purpose.
The questions the lab is built around. If your day job touches any of them, you're in the right place.
We develop the science of LLM evaluation, setting the standard for rigorous assessment and identifying hidden risks before they matter.
From bias and toxicity to agentic misalignment, we study the full spectrum of AI risk and develop the technical and governance tools to address it.
We build agentic systems that automate scientific knowledge synthesis and discovery, with a focus on agents that are reliable, transparent and domain-grounded.
We run large-scale empirical studies on how people use AI for high stakes decisions, from healthcare and law to policy and beyond.
ICML spotlights, Nature Medicine, NeurIPS Datasets & Benchmarks, ICLR, EMNLP. Click through and you're on arXiv or OpenReview, not a request form.
A benchmark that tells real navigation apart from stochastic search when agents work over document collections.
Ł Borchmann, J Van Landeghem, M Turski, S Padarha, RO Kearns, A Mahdi, et al.
LLM self-explanations are usually dismissed as unreliable. Measured the right way, they predict model behavior.
H Mayne, JS Kang, D Gould, K Ramchandran, A Mahdi, NY Siegel
A benchmark for whether web agents can be socially engineered into abandoning the user's task. Today's agents fall for it.
K Korgul, Y Yang, A Drohomirecki, P Błaszczyk, W Howard, L Aichberger, C Russell, P H S Torr, A Mahdi, A Bibi
A preregistered randomized study in Nature Medicine on how reliably LLMs serve as medical assistants for the general public.
AM Bean, RE Payne, G Parsons, HR Kirk, J Ciro, R Mosquera-Gómez, S Hincapié, AS Ekanayaka, L Tarassenko, L Rocher, A Mahdi
A construct-validity audit of widely-used LLM benchmarks: what they claim to measure versus what they capture.
AM Bean, RO Kearns, A Romanou, FS Hafner, H Mayne, J Batzner, et al.
How LLM judges degrade across languages, modalities, and domains, and where the failure modes sit.
S Padarha, E Semenova, B Vidgen, A Mahdi, S A Hale
Direct Preference Optimization reduces toxicity. We trace where it acts, neuron by neuron.
Y Yang, F Sondej, H Mayne, A Lee, A Mahdi
Ask an LLM "what would change your answer?" and it looks like introspection. It is often confabulation.
H Mayne, RO Kearns, Y Yang, AM Bean, E Delaney, C Russell, A Mahdi
A survey of multimodal ML in clinical practice, from data-fusion strategies through to deployment.
F Krones, U Marikkar, G Parsons, A Szmul, A Mahdi
A benchmark that obfuscates orthography to strip memorised knowledge out of reasoning problems, showing how much "reasoning" was recall.
J Khouja, K Korgul, S Hellsten, L Yang, V Neacsu, H Mayne, RO Kearns, A Bean, A Mahdi
A small team in Oxford. We write the papers, tag the datasets, and answer the workshop emails. When you book the lab, you get these people, not an account exec.
Oxford Internet Institute, University of Oxford
Adam leads OxRML. The group studies how language models reason, how people work with them, and how agentic systems behave on real scientific and decision-making tasks.
DPhil Student
Multimodal AI, digital health
DPhil Student
AI and food security, LLMs
DPhil Student
LLM evaluations, human–LLM interaction
DPhil Student
LLM & agentic post-training, AI alignment
DPhil Student
LLM interpretability, AI safety, LLM evaluations
DPhil Student
Knowledge graphs, metascience
DPhil Student
Healthcare AI, digital health
DPhil Student
AI safety, agentic AI
DPhil Student
Science of evals, reasoning in LLMs
DPhil Student
AI for science, AI safety, LLM evaluations
MSc Student
Human–LLM interaction, LLM evaluations
MSc Student
LLM evaluations, reasoning
Visiting Policy Fellow
LLMs and financial time series
Research Affiliate
Public health AI, LLM evaluations
Entrepreneur in Residence
Research-to-product translation
A working diary: papers shipped, talks given, awards we didn't see coming. Most recent first.
Papers accepted at ICML 2026!
OxRML at ICLR 2026
Ryan Othniel Kearns Wins MSc Thesis Prize
New Paper in Nature Medicine!
OxRML @ NeurIPS 2025
OxRML @ EMNLP 2025
Prof. Adam Mahdi Wins Teaching Excellence Award 2025
New Paper in Information Fusion!
OxRML @ NeurIPS 2024
Winners of 2024 PhysioNet Challenge
OxRML @ ICML 2024
Venues and institutions that have published or hosted our research. Each one ties to a paper or programme in the diary above.
New papers, open positions, partnership opportunities, and what we have been reading.