Press · ⌘K
For the index, search, jump to a section, or contact the editor.
Filed Hilary Term · University of Oxford
An empirical research group at the Oxford Internet Institute. We study LLM evaluation, safety, reasoning, and the agentic systems built from them.
Editor’s note. A research group working in the open: benchmarks that hold up under scrutiny, safety work that survives contact with deployment, and agentic systems for the high-stakes domains that already trust us in healthcare, policy, and science.
The Index
Six sections, page numbers included. Use this to skip ahead to whatever you came for.
§02
Four pillars
Read across the four columns below. Each one is a portfolio, not a project.
We work on the science of LLM evaluation: what benchmarks measure, where they mislead, and how to build ones that hold up.
Cross-ref. → Folio §03, Roster §04
We work on bias, toxicity, and agentic misalignment, and on the technical and governance tools that address them.
Cross-ref. → Folio §03, Roster §04
Agentic systems for scientific work. We focus on keeping them reliable, transparent, and grounded in the domain.
Cross-ref. → Folio §03, Roster §04
Empirical studies of how people use AI in high-stakes settings: healthcare, law, and policy.
Cross-ref. → Folio §03, Roster §04
§03
Selected works
10 papers · 2025-26
Lead · fig. 02ICML (Spotlight) · May 2026
Ł Borchmann, J Van Landeghem, M Turski, S Padarha, RO Kearns, A Mahdi, et al.
A benchmark that tells real navigation apart from stochastic search when agents work over document collections.
ICML
H Mayne, JS Kang, D Gould, K Ramchandran, A Mahdi, NY Siegel
LLM self-explanations are usually dismissed as unreliable. Measured the right way, they predict model behavior.
Read full text →
ICML
K Korgul, Y Yang, A Drohomirecki, P Błaszczyk, W Howard, L Aichberger, C Russell, P H S Torr, A Mahdi, A Bibi
A benchmark for whether web agents can be socially engineered into abandoning the user's task. Today's agents fall for it.
Read full text →
Nature Medicine
AM Bean, RE Payne, G Parsons, HR Kirk, J Ciro, R Mosquera-Gómez, S Hincapié, AS Ekanayaka, L Tarassenko, L Rocher, A Mahdi
A preregistered randomized study in Nature Medicine on how reliably LLMs serve as medical assistants for the general public.
Read full text →
NeurIPS Datasets and Benchmarks
AM Bean, RO Kearns, A Romanou, FS Hafner, H Mayne, J Batzner, et al.
A construct-validity audit of widely-used LLM benchmarks: what they claim to measure versus what they capture.
Read full text →
NeurIPS LLM Lifecycle Workshop
S Padarha, E Semenova, B Vidgen, A Mahdi, S A Hale
How LLM judges degrade across languages, modalities, and domains, and where the failure modes sit.
Read full text →§04
The contributors
DPhil students, MSc researchers, visiting fellows and affiliates. Each portrait is one column of the gazette.
No. 01Felix Krones
DPhil Student
Multimodal AI, digital health
No. 02Djavan De Clercq
DPhil Student
AI and food security, LLMs
No. 03Andrew M. Bean
DPhil Student
LLM evaluations, human–LLM interaction
No. 04Yushi Yang
DPhil Student
LLM & agentic post-training, AI alignment
No. 05Harry Mayne
DPhil Student
LLM interpretability, AI safety, LLM evaluations
No. 06Jessica Rodrigues
DPhil Student
Knowledge graphs, metascience
No. 07Guy Parsons
DPhil Student
Healthcare AI, digital health
No. 08Karolina Korgul
DPhil Student
AI safety, agentic AI
No. 09Ryan Othniel Kearns
DPhil Student
Science of evals, reasoning in LLMs
No. 10Shreyansh Padarha
DPhil Student
AI for science, AI safety, LLM evaluations
No. 11Mia Kussman
MSc Student
Human–LLM interaction, LLM evaluations
No. 12Caleb Tan
MSc Student
LLM evaluations, reasoning
Also contributing this term.
§05
Three doors in
Foundations, governments, and global operators have hired us to do three things. Read the briefs below.
On-site sessions for product and ML teams on evaluation, safety, and agent reliability.
Half-day to multi-week formats. For teams shipping LLM products in healthcare, finance, retail, and government.
Book a workshop →We work with engineering partners to turn lab work into tools other teams can run.
Evaluation harnesses, safety dashboards, agentic-research platforms. We build them with partners we trust, carrying the research methods through to the code.
See our builds →Applied research collaborations with foundations, governments, and large companies.
Multi-year programmes: shared roadmaps, sponsored DPhil studentships, named labs.
Start a conversation →In good company.
§06 · Stop press
Nine recent items, chronological, no filler. If you want the longer read, the Folio is next door.
Three OxRML papers accepted at ICML 2026 — including a Spotlight
paperOxRML presenting at ICLR 2026
conferenceNew paper in Nature Medicine on LLMs as medical assistants
paperRyan Othniel Kearns wins MSc Thesis Prize
awardOxRML at NeurIPS 2025
conferenceOxRML at EMNLP 2025
conferenceProf. Adam Mahdi wins Oxford Teaching Excellence Award 2025
awardNew review paper in Information Fusion
paperWinners of the 2024 PhysioNet Challenge
awardThe lab newsletter
New papers, open positions, partnership opportunities, and what we have been reading.