§01 · Research
H Mayne, RO Kearns, Y Yang, AM Bean, E Delaney, C Russell, A Mahdi
Ask an LLM "what would change your answer?" and it looks like introspection. It is often confabulation.
Y Yang, F Sondej, H Mayne, A Lee, A Mahdi
Direct Preference Optimization reduces toxicity. We trace where it acts, neuron by neuron.
S Padarha, E Semenova, B Vidgen, A Mahdi, S A Hale
How LLM judges degrade across languages, modalities, and domains, and where the failure modes sit.
AM Bean, RO Kearns, A Romanou, FS Hafner, H Mayne, J Batzner, et al.
A construct-validity audit of widely-used LLM benchmarks: what they claim to measure versus what they capture.
H Mayne, JS Kang, D Gould, K Ramchandran, A Mahdi, NY Siegel
LLM self-explanations are usually dismissed as unreliable. Measured the right way, they predict model behavior.
Ł Borchmann, J Van Landeghem, M Turski, S Padarha, RO Kearns, A Mahdi, et al.
A benchmark that tells real navigation apart from stochastic search when agents work over document collections.
K Korgul, Y Yang, A Drohomirecki, P Błaszczyk, W Howard, L Aichberger, C Russell, P H S Torr, A Mahdi, A Bibi
A benchmark for whether web agents can be socially engineered into abandoning the user's task. Today's agents fall for it.
AM Bean, RE Payne, G Parsons, HR Kirk, J Ciro, R Mosquera-Gómez, S Hincapié, AS Ekanayaka, L Tarassenko, L Rocher, A Mahdi
A preregistered randomized study in Nature Medicine on how reliably LLMs serve as medical assistants for the general public.
F Krones, U Marikkar, G Parsons, A Szmul, A Mahdi
A survey of multimodal ML in clinical practice, from data-fusion strategies through to deployment.
J Khouja, K Korgul, S Hellsten, L Yang, V Neacsu, H Mayne, RO Kearns, A Bean, A Mahdi
A benchmark that obfuscates orthography to strip memorised knowledge out of reasoning problems, showing how much "reasoning" was recall.
§ Next
Want to collaborate on a paper or dataset?