A research practice, recorded as kata.

A benchmark that tells document-collection navigation apart from stochastic search in multimodal agents.

A spotlight at ICML 2026 showing agents and humans diverge sharply on document QA. Agents look fluent, but their search pattern is not strategic.

Existing document benchmarks reward fluency over navigation, so failure modes were invisible. The task itself had to be redesigned.

Extend MAD-QA to dynamic collections and longer horizons; pair it with traces, so we measure the search policy, not the final answer.

A measurement of LLM self-explanation faithfulness that holds up well enough to use in evaluation pipelines.

A positive ICML result: self-explanations, treated with care, predict downstream model behaviour better than chance, sometimes substantially.

The literature had assumed self-explanations were noise. Demonstrating utility required isolating the conditions in which they carry signal.

Integrate the faithfulness probe into safety evals; test whether it survives distribution shift and adversarial prompting.

A benchmark that measures whether a web agent can be socially engineered into abandoning its user, before that happens in deployment.

TRAP at ICML: across frontier web agents, persuasion attacks redirect tasks at high rates. The vulnerability is not a corner case.

Web agents are tested for capability, not for resistance to manipulation. There was no shared eval harness for adversarial redirection.

Open-source the harness; partner with web-agent labs to track resistance as a first-class metric, alongside task success.

Empirical evidence on how the general public uses LLMs as medical assistants: preregistered, randomised, replicable.

Nature Medicine paper: people make worse triage decisions with current LLM assistants than without, in randomised trials.

Most prior work tested clinicians or simulated patients, not the public. The methodology had to be built from epidemiology, not ML.

Iterate on assistive UI patterns and re-run the trial; share the protocol so other teams can replicate across populations.

A construct-validity standard that every public LLM benchmark can be audited against.

NeurIPS Datasets & Benchmarks paper: a structured audit of widely-used benchmarks, with concrete gaps between claim and measurement.

Benchmarks are shipped without measurement-theory scaffolding. Adopting construct validity from psychometrics took translation work.

Release the audit framework as a checklist; lobby venues to require construct-validity statements for new benchmarks.

A characterisation of LLM-as-a-judge across the conditions under which it is deployed: multilingual, multimodal, multi-domain.

NeurIPS workshop paper: judge reliability drops, sometimes catastrophically, when you push outside the conditions it was calibrated on.

Judge-LLM evals are typically reported on English text. Multilingual and multimodal coverage required new datasets and rubrics.

Publish a multilingual judge-stress-test suite; use it to recalibrate workflows that depend on LLM-as-a-judge.

An account of how Direct Preference Optimization changes a model, not just whether it changes outputs.

EMNLP paper: DPO suppresses toxicity through a small set of identifiable neurons, a localised intervention rather than a global rewrite.

Behavioural metrics tell you DPO worked; they do not tell you what was edited. Neuron-level attribution required new probing tooling.

Extend to other preference signals; ask whether the same neurons carry capability, and what the trade-off looks like.

A diagnostic that distinguishes LLM introspection from confabulated counterfactual explanations.

EMNLP paper: self-generated counterfactuals systematically mis-locate the model's decision boundary. The explanation is fluent but wrong.

Counterfactual explanations were treated as an interpretability win. Showing them unreliable needed careful controlled experiments.

Look for self-explanation formats that DO track the boundary; quantify the gap so practitioners can calibrate trust.

A coherent map of multimodal ML in healthcare: fusion strategies, deployment realities, and where the field has and has not delivered.

Information Fusion review: a synthesis spanning fusion architectures, clinical workflows, and the gaps between lab metrics and care.

Healthcare multimodal work is scattered across imaging, EHR, and signal-processing literatures with little shared vocabulary.

Use the map to scope the lab’s next clinical collaborations; identify modalities where careful evaluation is most overdue.

A reasoning benchmark you can trust when LLMs are trained on most of the internet: one that separates reasoning from recall.

ICLR paper: templatised orthographic obfuscation breaks memorisation while preserving the underlying reasoning task; scores drop sharply.

Standard reasoning benchmarks leak into training data. Designing an obfuscation that preserves the problem was the hard part.

Scale LingOly-TOO to more languages; use the same obfuscation idea on math and code reasoning benchmarks.