Synergize Post-Training with the Science of Evaluation:

Making AI Agents Smarter and Safer

Terry Jingchen Zhang

Slides at terry-zhang.notion.site

Making AI agents smarter and safer at the same time

Guide Fine-tuning Towards Generalizable Reasoning

Improve (Multimodal) Reasoning

By probing transferability/generalizability of reasoning performance

Smarter

Uncover Agentic Misaligned Behavior

By analyzing when AI agents commit deception and harmful actions

Safer

Tackle Benchmark Saturation and Contamination

Internalize Consequence-Aware Reasoning for Safety

Look for Key Triggers of Emergent Scheming

Benchmark Saturation: From 20% to 100%

PioneerPhysics: By Researchers, For Researchers

https://pioneerphysics.netlify.app

Benchmark Contamination: Memorize vs. Generalize

Disentangle Key Capability

Various Use Case

Interdisciplinary Synergy: Physics-AI Reciprocity

Towards Trustworthy AI in Scientific Workflow

Making AI agents smarter and safer at the same time

Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy

Against Benchmark Contamination

Bernhard Schölkopf

Mrinmaya Sachan

Gopal

Dev*

Ning

Wang

Terry J. C. Zhang*

Zhijing

Jin

FoRLM @NeurIPS 2025

Read the Paper

​

From Research Papers to Questions

What did we do beyond reproduction study of RealMath?

1. We extended the framework in 3 aspects
1. Not just math, but also physics (and other science)
2. Use multimodal model for synthesis (o4-mini)
3. Omit “context” entry to synthesize self-contained question and avoid leakage (and save tokens)
2. More rigorous settings on #problems per month per domain to be more rigorous (RealMath simply partitioned the dataset to before vs. after without fine-grained comparison i.e. 2022 vs. 2025)
3. We tested on 4 leading model family, each represented by 2 models of similar size but different cutoff dates.

If we sum the model performance over the same time window (X months before vs. X months after cutoff) this also proves the RealMath explanation is not the actual reason for no decay

Quick Recall: Retrieval vs. Synthesis Benchmarking

LiveCodeBench (ICLR 2025)

RealMath (NeurIPS 2025)

Hypothesis: The synthesis process transformed the question such that the models do not recognize/memorize

​

From Retrieval to Synthesis: We synthesized questions based on LiveCodeBench

From Synthesis to Retrieval: We reverse-synthesized CLOZE questions on RealMath

Same code after synthesis

Same papers in CLOZE

Evaluation on CLOZE QAs

We generate CLOZE (fill-in-the-blank) questions from paper abstracts based on RealMath and our papers dataset.

​

These are direct recall type questions to test memorization in the models.

​

We observe consistent drop in performance in post-cutoff scenarios for all the models.

Drop in binary accuracy scores on CLOZE QAs

Evaluation on Transformed LiveCodeBench QAs

LiveCodeBench dataset is directly sourced from codeforces, LeetCode and AtCoder problems, which show clear temporal post-cutoff decay.

​

We apply the reasoning-driven synthesis to transform this dataset into QA pairs with same core solution.

​

We reevaluate the same models and the post-cutoff decay disappears!

​

This stands in contrast to previous work of longitudinal study LiveCodeBench (ICLR 2025) and ToTheCutoff (ICLR 2024).

Relatively stable performance on synthesised dataset

The “LiveBench” approach: use new info, periodically

When new models come out => Use a new datasets => Eval all the models again using this new dataset

(It’s costly, and papers from different time focus on different things)

​

We want to make a further point on longitudinal analysis:

It’s not (just) the new contents that matters

Rather “just” the synthesis process (i.e. transformation by a reasoning model) makes the difference (post-cutoff decay is not seen in synthesis-based benchmark, but apparent in retrieval-based benchmark)

What is different within RealMath+us vs. LiveCode/ToTheCutOff

The “LiveBench” approach: use new info, periodically

When new models come out => Use a new datasets => Eval all the models again using this new dataset

(It’s costly, and papers from different time focus on different things)

​

We want to make a further point on longitudinal analysis:

It’s not (just) the new contents that matters

Rather “just” the synthesis process (i.e. transformation by a reasoning model) makes the difference (post-cutoff decay is not seen in synthesis-based benchmark, but apparent in retrieval-based benchmark)

Validation Experiment: We test both ways

From Retrieval to Synthesis: We synthesized questions based on LiveCodeBench

From Synthesis to Retrieval: We reverse-synthesized CLOZE questions on RealMath

Same code after synthesis

Same papers in CLOZE

Collective Intelligence: On the Promise and Reality of Multi-Agent Systems for AI-Driven Scientific Discovery

Sirui

Lu

Bernhard Schölkopf

​

Yongjin

Yang

Yinya

Huang

Terry J. C. Zhang*

Zhijing

Jin

Preprint 2025

Read the Paper

​

Human Scientists Collaborate to Progress, Why Not AI?

​

What do MAS have to offer across 5 research stages?

Future AI Co-Scientist: Co-Evolve with Human Researcher