1 of 11

FabScore: Fine-Grained Evaluation of Fabrications in Automated AI Research

Hui Chen, James Xu Zhao, Dongfu Jiang, Qianyun Guo, Jiefeng Chen,

Yiwei Wang, Muhao Chen, See-Kiong Ng, Pang Wei Koh, Bryan Hooi

AI4Science Workshop · ICML 2026 · Seoul

2 of 11

Motivation

AI can now do research end-to-end — ideas, code, experiments, papers

Are the results reliable and reproducible?

Fabrication: paper's methods or reported results ≠ what the code implements or reproduces

We need fine-grained, execution-grounded fabrication evaluation

Reproduced from The AI Scientist (2024).

3 of 11

FabScore: Fine-Grained Fabrication Evaluation

Given a paper + its code, FabScore checks every numerical claim by analyzing and executing the code.

RQ1: Can we detect fabrications reliably?

RQ2: Which types are most common, and what are the reasons?

RQ3: How do fabrication rates differ across different sources?

RQ4: Can we catch what current AI reviewers miss?

4 of 11

FabScore Pipeline

Research paper and code

(1) Result Extraction

# Claims to Be Verified

## Tables

1. Table 1, Baseline, KL Divergence: 0.354

2. Table 1, Baseline, Training Time (s): 37.42

## Figures

1. Figure 1: Validation accuracy over training steps.

…

5. Figure 5: Training loss over time for each dataset.

## Results Section

1. Inference time increased by an average of 42% for our model compared to baseline.

(2) Static Analysis

Extracted claims

Execution

required

(3) Code Execution

(4) Verdict Generation

No execution required: verdicts via static analysis

Verdicts via code execution

Coding Agent

Inspect source code for inconsistencies

Execute code in a shell environment

Coding Agent

< > Codebase

experiment.py

> Bash ( python experiment.py –out_dir run_0 2>&1 )

> Running experiment.py in the background with —out_dir run_0. It will run 4 datasets ( x_div_y, x_minus_y, x_plus_y, permutation ) x 3 seeds each. I’ll notify you when it completes.

5 of 11

RQ1: Reliability of FabScore

Human evaluation: 6 experts, 30 papers across various sources: (1) The AI Scientist, (2) MLR-Agent, (3) Agents4Science Open Conference (Acc./Rej.), (4) FARS

Compare two coding agents: Claude Code vs Codex

Claude Code: 98.6% precision — reliable detection with almost no false positives → adopted for all experiments.

	Claude Code	Codex
Precision (# confirmed / # detected)	98.6%	80.1%
Label Accuracy (% type matches)	83.8%	92.2%

6 of 11

RQ2: Common Types of Fabrications

6,978 claims from 144 papers across various sources introduced earlier

Overall claim-level fabrication rate: 21.2% — only ~50% reproducible

Experiment fabrication is the most common type

A large gap between research ideas and correct code implementation!

1.4% + 16.8% + 3.0% = 21.2%

7 of 11

RQ2: Reasons for Fabrications

Data fabrication — mock / synthetic data instead of real data (64.6%)

Experiment fabrication — hardcoded / simulated values replace real experiments (44.6%)

Result fabrication — reported value conflicts with its own execution logs (85.9%)

When ideas are too complex to implement, AI systems tend to take shortcuts!

Top reason within each fabrication type:

Experiment Fabrication - Submission 220 from Agents4Science

Paper’s Claim: Table 2, Success Rate, BiCA: 85.5 ± 4.5%

Why This is a Fabrication: This result is derived from a hardcoded metric value. Specifically, the evaluation script sets a fixed base success rate of 0.85 and uses np.random.normal() to simulate metric values. However, Section 4.3.3 of the paper defines success rate as the fraction of episodes that reach the task goal within the step limit. Therefore, the reported metric is fabricated.

The source code for the metric hardcodes the experimental result values!

8 of 11

RQ3: Fabrication Rates in Different Sources

(1) All sources contain fabrications — claim-level 0.4%–53.6%, paper-level 10%–81.5%

(2) >70% of real conference submissions contain fabrications — even accepted ones reach 59.3%

(3) Rejected papers fabricate more → negatively correlated with acceptance

Even with human involvement, AI-authored research is not guaranteed to be rigorous!

9 of 11

RQ4: Comparison with AI Reviewers

(1) AI reviewers cover only 13.8% of FabScore-detected fabrications → over 85% missed

(2) Worst on experiment fabrication — the most common type — only 10.2% caught

(3) AI reviews rely on intra-paper contradictions (63.3%), almost no code analysis (4.1%)

AI reviewers mainly read text, not code — FabScore can provide execution-based evidence.

10 of 11

Takeaways

(1) Fabrications are prevalent: >70% of real AI-authored submissions contain them — even accepted ones (59.3%)

“Be cautious when using AI to do research.”

(2) Experiment fabrication dominates → AI systems take shortcuts when ideas are too hard to implement

“Make sure the idea is clear and feasible before implementing.”

(3) AI reviewers miss 85% — FabScore is a complementary tool with autonomous code analysis & execution

“Don't just read the paper — check the code too.”

As AI takes on a greater role in science, research integrity matters more than ever!