Droid: A Resource Suite for AI-Generated Code Detection

Daniil Orel, Indraneil Paul, Iryna Gurevych, Preslav Nakov

​

Presenter: Daniil Orel (daniil.orel@mbzuai.ac.ae)

​

Problem Statement

Current tools for detecting AI-generated code rely on narrow datasets-limited models, languages, and domains-making them unreliable. As post-training methods advance, code copilots may even be optimized to evade such detectors. This calls for more robust approaches to identify human, AI, hybrid authorship, and adversarial generations across diverse scenarios.

​

Building DroidCollection (AI-Generated data)

Open-Source Datasets: TACO, CodeNet, StarCoderData, etc

Problem

Statement

Docstring

Comments

Source

Code

Solution

Implementation

Inverse

Instruction

Implementation

DroidCollection

Building DroidCollection (Unconditional Data)

Programmer

Description

Potential

Tasks

Solution

We generate programmer descriptions based on: primary programming language, field of work (web, DS/ML, IoT, game development, etc), code commenting style, error-proneness, debugging strategies, documentation habits, function length.

Building DroidCollection (Hybrid Data)

We take human-written code and:

• Ask LLMs to rewrite it
• Remove N% of the code from the beginning, from the end or in the middle and ask LLMs to fill-in these gaps

Code

Sample

Rewritten

Code

Incomplete

Code

Code Completion

Building DroidCollection (Adversarial Data)

We collected a preference dataset (DroidCollection-Pref) of 157K Human-AI authored code pairs. Then we run LoRA DPO training to make models prefer Human-like code over AI-styled one

Original

LLM

DPO

Fine-Tuning

Adversarial

Code

​

Human-Like

LLM

Varying Sampling Parameters

Previous work [1] has shown that AI-generated code detection with greedy decoding is easier than detection with varied sampling strategies. That is why we vary sampling strategies in our dataset.

[1] Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. Automatic Detection of Generated Text is Easiest when Humans are Fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808–1822.

Data Filtering

• Parsed only valid ASTs; discarded unparsable code.
• Kept AST depth between 2 and 31.
• Enforced max line length (12–400 chars) and avg line length (5–140 chars).
• Retained samples with 6–300 lines of code.
• Required alphanumeric fraction between 0.2 and 0.75 to exclude configs/auto-generated code.
• Kept only docstrings with >99% English confidence (using Lingua detector).
• Removed near-duplicates via MinHash (shingle size = 8, threshold = 0.8)

To ensure that the data quality is high, we apply a series of filters to the dataset.

​

We did not filter out human-written codes that appear after the spark of AI-driven code copilots

​

Do we really need a large dataset?

Zero-Shot

Fine-Tuned

Ours

Is adversarial data challenging?

Zero-Shot

Fine-Tuned

We see that recall of zero-shot models on adversarial data is very low. Only M4 and CoDet-M4 show recall over 0.5, but they are just biased towards positive class

Training DroidDetect

• Take ModernBERT backbone
• Use Metric Learning to improve performance
• We use Triple-Loss to make representation of samples from different classes more distant
• Resample dataset to remove noisy and potentially incorrectly labeled samples
• We use Monte-Carlo dropout to measure model’s uncertainty on Human-Writtent class samples in the training set. 7% of the less certain samples are removed
• Then the model is re-trained

​

How input length affects performance?

We observe that while other models tend to perform worse on larger inputs (likely because they were trained on short code snippets), our model performs better with longer inputs

Can our model generalize?

We evaluate our model on 15k samples from unseen datasets:

• Swallow-Code - a dataset of AI-generated codes, obtained using Llama3.3 70B
• The Heap - a dataset, which includes samples which are not present in The Stack (which was our main data source)

Conclusion

We introduce DroidCollection, a large, diverse dataset suite designed for training and evaluating robust AI-generated code detectors. It supports common LLM code copilot scenarios—code completion, rewriting, and adversarial uses—and offers broader coverage than existing public corpora in terms of generators, generation settings, programming languages, and domains. Using DroidCollection, we developed DroidDetect, a family of detection models (base and large), which outperform existing approaches across multiple tasks. We also conducted extensive ablation studies to identify the most effective training strategies.

​

Dataset & Paper & Contact

mbzuai.ac.ae

Mohamed bin Zayed University of Artificial Intelligence

Masdar City, Abu Dhabi, United Arab Emirates

​

Thank you