Is This You, LLM? Recognizing AI-written Programs with Multilingual Code Stylometry - Laboratoire Traitement et Communication de l'Information
Communication Dans Un Congrès Année : 2025

Is This You, LLM? Recognizing AI-written Programs with Multilingual Code Stylometry

Résumé

With the increasing popularity of LLM-based code completers, like GitHub Copilot, the interest in automatically detecting AI-generated code is also increasing-in particular in contexts where the use of LLMs to program is forbidden by policy due to security, intellectual property, or ethical concerns. We introduce a novel technique for AI code stylometry, i.e., the ability to distinguish code generated by LLMs from code written by humans, based on a transformer-based encoder classifier. Differently from previous work, our classifier is capable of detecting AI-written code across 10 different programming languages with a single machine learning model, maintaining high average accuracy across all languages (84.1% ± 3.8%). Together with the classifier we also release H-AIRosettaMP, a novel open dataset for AI code stylometry tasks, consisting of 121 247 code snippets in 10 popular programming languages, labeled as either human-written or AI-generated. The experimental pipeline (dataset, training code, resulting models) is the first fully reproducible one for the AI code stylometry task. Most notably our experiments rely only on open LLMs, rather than on proprietary/closed ones like ChatGPT.
Fichier principal
Vignette du fichier
main.pdf (438.73 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04845581 , version 1 (18-12-2024)

Identifiants

  • HAL Id : hal-04845581 , version 1

Citer

Andrea Gurioli, Maurizio Gabbrielli, Stefano Zacchiroli. Is This You, LLM? Recognizing AI-written Programs with Multilingual Code Stylometry. IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2025, Mar 2025, Montréal, Canada. ⟨hal-04845581⟩
0 Consultations
0 Téléchargements

Partager

More