An Empirical Study of Deep Learning Models for Vulnerability Detection

Deep learning (DL) models of code have recently reported great progress for vulnerability detection. In some cases, DL-based models have outperformed static analysis tools. Although many great models have been proposed, we do not yet have a good understanding of these models. This limits the further advancement of model robustness, debugging, and deployment for the vulnerability detection. In this paper, we surveyed and reproduced 9 state-of-the-art (SOTA) deep learning models on 2 widely used vulnerability detection datasets: Devign and MSR. We investigated 6 research questions in three areas, namely model capabilities, training data, and model interpretation. We experimentally demonstrated the variability between different runs of a model and the low agreement among different models’ outputs. We investigated models trained for specific types of vulnerabilities compared to a model that is trained on all the vulnerabilities at once. We explored the types of programs DL may consider ”hard” to handle. We investigated the relations of training data sizes and training data composition with model performance. Finally, we studied model interpretations and analyzed important features that the models used to make predictions. We believe that our findings can help better understand model results, provide guidance on preparing training data, and improve the robustness of the models.

近年来，代码深度学习（Deep Learning, DL）模型在漏洞检测领域取得了长足进展，部分基于深度学习的模型性能已超越传统静态分析工具。尽管已有诸多优秀模型被提出，但目前学界对这类模型的内在机制仍缺乏充分认知，这一现状限制了漏洞检测场景下模型鲁棒性提升、调试与部署工作的进一步推进。本文针对Devign与MSR这两个广泛使用的漏洞检测数据集，对9个当前最优（State-of-the-Art, SOTA）深度学习模型开展了调研与复现工作。我们从三大研究维度共设置了6项研究问题，分别为模型能力、训练数据特性与模型可解释性。实验层面，我们验证了模型多次训练运行间的性能差异性，以及不同模型输出结果间的低一致性；对比了针对特定漏洞类型训练的模型与基于全漏洞类型统一训练的模型；探究了深度学习模型难以处理的程序类型；分析了训练数据规模、训练数据构成与模型性能之间的关联；最后针对模型解释展开研究，剖析了模型用于生成预测结果的关键特征。我们认为，本文的研究发现可助力学界更深入地理解模型输出结果，为训练数据的准备工作提供实践指导，并有效推动漏洞检测模型鲁棒性的提升。