Assessment of the Generalization Abilities of Machine-Learning Scoring Functions for Structure-Based Virtual Screening

In structure-based virtual screening (SBVS), it is critical that scoring functions capture protein–ligand atomic interactions. By focusing on the local domains of ligand binding pockets, a standardized pocket Pfam-based clustering (Pfam-cluster) approach was developed to assess the cross-target generalization ability of machine-learning scoring functions (MLSFs). Subsequently, 12 typical MLSFs were evaluated using random cross-validation (Random-CV), protein sequence similarity-based cross-validation (Seq-CV), and pocket Pfam-based cross-validation (Pfam-CV) methods. Surprisingly, all of the tested models showed decreased performances from Random-CV to Seq-CV to Pfam-CV experiments, not showing satisfactory generalization capacity. Our interpretable analysis suggested that the predictions on novel targets by MLSFs were dependent on buried solvent-accessible surface area (SASA)-related features of complex structures, with greater predicted binding affinities on complexes owning larger protein–ligand interfaces. By combining buried SASA-related features with target-specific patterns that were only shared among structurally similar compounds in the same cluster, the random forest (RF)-Score attained a good performance in the Random-CV test. Based on these findings, we strongly advise assessing the generalization ability of MLSFs with the Pfam-cluster approach and being cautious with the features learned by MLSFs.

在基于结构的虚拟筛选（structure-based virtual screening, SBVS）中，打分函数能否精准捕捉蛋白质与配体的原子间相互作用，是该领域的核心关键。本研究聚焦配体结合口袋的局部区域，开发了一种标准化的基于Pfam家族的口袋聚类（Pfam-cluster）方法，用于评估机器学习打分函数（machine-learning scoring functions, MLSFs）的跨靶标泛化能力。随后，我们采用随机交叉验证（Random-CV）、基于蛋白质序列相似性的交叉验证（Seq-CV）以及基于口袋Pfam家族的交叉验证（Pfam-CV）三种方法，对12种典型机器学习打分函数开展了性能评测。令人意外的是，所有受试模型的性能均呈现出从随机交叉验证到序列相似性交叉验证，再到Pfam家族聚类交叉验证逐步下降的趋势，未表现出令人满意的泛化能力。本研究的可解释性分析表明，机器学习打分函数对全新靶标的预测结果，依赖于复合物结构中与埋藏式溶剂可及表面积（solvent-accessible surface area, SASA）相关的特征；具体而言，拥有更大蛋白质-配体界面的复合物，其预测结合亲和力更高。通过将埋藏式溶剂可及表面积相关特征与仅在同一聚类中结构相似化合物间共享的靶标特异性模式相结合，随机森林（random forest, RF）-Score在随机交叉验证测试中取得了优异的性能表现。基于上述研究发现，我们强烈建议在评估机器学习打分函数的泛化能力时，采用Pfam-cluster方法，并对机器学习打分函数学习到的特征保持审慎态度。