Extrapolated Cross-Validation for Randomized Ensembles

Ensemble methods such as bagging and random forests are ubiquitous in various fields, from finance to genomics. Despite their prevalence, the question of the efficient tuning of ensemble parameters has received relatively little attention. This article introduces a cross-validation method, Extrapolated Cross-Validation (ECV), for tuning the ensemble and subsample sizes in randomized ensembles. Our method builds on two primary ingredients: initial estimators for small ensemble sizes using out-of-bag errors and a novel risk extrapolation technique that leverages the structure of prediction risk decomposition. By establishing uniform consistency of our risk extrapolation technique over ensemble and subsample sizes, we show that ECV yields <i>δ</i>-optimal (with respect to the oracle-tuned risk) ensembles for squared prediction risk. Our theory accommodates general predictors, only requires mild moment assumptions, and allows for high-dimensional regimes where the feature dimension grows with the sample size. As a practical case study, we employ ECV to predict surface protein abundances from gene expressions in single-cell multiomics using random forests under a computational constraint on the maximum ensemble size. Compared to sample-split and <i>K</i>-fold cross-validation, ECV achieves higher accuracy by avoiding sample splitting. Meanwhile, its computational cost is considerably lower owing to the use of the risk extrapolation technique. Supplementary materials for this article are available online.装袋（bagging）、随机森林（random forests）等集成学习方法在从金融到基因组学的诸多领域中均应用广泛。尽管这类方法普及率极高，但集成模型参数的高效调优问题却相对未受到足够重视。本文提出一种用于随机集成模型中集成规模与子采样规模调优的交叉验证方法：外推交叉验证（Extrapolated Cross-Validation，ECV）。我们的方法基于两大核心要素：一是利用袋外误差（out-of-bag errors）构建小型集成规模下的初始估计量，二是一种全新的风险外推技术，该技术充分利用了预测风险分解的结构特性。通过证明我们的风险外推技术在集成规模与子采样规模范围内具有一致相合性，本文证明：针对平方预测风险而言，ECV可生成相对于神谕调优风险的δ最优集成模型。我们的理论适用于一般预测器，仅需温和的矩条件假设，且可适配特征维度随样本量增长的高维场景。作为实际案例研究，我们在最大集成规模的计算约束下，借助随机森林基于单细胞多组学的基因表达数据预测表面蛋白丰度。相较于样本拆分法与K折交叉验证，ECV通过规避样本拆分步骤实现了更高的预测精度。与此同时，得益于风险外推技术的应用，其计算成本显著更低。本文的补充材料可在线获取。