Gibbs Priors for Bayesian Nonparametric Variable Selection with Weak Learners

We consider the problem of high-dimensional Bayesian nonparametric variable selection using an aggregation of so-called “weak learners.” The most popular variant of this is the Bayesian additive regression trees (BART) model, which is the natural Bayesian analog to boosting decision trees. In this article, we use Gibbs distributions on random partitions to induce sparsity in ensembles of weak learners. Looking at BART as a special case, we show that the class of Gibbs priors includes two recently proposed models—the Dirichlet additive regression trees (DART) model and the spike-and-forest model—as extremal cases, and we show that certain Gibbs priors are capable of achieving the benefits of both the DART and spike-and-forest models while avoiding some of their key drawbacks. We then show the promising performance of Gibbs priors for other classes of weak learners, such as tensor products of spline basis functions. A Pólya Urn scheme is developed for efficient computations. Supplementary materials for this article are available online.

本文研究基于所谓“弱学习器（weak learner）”集成的高维贝叶斯非参数变量选择问题。其中最主流的变体为贝叶斯可加回归树（Bayesian additive regression trees, BART）模型，其是决策树提升方法的自然贝叶斯类比。本文采用随机划分上的吉布斯分布（Gibbs distribution）在弱学习器集成中引入稀疏性。以BART作为特例，我们证明该类吉布斯先验包含了近期提出的两种模型——狄利克雷可加回归树（Dirichlet additive regression trees, DART）模型与尖峰-森林模型（spike-and-forest model）——作为其极端情形；同时表明，部分吉布斯先验能够兼具DART模型与尖峰-森林模型的优势，同时规避二者的部分关键缺陷。随后，我们验证了吉布斯先验在其他类别的弱学习器（如样条基函数的张量积）上同样展现出颇具潜力的优异表现。本文开发了用于高效计算的波利亚瓮（Pólya Urn）方案，相关补充材料可在线获取。