OLMoE Mix

该数据集被用于训练专家混合（Mixture of Experts, MoE）模型，其中包含了1.3亿个活跃参数，总计参数量为6.9亿。在规模上，该数据集涉及到的活跃参数为10亿，总参数量为70亿，其任务是对语言建模进行专家混合模型的训练。This dataset is utilized for training Mixture of Experts (MoE) models. It contains 130 million active parameters, with a total parameter count of 690 million. In terms of scale, this dataset involves 1 billion active parameters and a total parameter count of 7 billion, and its core task is to train Mixture of Experts models for language modeling.