Ai2 opens a trillion-scale MoE training stack

Ai2 released Olmo-core 3 on October 1, a redesigned open training system for mixture-of-experts models. In one benchmark, expanding from eight to 128 experts while activating only four per token grew total capacity from 4.6B to 47B parameters with less than a 5% throughput decline. On eight NVIDIA B300 GPUs, the new stack processed 52,000 tokens per second per GPU for a 47B MoE, about 2.7 times the prior implementation. Ai2 also demonstrated a 1.2-trillion-parameter systems benchmark across 512 GPUs, while stressing that it measured infrastructure performance, not the quality of a fully trained model. Code, a technical report and an interactive walkthrough are open for inspection and adaptation.