Nemotron 3 Super

Nemotron 3 Super is NVIDIA's open-weight large language model that combines three architectural innovations — Hybrid Mamba-Attention, LatentMoE, and Multi-Token Prediction — into a single 120B total parameter model with only 12B active parameters per forward pass (12.7B including embeddings). The model uses a vocabulary of 131,072 tokens, a hidden dimension of $d = 4096$, and spans 88 layers with a mix of Mamba-2 SSM blocks and grouped-query attention layers.

The model achieves comparable or better accuracy than GPT-OSS-120B (DeepSeek-V3-0324) and Qwen3.5-122B across reasoning, coding, agentic, and long-context benchmarks, while delivering dramatically better serving efficiency: 2.2x higher throughput than GPT-OSS-120B and 7.5x higher throughput than Qwen3.5-122B on realistic workloads (8K input / 64K output tokens) on NVIDIA B300 GPUs. This efficiency advantage comes from three sources: (1) Mamba-2 blocks eliminate KV cache growth for the majority of layers, (2) LatentMoE reduces memory bandwidth per expert by 4x through latent projection, and (3) Multi-Token Prediction enables native speculative decoding with an average acceptance length of 3.45 tokens — no external draft model required.

Key architectural and training advances:

• LatentMoE: Projects tokens from $d = 4096$ to a latent space of $\ell = 1024$ before expert routing, enabling 512 total experts with top-22 activation at the same inference cost as standard MoE with ~64 experts. Expert weight matrices shrink from $d \times m$ to $\ell \times m$ (4x smaller), reducing memory bandwidth by 4x per expert read — critical for low-latency serving where expert weight loading dominates
• Hybrid Mamba-2 + Attention: 88 layers with a periodic pattern of Mamba-2 blocks ($O(n)$ linear time, constant-size recurrent state of dimension 128) and sparse attention "anchors" using GQA (32 Q-heads / 2 KV-heads). Mamba blocks handle local sequential patterns cheaply; attention anchors enable global information routing. Only the attention layers produce KV cache entries, keeping total cache small
• Multi-Token Prediction (MTP): 2 shared-weight auxiliary prediction heads trained across multiple future offsets. The shared-weight design is critical — it makes the heads robust to the distribution shift of conditioning on self-generated states during speculative decoding. Average acceptance length: 3.45 tokens on SPEED-Bench (best among all tested models, beating DeepSeek-R1's 2.70)
• NVFP4 pretraining: First model to demonstrate stable large-scale training in NVFP4 (E2M1) precision across 25 trillion tokens on NVIDIA Blackwell GPUs, with a two-phase data curriculum (20T diverse + 5T high-quality). Mixed precision strategy: NVFP4 for most linear layers, BF16 for attention, latent projections, MTP heads, and the final 15% of layers
• Agentic post-training: 4-stage pipeline — SFT (7M samples) → multi-environment RLVR (37 datasets, 21 environments, up to 1K GPUs) → SWE-RL (end-to-end RL on real GitHub issues) → RLHF (GenRM on Qwen3-235B-Thinking) + MTP healing. PivotRL technique for efficient agentic RL training on "pivot" turns in expert trajectories

Nemotron 3 Super is fully open-sourced on HuggingFace — base, post-trained, FP8, and NVFP4 checkpoints — along with all training datasets (Nemotron-Pretraining-Specialized-v1.1 and Nemotron-Super-Post-Training-Data).