Large Language Models2026

Nemotron 3 Super

Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer for Agentic Reasoning

NVIDIA

Paper Overview

Nemotron 3 Super is NVIDIA's open-weight large language model that combines three architectural innovations — Hybrid Mamba-Attention, LatentMoE, and Multi-Token Prediction — into a single 120B total parameter model with only 12B active parameters per forward pass (12.7B including embeddings). The model uses a vocabulary of 131,072 tokens, a hidden dimension of $d = 4096$ , and spans 88 layers with a mix of Mamba-2 SSM blocks and grouped-query attention layers.

The model achieves comparable or better accuracy than GPT-OSS-120B (DeepSeek-V3-0324) and Qwen3.5-122B across reasoning, coding, agentic, and long-context benchmarks, while delivering dramatically better serving efficiency: 2.2x higher throughput than GPT-OSS-120B and 7.5x higher throughput than Qwen3.5-122B on realistic workloads (8K input / 64K output tokens) on NVIDIA B300 GPUs. This efficiency advantage comes from three sources: (1) Mamba-2 blocks eliminate KV cache growth for the majority of layers, (2) LatentMoE reduces memory bandwidth per expert by 4x through latent projection, and (3) Multi-Token Prediction enables native speculative decoding with an average acceptance length of 3.45 tokens — no external draft model required.

Key architectural and training advances:

LatentMoE: Projects tokens from $d = 4096$ to a latent space of $\ell = 1024$ before expert routing, enabling 512 total experts with top-22 activation at the same inference cost as standard MoE with ~64 experts. Expert weight matrices shrink from $d \times m$ to $\ell \times m$ (4x smaller), reducing memory bandwidth by 4x per expert read — critical for low-latency serving where expert weight loading dominates
Hybrid Mamba-2 + Attention: 88 layers with a periodic pattern of Mamba-2 blocks ( $O(n)$ linear time, constant-size recurrent state of dimension 128) and sparse attention "anchors" using GQA (32 Q-heads / 2 KV-heads). Mamba blocks handle local sequential patterns cheaply; attention anchors enable global information routing. Only the attention layers produce KV cache entries, keeping total cache small
Multi-Token Prediction (MTP): 2 shared-weight auxiliary prediction heads trained across multiple future offsets. The shared-weight design is critical — it makes the heads robust to the distribution shift of conditioning on self-generated states during speculative decoding. Average acceptance length: 3.45 tokens on SPEED-Bench (best among all tested models, beating DeepSeek-R1's 2.70)
NVFP4 pretraining: First model to demonstrate stable large-scale training in NVFP4 (E2M1) precision across 25 trillion tokens on NVIDIA Blackwell GPUs, with a two-phase data curriculum (20T diverse + 5T high-quality). Mixed precision strategy: NVFP4 for most linear layers, BF16 for attention, latent projections, MTP heads, and the final 15% of layers
Agentic post-training: 4-stage pipeline — SFT (7M samples) → multi-environment RLVR (37 datasets, 21 environments, up to 1K GPUs) → SWE-RL (end-to-end RL on real GitHub issues) → RLHF (GenRM on Qwen3-235B-Thinking) + MTP healing. PivotRL technique for efficient agentic RL training on "pivot" turns in expert trajectories

Nemotron 3 Super is fully open-sourced on HuggingFace — base, post-trained, FP8, and NVFP4 checkpoints — along with all training datasets (Nemotron-Pretraining-Specialized-v1.1 and Nemotron-Super-Post-Training-Data).

Chapter Roadmap

Click any topic to jump in

Pipeline Summary

LatentMoE → hybrid SSM → NVFP4 pretrain → multi-env RL → deployment quantization compound into 3× efficiency.

combines compute + memory wins via

LatentMoE

Latent projection before expert compute makes wall-clock cost invariant to expert count, enabling 256 experts.

Hybrid Mamba-2 + Attn

Interleaved SSM and attention layers give linear-context reasoning with 5× smaller KV cache at 128K.

scaled efficiently by

Multi-Token Prediction

Auxiliary heads predicting k future tokens densify supervision and enable free speculative decoding.

NVFP4 Pretraining

4-bit microscaled format halves memory and doubles tensor-core throughput with <0.3% loss gap.

aligned and shipped via

Multi-Env RL

Simultaneous RL across math/code/tool/chat with per-env rewards prevents single-domain reward hacking.

Deployment Quantization

Per-layer FP4/FP8/INT8 search yields 4× throughput with <1% quality loss via error-budget optimization.

The complete Nemotron 3 Super architecture — from hybrid Mamba-Attention backbone through LatentMoE sparse computation and multi-token prediction to efficient quantized deployment. Each component addresses a specific bottleneck in serving efficiency while maintaining or improving model quality.

Key Points

120B total parameters, 12B active per forward pass (12.7B including embeddings) — 10:1 sparsity ratio via LatentMoE expert selection

88-layer hybrid stack: periodic pattern of Mamba-2 SSM blocks (state dim 128, 128 heads, 8 groups, head dim 64) + GQA attention anchors (32 Q / 2 KV heads, head dim 128) + LatentMoE FFN layers (512 experts, top-22 in 1024-dim latent space)

LatentMoE: token projected d=4096 → l=1024, then routed to 22 of 512 experts operating in latent space — 4x less memory bandwidth per expert, exponentially larger combinatorial routing space

Multi-Token Prediction: 2 shared-weight auxiliary heads predicting future offsets — average acceptance length 3.45 tokens on SPEED-Bench (vs DeepSeek-R1's 2.70), shifting throughput-latency Pareto frontier to ~1000 TPS at D=3

Pre-trained on 25 trillion tokens in NVFP4 (E2M1 with 16-element micro-blocks): Phase 1 = 20T tokens diverse web/code/academic/multilingual, Phase 2 = 5T tokens high-quality PDFs/Wikipedia/curated

Post-trained via 5 stages: SFT (7M samples, 2-stage loss) → RLVR (21 environments, 37 datasets, async GRPO on up to 1K GPUs) → SWE-RL (end-to-end GitHub issues in Apptainer containers) → RLHF (GenRM on Qwen3-235B-Thinking) → MTP healing (frozen backbone, retrain speculative heads)

Quantized to FP8 (99.5% accuracy) and NVFP4 (99.8% accuracy) via AutoQuantize — NAS-inspired knapsack optimization for per-operator precision under 4.75-bit budget, with stochastic rounding for Mamba SSM cache

2.2x throughput vs GPT-OSS-120B and 7.5x vs Qwen3.5-122B on B300 GPUs (8K in / 64K out) — fully open-sourced: BF16, FP8, NVFP4 checkpoints + all training data on HuggingFace

Mathematical Intuition

Nemotron 3 Super composes four ideas in sequence: LatentMoE (hardware-aware sparsity) $\to$ Mamba-2 + attention hybrid (linear context) $\to$ NVFP4 pretraining (4-bit compute) $\to$ multi-environment RL post-training. Each stage attacks a different bottleneck — compute, memory, precision, and alignment — giving a compounding $\sim 3\times$ efficiency gain over dense baselines.

Back to All Papers

Study Plans

LLM Study Plan All Papers Practice Problems

Key architectural and training advances:

LatentMoE: Projects tokens from $d = 4096$ to a latent space of $\ell = 1024$ before expert routing, enabling 512 total experts with top-22 activation at the same inference cost as standard MoE with ~64 experts. Expert weight matrices shrink from $d \times m$ to $\ell \times m$ (4x smaller), reducing memory bandwidth by 4x per expert read — critical for low-latency serving where expert weight loading dominates
Hybrid Mamba-2 + Attention: 88 layers with a periodic pattern of Mamba-2 blocks ( $O(n)$ linear time, constant-size recurrent state of dimension 128) and sparse attention "anchors" using GQA (32 Q-heads / 2 KV-heads). Mamba blocks handle local sequential patterns cheaply; attention anchors enable global information routing. Only the attention layers produce KV cache entries, keeping total cache small
Multi-Token Prediction (MTP): 2 shared-weight auxiliary prediction heads trained across multiple future offsets. The shared-weight design is critical — it makes the heads robust to the distribution shift of conditioning on self-generated states during speculative decoding. Average acceptance length: 3.45 tokens on SPEED-Bench (best among all tested models, beating DeepSeek-R1's 2.70)
NVFP4 pretraining: First model to demonstrate stable large-scale training in NVFP4 (E2M1) precision across 25 trillion tokens on NVIDIA Blackwell GPUs, with a two-phase data curriculum (20T diverse + 5T high-quality). Mixed precision strategy: NVFP4 for most linear layers, BF16 for attention, latent projections, MTP heads, and the final 15% of layers
Agentic post-training: 4-stage pipeline — SFT (7M samples) → multi-environment RLVR (37 datasets, 21 environments, up to 1K GPUs) → SWE-RL (end-to-end RL on real GitHub issues) → RLHF (GenRM on Qwen3-235B-Thinking) + MTP healing. PivotRL technique for efficient agentic RL training on "pivot" turns in expert trajectories

Nemotron 3 Super

Paper Overview

Chapter Roadmap

Pipeline Summary

LatentMoE

Hybrid Mamba-2 + Attn

Multi-Token Prediction

NVFP4 Pretraining

Multi-Env RL

Deployment Quantization

Pipeline Summary

Key Points

LatentMoE: Hardware-Aware Expert Design

Hybrid Mamba-2 + Attention Architecture

Multi-Token Prediction (MTP)

NVFP4 Pretraining at Scale

Multi-Environment Reinforcement Learning

Quantization for Deployment

Related Papers

Study Plans

Nemotron 3 Super

Paper Overview

Chapter Roadmap

Pipeline Summary

LatentMoE

Hybrid Mamba-2 + Attn

Multi-Token Prediction

NVFP4 Pretraining

Multi-Env RL

Deployment Quantization

Pipeline Summary

Key Points

LatentMoE: Hardware-Aware Expert Design

Hybrid Mamba-2 + Attention Architecture

Multi-Token Prediction (MTP)

NVFP4 Pretraining at Scale

Multi-Environment Reinforcement Learning

Quantization for Deployment

Related Papers

Study Plans