The Llama 3 Herd of Models
A 405B Dense Transformer, Its Scaling Laws, 16K-GPU Training Stack, SFT + Rejection Sampling + DPO Post-Training, and Compositional Vision & Speech Adapters
Llama Team, AI @ Meta
Read the Paper on arXivPaper Overview
The Llama 3 Herd of Models (Llama Team, AI @ Meta, 2024) is a 92-page engineering report, and it makes an unfashionable argument: a frontier model does not need a new architecture. The flagship is a dense Transformer with 405B parameters and a 128K-token context window, pre-trained on 15.6T tokens with FLOPs — almost 50x the largest Llama 2 — and it performs on par with GPT-4 across a wide range of tasks.
The authors name three levers: data, scale, and managing complexity. Data means roughly 15T multilingual tokens (versus 1.8T for Llama 2) with much stricter curation. Scale means a flagship sized by the team's own scaling laws to be approximately compute-optimal for its budget. Managing complexity means choosing the boring option whenever it scales better: a standard dense Transformer instead of mixture-of-experts, and SFT + rejection sampling + DPO instead of PPO-style RLHF.
This breakdown follows the method. It covers the architecture tweaks (GQA, RoPE base 500,000, a 128K vocabulary), the IsoFLOP scaling laws that picked 405B, the data mix and annealing, the six-stage extension to 128K context, the 4D-parallel training stack on 16K H100s, the post-training loop, and the compositional vision and speech adapters that add modalities without touching the language model's weights.
Chapter Roadmap
Click any topic to jump in
Data, Scale, Managing Complexity
Meta's thesis: frontier quality comes from better data, far more compute, and deliberately boring design choices that scale.
A Deliberately Standard Dense Transformer
No mixture-of-experts. GQA with 8 KV heads, RoPE base 500,000, a 128K-token vocabulary and a cross-document attention mask.
Compute-Optimal Sizing of the 405B
IsoFLOP parabolas from 6e18 to 1e22 FLOPs extrapolate to 402B params on 16.55T tokens at a 3.8e25 FLOP budget.
What the model sees
Data Mix + Annealing
~15T curated multilingual tokens: 50% general, 25% math/reasoning, 17% code, 8% multilingual — finished by annealing on high-quality data.
Extending to 128K Context
Six gated stages from 8K to 128K using ~800B tokens, only advancing once short-context scores recover and needle retrieval is perfect.
4D Parallelism on 16K H100s
TP, CP, PP and FSDP nested by bandwidth need; 38-43% BF16 MFU and >90% effective training time despite 419 unexpected interruptions.
SFT + Rejection Sampling + DPO Rounds
Six rounds of reward model, best-of-K rejection sampling, SFT and DPO — no PPO — with model averaging at every stage.
Compositional Vision & Speech
Frozen LLM plus trained encoders: cross-attention adapters for images/video, a token-embedding adapter for speech.
Results
405B lands on par with GPT-4-class models (87.3 MMLU, 96.8 GSM8K, 89.0 HumanEval); the adapters beat GPT-4V and Whisper.
Most frontier-model papers argue for a new mechanism. Llama 3 argues against needing one. The report says its performance gains come primarily from better data quality and diversity and from more training scale. The architecture changes very little from Llama 2.
The three levers are worth taking one at a time, because each one decides something later in the paper. Data: Llama 3 is pre-trained on a corpus of about 15T multilingual tokens, against 1.8T for Llama 2. The report adds more careful pre-processing and curation for pre-training, and more rigorous quality assurance and filtering for post-training data. Scale: the flagship was trained with FLOPs, almost 50x the largest Llama 2, on 15.6T text tokens. The scaling laws say that size is roughly compute-optimal. The 8B and 70B models are deliberately trained far longer than compute-optimal, because an over-trained small model beats a compute-optimal one at the same inference cost.
The third lever, managing complexity, is the unusual one. The team picks a standard dense Transformer over a mixture-of-experts model to maximize training stability. It picks a simple post-training loop — supervised finetuning, rejection sampling, and DPO — over reinforcement-learning algorithms it describes as less stable and harder to scale. At 16,000 GPUs and 54-day training stretches, every source of instability costs real time, and simplicity is a capability in its own right.
The overall pipeline has two stages (Figure 1). Pre-training converts a large multilingual corpus into tokens and trains next-token prediction with an 8K window, then continues pre-training to reach 128K. Post-training aligns the model with human feedback over several rounds and adds capabilities such as tool use. Separately, the paper runs compositional experiments that attach image, video, and speech encoders to the finished model through adapters. Those multimodal models were still under development and were not released with the paper.
Key Points
Three released sizes — 8B, 70B, 405B — all multilingual, all with a context window of up to 128K tokens in the Llama 3.1 release; every result in the paper is for Llama 3.1
Flagship budget: FLOPs, almost 50x the largest Llama 2, on 15.6T text tokens
Pre-training corpus of about 15T multilingual tokens, versus 1.8T for Llama 2
The 405B is roughly compute-optimal. The 8B and 70B are trained far past compute-optimal because that gives better models at a fixed inference budget
Dense, not MoE — chosen explicitly to maximize training stability
SFT + rejection sampling + DPO, not PPO — chosen because the RL alternatives are less stable and harder to scale
The flagship is also used to improve the smaller models during post-training
Released alongside Llama Guard 3 for input and output safety; the vision and speech extensions were not released