The Llama 3 Herd of Models

The Llama 3 Herd of Models (Llama Team, AI @ Meta, 2024) is a 92-page engineering report, and it makes an unfashionable argument: a frontier model does not need a new architecture. The flagship is a dense Transformer with 405B parameters and a 128K-token context window, pre-trained on 15.6T tokens with $3.8 \times 10^{25}$ FLOPs — almost 50x the largest Llama 2 — and it performs on par with GPT-4 across a wide range of tasks.

The authors name three levers: data, scale, and managing complexity. Data means roughly 15T multilingual tokens (versus 1.8T for Llama 2) with much stricter curation. Scale means a flagship sized by the team's own scaling laws to be approximately compute-optimal for its budget. Managing complexity means choosing the boring option whenever it scales better: a standard dense Transformer instead of mixture-of-experts, and SFT + rejection sampling + DPO instead of PPO-style RLHF.

This breakdown follows the method. It covers the architecture tweaks (GQA, RoPE base 500,000, a 128K vocabulary), the IsoFLOP scaling laws that picked 405B, the data mix and annealing, the six-stage extension to 128K context, the 4D-parallel training stack on 16K H100s, the post-training loop, and the compositional vision and speech adapters that add modalities without touching the language model's weights.