# DeepSeek (model family)
The DeepSeek family is a line of open-weight large language models built on mixture-of-experts (MoE) architecture and an efficiency-first design philosophy, published across 2024 and January 2025. Its defining releases are DeepSeekMoE, DeepSeek-V2, DeepSeek-V3, and the reasoning models [DeepSeek-R1](https://www.edgepedia.org/deepseek-r1) and R1-Zero, all published with downloadable weights and, in R1's case, an MIT license that explicitly permits distillation (training a smaller model to imitate a larger one). The company that makes the models, its founders, and the consumer chatbot built on them are covered in separate articles.
| Key fact | Detail |
|---|---|
| Architecture | Mixture-of-experts with Multi-head Latent Attention; 671B total parameters, 37B activated per token in V3 and R1[1](https://arxiv.org/pdf/2412.19437v2) • [2](https://github.com/deepseek-ai/DeepSeek-r1) |
| V3 training cost (vendor-reported) | 2.788M H800 GPU-hours, priced at $5.576M at $2 per GPU-hour, excluding prior research and ablations[1](https://arxiv.org/pdf/2412.19437v2) |
| V3 pre-training data | 14.8 trillion tokens, followed by supervised fine-tuning and reinforcement learning[3](https://github.com/deepseek-ai/DeepSeek-V3?tab=readme-ov-file) |
| R1 reasoning recipe | Reinforcement learning applied directly to the base model (R1-Zero), then cold-start data plus RL (R1)[2](https://github.com/deepseek-ai/DeepSeek-r1) |
| R1-Zero AIME 2024 gain (vendor-reported) | pass@1 rose from 15.6% to 71.0% during RL; 86.7% with majority voting[4](https://raw.githubusercontent.com/deepseek-ai/DeepSeek-R1/main/DeepSeek_R1.pdf) |
| License | MIT for R1 code and weights, including distillation; distills inherit Qwen (Apache 2.0) and Llama 3.1/3.3 licenses[2](https://github.com/deepseek-ai/DeepSeek-r1) • [5](https://huggingface.co/deepseek-ai/DeepSeek-R1) |
| Distilled models | Six dense models of 1.5B, 7B, 8B, 14B, 32B, and 70B parameters, based on Qwen and Llama[4](https://raw.githubusercontent.com/deepseek-ai/DeepSeek-R1/main/DeepSeek_R1.pdf) |
## What the DeepSeek family is
The family's consistent design choice is to trade total model size for per-token compute. Each flagship model is large in parameter count but activates only a small fraction of its weights for any given token, which lowers both training and inference cost relative to a dense model of comparable quality. The lab publishes technical reports, weights, and (for R1) a training recipe.
## Release timeline and versions
- **DeepSeekMoE.** A 16B-parameter MoE model that, according to the peer-reviewed ACL 2024 paper, achieved performance comparable to DeepSeek 7B and LLaMA2 7B using only about 40% of the computations. It established the fine-grained expert-specialization design used throughout the family.[6](https://aclanthology.org/2024.acl-long.70.pdf)
- **DeepSeek-V2 (May 2024).** Pre-trained on 8.1 trillion tokens; the vendor reported that it cut training costs by 42.5%, reduced the [KV cache](https://www.edgepedia.org/kv-cache) by 93.3% and raised maximum generation throughput 5.76 times relative to its predecessor.[7](https://arxiv.org/pdf/2405.04434v5)
- **DeepSeek-V3 (December 2024).** A 671B-parameter MoE model with 37B activated per token, pre-trained on 14.8 trillion tokens and then fine-tuned with supervised learning and reinforcement learning.[3](https://github.com/deepseek-ai/DeepSeek-V3?tab=readme-ov-file)
- **DeepSeek-R1 and R1-Zero (January 2025).** [Reasoning models](https://www.edgepedia.org/reasoning-models) built on V3, released together with six dense distilled models of 1.5B, 7B, 8B, 14B, 32B, and 70B parameters based on Qwen, and [Llama](https://www.edgepedia.org/llama-model-family).[4](https://raw.githubusercontent.com/deepseek-ai/DeepSeek-R1/main/DeepSeek_R1.pdf)
## Architecture and training as published
The family's efficiency rests on several vendor-published techniques. Multi-head Latent Attention (MLA) is adopted for efficient inference, and DeepSeekMoE provides cost-effective training through many fine-grained experts.[1](https://arxiv.org/pdf/2412.19437v2) V3 adds an auxiliary-loss-free load-balancing strategy and a multi-token prediction training objective.[1](https://arxiv.org/pdf/2412.19437v2)
The reasoning line applies reinforcement learning in a distinctive way. R1-Zero was trained with large-scale RL applied directly to the base model without any supervised fine-tuning stage; DeepSeek describes this as the first open research to validate that LLM reasoning can be incentivized purely through RL. The same report notes the costs of skipping supervised data: R1-Zero showed endless repetition, poor readability, and language mixing. R1 addressed these by introducing cold-start data before the RL stage.[2](https://github.com/deepseek-ai/DeepSeek-r1)
## By the numbers (vendor-reported)
Every benchmark and cost figure in this section comes from DeepSeek's own technical reports and model cards; no independent evaluation appears in the record for this article.
**Training cost.** DeepSeek reported that V3's full training used 2.788M H800 GPU-hours, broken down as 2,664K for pre-training, 119K for context extension, and 5K for post-training, which the vendor priced at $5.576M assuming $2 per H800 GPU-hour. The report explicitly excludes costs of prior research, ablation experiments, and architecture work from this figure.[1](https://arxiv.org/pdf/2412.19437v2) Pre-training consumed 180K H800 GPU-hours per trillion tokens, about 3.7 days on the lab's 2,048-GPU H800 cluster, with the whole pre-training run completing in under two months.[1](https://arxiv.org/pdf/2412.19437v2)
**Reasoning benchmarks.** DeepSeek reported that R1 achieves performance comparable to OpenAI-o1-1217 on reasoning tasks.[4](https://raw.githubusercontent.com/deepseek-ai/DeepSeek-R1/main/DeepSeek_R1.pdf) During R1-Zero's RL training, the pass@1 score on AIME 2024 rose from 15.6% to 71.0%, and to 86.7% with majority voting.[4](https://raw.githubusercontent.com/deepseek-ai/DeepSeek-R1/main/DeepSeek_R1.pdf) The vendor also claims DeepSeek-R1-Distill-Qwen-32B outperforms OpenAI-o1-mini across various benchmarks, and that V3 outperforms other open-source models and is comparable to leading closed-source models with a remarkably stable training process.[2](https://github.com/deepseek-ai/DeepSeek-r1) • [3](https://github.com/deepseek-ai/DeepSeek-V3?tab=readme-ov-file)
## Licensing and availability
The R1 code repository and model weights are MIT-licensed, supporting commercial use, any modifications and derivative works, explicitly including distillation. The weights are published on [Hugging Face](https://www.edgepedia.org/hugging-face).[2](https://github.com/deepseek-ai/DeepSeek-r1) • [5](https://huggingface.co/deepseek-ai/DeepSeek-R1) The six distilled models carry inherited licenses: the Qwen-based distills were fine-tuned from Apache 2.0 Qwen-2.5 models on 800K samples curated with DeepSeek-R1, while the Llama-based distills carry Llama 3.1/3.3 licenses.[2](https://github.com/deepseek-ai/DeepSeek-r1)
## References
1. DeepSeek-V3 Technical Report (arXiv), https://arxiv.org/pdf/2412.19437v2
2. deepseek-ai/DeepSeek-R1 (GitHub model card), https://github.com/deepseek-ai/DeepSeek-r1
3. deepseek-ai/DeepSeek-V3 (GitHub README), https://github.com/deepseek-ai/DeepSeek-V3?tab=readme-ov-file
4. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv), https://raw.githubusercontent.com/deepseek-ai/DeepSeek-R1/main/DeepSeek_R1.pdf
5. deepseek-ai/DeepSeek-R1 (Hugging Face model page), https://huggingface.co/deepseek-ai/DeepSeek-R1
6. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (ACL 2024), https://aclanthology.org/2024.acl-long.70.pdf
7. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (arXiv), https://arxiv.org/pdf/2405.04434v5
---
*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Model families and named models › Large language model families › Open-weight model families*
*Initially written Sep 17, 2026 · Edited: Oct 11, 2026; Oct 15, 2026*
*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*
License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license