{
 "id": "epsj2qajvm",
 "slug": "albert-language-model",
 "title": "ALBERT (language model)",
 "updated": "2026-10-15",
 "topic_path": [
  {
   "id": "technology",
   "label": "Technology and the built world",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology"
  },
  {
   "id": "technology.computing",
   "label": "Computing and digital systems",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing"
  },
  {
   "id": "technology.computing.modern_ai",
   "label": "Modern AI: foundation models, generative AI, and the AI industry",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai"
  },
  {
   "id": "technology.computing.modern_ai.models",
   "label": "Model families and named models",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai.models"
  },
  {
   "id": "technology.computing.modern_ai.models.llm_families",
   "label": "Large language model families",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai.models.llm_families"
  },
  {
   "id": "technology.computing.modern_ai.models.llm_families.open-weight-model-families",
   "label": "Open-weight model families",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.modern_ai.models.llm_families.open-weight-model-families"
  }
 ],
 "geo": [
  {
   "id": "geo.nongeo.t2001.technology.computing",
   "label": "Non-geographic · 2001 to 2020: Computing and digital systems",
   "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing",
   "path": [
    {
     "id": "geo.nongeo",
     "label": "Non-geographic",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo"
    },
    {
     "id": "geo.nongeo.t2001",
     "label": "Non-geographic · 2001 to 2020",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001"
    },
    {
     "id": "geo.nongeo.t2001.technology",
     "label": "Technology and the built world",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology"
    },
    {
     "id": "geo.nongeo.t2001.technology.computing",
     "label": "Computing and digital systems",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing"
    }
   ]
  }
 ],
 "excerpt": "ALBERT is a transformer-based language model architecture that reduces BERT-style encoder parameters through factorized embeddings and cross-layer sharing, reported in 2019 and released as open source.",
 "snippet": "ALBERT is a transformer-based language model architecture that reduces BERT-style encoder parameters through factorized embeddings and cross-layer sharing, reported in 2019 and released as open source.",
 "node": "technology.computing.modern_ai.models.llm_families.open-weight-model-families",
 "markdown": "# ALBERT (language model)\n\nALBERT is a transformer-based language model architecture that reduces the number of parameters in a BERT-style encoder through factorized embedding parameterization and cross-layer parameter sharing, while keeping competitive accuracy on natural language understanding tasks.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> An ALBERT configuration matched to BERT-large has 18 times fewer parameters, 18M versus 334M, and can be trained about 1.7 times faster.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> The design addressed a practical problem of 2019-era pre-training: scaling hidden size and depth in models like BERT inflates memory use, so a model that holds state-of-the-art accuracy with far fewer parameters is cheaper to store and distribute. The paper reporting ALBERT was accepted at ICLR 2020 and advanced reported state-of-the-art performance on 12 NLP tasks, and the models were released as open source on [TensorFlow](https://www.edgepedia.org/tensorflow).<sup>[2](https://research.google/blog/albert-a-lite-bert-for-self-supervised-learning-of-language-representations/)</sup>\n\n| Key fact | Value |\n|---|---|\n| Parameter-reduction techniques | Factorized embedding parameterization and cross-layer parameter sharing<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> |\n| ALBERT-base | 12M parameters, 12 layers, hidden 768, embedding 128<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> |\n| ALBERT-large | 18M parameters, 24 layers, hidden 1024, embedding 128<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> |\n| ALBERT-xxlarge | 235M parameters, 12 layers, hidden 4096, embedding 128<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> |\n| Pre-training objective | Masked language modeling with n-gram masking plus sentence-order prediction (SOP)<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> |\n| Speed vs BERT-large | ALBERT-large about 1.7x faster; ALBERT-xxlarge about 3x slower<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> |\n| Availability | Open source since 2019-09-26; in Hugging Face Transformers since 2020-11-16<sup>[3](https://huggingface.co/docs/transformers/main/en/model_doc/albert)</sup> |\n\n## How it works\n\nALBERT keeps the encoder structure of BERT but changes how parameters are spent. Factorized embedding parameterization splits the vocabulary embedding matrix into two smaller matrices: one maps each vocabulary token to a relatively low-dimensional embedding (for example 128 dimensions), and a second projects that embedding up to the hidden size of the encoder (768 or more). This reduces the embedding parameters from \\( O(V \\times H) \\) to \\( O(V \\times E + E \\times H) \\), where \\( V \\) is vocabulary size, \\( E \\) the embedding dimension, and \\( H \\) the hidden size; the saving is significant when \\( H \\gg E \\).<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> Applied alone, factorization cuts the parameters of the projection block by 80% while SQuAD2.0 drops only from 80.4 to 80.3.<sup>[2](https://research.google/blog/albert-a-lite-bert-for-self-supervised-learning-of-language-representations/)</sup>\n\nCross-layer parameter sharing applies the same layer, in effect, on top of itself across the depth of the network, so parameters do not grow with depth.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> The motivation is that [Transformer](https://www.edgepedia.org/transformer) layers often learn similar operations at various layers, a redundancy that sharing eliminates.<sup>[2](https://research.google/blog/albert-a-lite-bert-for-self-supervised-learning-of-language-representations/)</sup> Sharing achieves a 90% parameter reduction for the attention-feedforward block, about 70% overall, at a cost of -0.3 on SQuAD2.0 (to 80.0) and -3.9 on RACE (to 64.0) when combined with factorization.<sup>[2](https://research.google/blog/albert-a-lite-bert-for-self-supervised-learning-of-language-representations/)</sup> The reduction techniques also act as a form of regularization that stabilizes training and helps generalization.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup>\n\nALBERT also replaces BERT's next-sentence prediction (NSP) loss with a self-supervised sentence-order prediction (SOP) loss, which focuses on inter-sentence coherence and was designed to address the documented ineffectiveness of NSP.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> SOP uses two consecutive segments from the same document as positive examples and the same two segments with their order swapped as negatives.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> The motivation is measurable: on an intrinsic SOP task, a model trained with NSP reaches only 52.0% accuracy, at random-guess level, while a model trained with SOP reaches 78.9% on NSP and 86.5% on SOP, and SOP improves downstream multi-sentence tasks by about +1% on SQuAD1.1, +2% on SQuAD2.0, and +1.7% on RACE.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup>\n\n## How it is done\n\nPre-training follows the BERT recipe with ALBERT's modifications. All model updates use a batch size of 4096 and the LAMB optimizer with a learning rate of 0.00176, for 125,000 steps unless otherwise specified, on Cloud TPU V3 hardware.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> Masked-language-modeling targets use n-gram masking with a maximum n-gram length of 3.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> The trained checkpoints are then fine-tuned on downstream tasks; the model cards describe ALBERT as aimed at tasks that use a whole sentence to make decisions, such as sequence classification, token classification, and question answering.<sup>[4](https://huggingface.co/albert/albert-base-v2)</sup> An independent replication study pre-trained ALBERT-base with the paper's hyperparameters on a 16GB corpus of [English Wikipedia](https://www.edgepedia.org/english-wikipedia) plus [Project Gutenberg](https://www.edgepedia.org/project-gutenberg), finishing 1M pre-training steps in eight days on a single Cloud TPU V3 at a cost of around 700 USD.<sup>[5](https://aclanthology.org/2020.emnlp-main.553.pdf)</sup>\n\n## Origin\n\nALBERT was reported by Zhenzhong Lan and colleagues in \"ALBERT: A Lite BERT for Self-supervised Learning of Language Representations\", released on arXiv in 2019.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> The model was released on 2019-09-26 and added to [Hugging Face Transformers](https://www.edgepedia.org/hugging-face-transformers) on 2020-11-16.<sup>[3](https://huggingface.co/docs/transformers/main/en/model_doc/albert)</sup> The paper builds on BERT's pre-training framework, replacing BERT's NSP loss with SOP, and its parameter-sharing design follows the observation that Transformer layers learn similar operations; the same period produced related pre-training alternatives such as XLNet, reported by Zhilin Yang and colleagues in 2019.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup><sup> • </sup><sup>[6](https://doi.org/10.48550/arxiv.1906.08237)</sup>\n\n## Variants\n\nThe paper's architecture table gives four shared-parameter configurations: ALBERT-base with 12M parameters (12 layers, hidden 768, embedding 128), ALBERT-large with 18M (24 layers, 1024, 128), ALBERT-xlarge with 60M (24 layers, 2048, 128), and ALBERT-xxlarge with 235M (12 layers, 4096, 128).<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> The model cards list albert-base-v2 with 12 repeating layers, 128 embedding dimension, 768 hidden dimension, 12 attention heads, and 11M parameters, and albert-large-v2 with 24 repeating layers, 128 embedding dimension, 1024 hidden dimension, 16 attention heads, and 17M parameters.<sup>[4](https://huggingface.co/albert/albert-base-v2)</sup>\n\nWith around 70% of BERT-large's parameters, ALBERT-xxlarge improves over BERT-large on development sets by +1.9% on SQuAD v1.1, +3.1% on SQuAD v2.0, +1.4% on MNLI, +2.2% on SST-2, and +8.4% on RACE; its absolute scores are 94.1/88.3 on SQuAD1.1, 88.1/85.1 on SQuAD2.0, 88.0 on MNLI, 95.2 on SST-2, and 82.3 on RACE.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup>\n\nThe v2 checkpoints changed the ranking of sizes. The repository reports v2 averages of 82.3 for ALBERT-base, 85.7 for ALBERT-large, 87.9 for ALBERT-xlarge, and 90.9 for ALBERT-xxlarge, while v1 xxlarge averaged 91.0.<sup>[7](https://github.com/google-research/albert/)</sup> For base, large, and xlarge, v2 is much better than v1, which the authors attribute to three training strategies applied in v2: no dropout, additional training data, and longer training time.<sup>[7](https://github.com/google-research/albert/)</sup>\n\n## Applications\n\nIn practice, ALBERT checkpoints are used by fine-tuning on sentence-level tasks such as sequence classification, token classification, and question answering.<sup>[4](https://huggingface.co/albert/albert-base-v2)</sup> Chinese ALBERT models were released in the official repository, with training data provided by the CLUE team.<sup>[7](https://github.com/google-research/albert/)</sup> The architecture remains supported in current Hugging Face Transformers documentation.<sup>[3](https://huggingface.co/docs/transformers/main/en/model_doc/albert)</sup>\n\nNew ALBERT-based models continued to appear after 2023. MiniALBERT credits ALBERT with popularizing shared parameterisation and builds ALBERT-like recursive-transformer models of 12M to 32M parameters distilled from larger language models.<sup>[8](https://aclanthology.org/2023.eacl-main.83.pdf)</sup> mALBERT pre-trains a new multilingual ALBERT from scratch, describing ALBERT as the smallest pre-trained model family at the time with 12 million parameters and a model size under 50 megabytes, and noting its ecological advantages over bigger models.<sup>[9](http://www.lrec-conf.org/proceedings/lrec-coling-2024/pdf/2024.main-1.960.pdf)</sup>\n\n## Limitations and alternatives\n\nALBERT's central limitation is that parameter efficiency does not translate into inference efficiency. Parameter sharing cuts memory but not computation: every layer is still executed at each depth step, so a shared model with a large hidden size does as much arithmetic as an unshared one of the same shape. Using BERT-large as the baseline, ALBERT-large is about 1.7 times faster in iterating through the data, while ALBERT-xxlarge is about 3 times slower because of its larger structure.<sup>[1](https://doi.org/10.48550/arxiv.1909.11942)</sup> The two reduction techniques also trade accuracy differently: factorization alone costs almost nothing, while adding sharing costs more, and later work states the same downside plainly, that a common downside of ALBERT-style sharing is slower inference time and reduced performance relative to fully parameterized models.<sup>[2](https://research.google/blog/albert-a-lite-bert-for-self-supervised-learning-of-language-representations/)</sup><sup> • </sup><sup>[8](https://aclanthology.org/2023.eacl-main.83.pdf)</sup> Training cost remains substantial even for the small variant, as the eight-day, roughly 700 USD single-TPU replication of ALBERT-base shows.<sup>[5](https://aclanthology.org/2020.emnlp-main.553.pdf)</sup>\n\nAlternatives take different routes to efficiency. ELECTRA-Large, trained with a discriminator-based objective, outperforms ALBERT on GLUE and performs comparably to RoBERTa and XLNet despite a smaller compute budget.<sup>[10](https://arxiv.org/html/2003.10555v1)</sup> [DistilBERT](https://www.edgepedia.org/distilbert) compresses BERT by knowledge distillation instead of sharing: it has 40% fewer parameters than BERT, is 60% faster, retains 97% of BERT's language understanding capability, and uses a triple loss combining language modeling, distillation, and cosine-distance losses.<sup>[11](https://arxiv.org/pdf/1910.01108v4)</sup>\n\n## References\n\n1. [Lan, Zhenzhong and colleagues (2019). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1909.11942)\n2. [ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations (Google Research blog)](https://research.google/blog/albert-a-lite-bert-for-self-supervised-learning-of-language-representations/)\n3. [ALBERT · Hugging Face Transformers documentation](https://huggingface.co/docs/transformers/main/en/model_doc/albert)\n4. [albert/albert-base-v2 model card](https://huggingface.co/albert/albert-base-v2)\n5. [Pretrained Language Model Embryology: The Birth of ALBERT](https://aclanthology.org/2020.emnlp-main.553.pdf)\n6. [Yang, Zhilin and colleagues (2019). XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1906.08237)\n7. [google-research/albert (official repository README)](https://github.com/google-research/albert/)\n8. [MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers](https://aclanthology.org/2023.eacl-main.83.pdf)\n9. [mALBERT: Is a Compact Multilingual BERT Model Still Worth It?](http://www.lrec-conf.org/proceedings/lrec-coling-2024/pdf/2024.main-1.960.pdf)\n10. [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://arxiv.org/html/2003.10555v1)\n11. [DistilBERT: a smaller, faster, cheaper and lighter version of BERT](https://arxiv.org/pdf/1910.01108v4)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Model families and named models › Large language model families › Open-weight model families*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n",
 "same_as": [],
 "url": "https://www.edgepedia.org/albert-language-model",
 "markdown_url": "https://www.edgepedia.org/albert-language-model.md",
 "license": {
  "name": "Edgepedia Community License 1.0",
  "url": "https://www.edgepedia.org/edgepedia/license",
  "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.",
  "spdx": "LicenseRef-Edgepedia-Community-1.0"
 },
 "credit": "\"ALBERT (language model)\", Edgepedia (EdgeChat), https://www.edgepedia.org/albert-language-model. Edgepedia Community License 1.0.",
 "credit_md": "\"[ALBERT (language model)](https://www.edgepedia.org/albert-language-model)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/albert-language-model](https://www.edgepedia.org/albert-language-model). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).",
 "credit_html": "\"<a href=\"https://www.edgepedia.org/albert-language-model\">ALBERT (language model)</a>\", Edgepedia (EdgeChat), <a href=\"https://www.edgepedia.org/albert-language-model\">https://www.edgepedia.org/albert-language-model</a>. <a href=\"https://www.edgepedia.org/edgepedia/license\">Edgepedia Community License 1.0</a>.",
 "speakable": "ALBERT is a transformer-based language model architecture that reduces BERT-style encoder parameters through factorized embeddings and cross-layer sharing, reported in 2019 and released as open source."
}