{
 "id": "ep0tq37sth",
 "slug": "layer-normalization",
 "title": "Layer normalization",
 "updated": "2026-10-15",
 "topic_path": [
  {
   "id": "technology",
   "label": "Technology and the built world",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology"
  },
  {
   "id": "technology.computing",
   "label": "Computing and digital systems",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing"
  },
  {
   "id": "technology.computing.ai",
   "label": "Artificial intelligence and data",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai"
  },
  {
   "id": "technology.computing.ai.ml_computation",
   "label": "Machine learning and neural computation",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation"
  },
  {
   "id": "technology.computing.ai.neural_deep",
   "label": "Neural networks and deep learning",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.neural_deep"
  },
  {
   "id": "technology.computing.ai.neural_deep.alpha-i-to-z",
   "label": "Titles I to Z",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.neural_deep.alpha-i-to-z"
  }
 ],
 "geo": [
  {
   "id": "geo.nongeo.t2001.technology.computing.ai.neural_deep",
   "label": "Non-geographic · 2001 to 2020: Neural networks and deep learning",
   "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.neural_deep",
   "path": [
    {
     "id": "geo.nongeo",
     "label": "Non-geographic",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo"
    },
    {
     "id": "geo.nongeo.t2001",
     "label": "Non-geographic · 2001 to 2020",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001"
    },
    {
     "id": "geo.nongeo.t2001.technology",
     "label": "Technology and the built world",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology"
    },
    {
     "id": "geo.nongeo.t2001.technology.computing",
     "label": "Computing and digital systems",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing"
    },
    {
     "id": "geo.nongeo.t2001.technology.computing.ai",
     "label": "Artificial intelligence and data",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai"
    },
    {
     "id": "geo.nongeo.t2001.technology.computing.ai.ml_computation",
     "label": "Machine learning and neural computation",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_computation"
    },
    {
     "id": "geo.nongeo.t2001.technology.computing.ai.neural_deep",
     "label": "Neural networks and deep learning",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.neural_deep"
    }
   ]
  }
 ],
 "excerpt": "Layer normalization is a deep learning technique that standardizes a neural network layer's activations per example, introduced by Jimmy Lei Ba and Geoffrey Hinton in 2016 and widely used in transformers.",
 "snippet": "Layer normalization is a deep learning technique that standardizes a neural network layer's activations per example, introduced by Jimmy Lei Ba and Geoffrey Hinton in 2016 and widely used in transformers.",
 "node": "technology.computing.ai.neural_deep.alpha-i-to-z",
 "markdown": "# Layer normalization\n\nLayer normalization is a technique in deep learning that standardizes the activations of a neural network layer across its feature dimension for each example individually, stabilizing and speeding up training without using batch statistics. It was introduced by transposing batch normalization: instead of computing the mean and variance from a mini-batch of training cases, it computes them from all of the summed inputs to the neurons in a layer on a single training case.<sup>[1](https://arxiv.org/abs/1607.06450)</sup> This removes any constraint on mini-batch size, so the method works in the pure online regime with batch size 1.<sup>[1](https://arxiv.org/abs/1607.06450)</sup> Recurrent networks benefit the most, especially for long sequences and small mini-batches, and the technique later became a standard component of transformers.<sup>[1](https://arxiv.org/abs/1607.06450)</sup> The Transformer's original design used LayerNorm around each sublayer, and BERT inherited this post-LN pattern, while early Vision Transformers instead place LayerNorm before each sublayer.<sup>[2](https://awesome.papernotes.org/en/era2_deep_renaissance/2016_layer_norm/)</sup>\n\n| Key fact | Detail |\n|---|---|\n| Statistics | Mean and variance over the hidden units of one layer, per training case<sup>[1](https://arxiv.org/abs/1607.06450)</sup> |\n| Batch independence | No mini-batch size constraint; usable with batch size 1<sup>[1](https://arxiv.org/abs/1607.06450)</sup> |\n| Train/test behavior | Identical computation at training and test time<sup>[1](https://arxiv.org/abs/1607.06450)</sup> |\n| Affine parameters | Learnable per-element gain γ and bias β; variance uses the biased estimator<sup>[3](http://docs.pytorch.org/docs/main/generated/torch.nn.LayerNorm.html)</sup> |\n| Placement | Pre-LN (normalization before attention and feed-forward layers) is the preferred choice in modern architectures such as GPT, Llama, and Vision Transformers<sup>[4](https://proceedings.neurips.cc/paper_files/paper/2025/file/6f7d90b1198fec96defd80b5ebd5bc81-Paper-Conference.pdf)</sup> |\n| Main variant | RMSNorm drops mean subtraction and reduces running time by 7%–64% across models<sup>[5](https://proceedings.neurips.cc/paper_files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf)</sup> |\n\n## How it works\n\nFor a layer with H hidden units and summed inputs \\( a_i^{l} \\), layer normalization computes\n\n\\[ \\mu^{l} = \\frac{1}{H} \\sum_{i} a_i^{l}, \\qquad \\sigma^{l} = \\sqrt{\\frac{1}{H} \\sum_{i} \\left( a_i^{l} - \\mu^{l} \\right)^{2}} \\]\n\nover the feature dimension of a single example. All hidden units in a layer share the same normalization terms, but different training cases have different terms.<sup>[1](https://arxiv.org/abs/1607.06450)</sup> An equivalent statement is that for a vector of d feature components, \\( \\mu = (1/d) \\sum x_i \\) and \\( \\sigma = \\sqrt{(1/d) \\sum (x_i - \\mu)^{2}} \\), with statistics calculated for each vector independently; this is the contrast with batch normalization, which relies on statistics from a batch of data points.<sup>[6](https://arxiv.org/html/2409.12951v1)</sup>\n\nGeometrically, the standardization step can be understood in three steps: remove the component of a vector along the uniform vector, normalize the remaining vector, and scale the resultant vector.<sup>[7](https://aclanthology.org/2026.findings-eacl.20/)</sup> Because every normalized activation depends on all the others in the same layer, the normalization terms make the layer invariant to re-scaling all of its summed inputs, which produces much more stable hidden-to-hidden dynamics in recurrent networks and mitigates exploding and vanishing gradients.<sup>[1](https://arxiv.org/abs/1607.06450)</sup>\n\n[Batch normalization](https://www.edgepedia.org/batch-normalization) normalizes each scalar feature independently to zero mean and unit variance, \\( \\widehat{x}^{(k)} = (x^{(k)} - \\mathrm{E}[x^{(k)}]) / \\sqrt{\\mathrm{Var}[x^{(k)}]} \\), then applies a learnable scale and shift \\( y^{(k)} = \\gamma^{(k)} \\widehat{x}^{(k)} + \\beta^{(k)} \\), with statistics computed over the training mini-batch.<sup>[8](https://ar5iv.labs.arxiv.org/html/1502.03167)</sup> In framework terms, batch and instance normalization apply a scalar scale and bias per channel or plane, whereas layer normalization applies a per-element scale and bias, and it uses statistics computed from the input data in both training and evaluation modes.<sup>[3](http://docs.pytorch.org/docs/main/generated/torch.nn.LayerNorm.html)</sup>\n\n## How it is done\n\nA practitioner normalizes the summed inputs of a layer over its feature dimensions, then applies a learnable affine transform. In PyTorch, the mean and standard deviation are calculated over the last D dimensions given by normalized_shape; for normalized_shape (3, 5), statistics are computed over the last 2 dimensions. The parameters γ and β are learnable affine parameters of normalized_shape when elementwise_affine is True, and the variance uses the biased estimator, equivalent to torch.var(input, correction=0).<sup>[3](http://docs.pytorch.org/docs/main/generated/torch.nn.LayerNorm.html)</sup>\n\nIn a layer-normalized recurrent network, each time step gets its own statistics while the affine parameters are shared over time:\n\n\\[ h^{t} = f \\left[ \\frac{g}{\\sigma^{t}} \\odot \\left( a^{t} - \\mu^{t} \\right) + b \\right] \\]\n\nso each neuron has an adaptive bias and gain applied after normalization but before the non-linearity.<sup>[1](https://arxiv.org/abs/1607.06450)</sup> The gain should be learned: in machine-translation [Transformer](https://www.edgepedia.org/transformer) experiments, models with a fixed gain g performed much worse on ar→en, en→he, and en→vi pairs, suggesting that learning g is required to accommodate layer gradients.<sup>[9](https://aclanthology.org/2019.iwslt-1.17.pdf)</sup>\n\n## Origin\n\nLayer normalization was reported by Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton in \"Layer Normalization\" (arXiv, 2016).<sup>[1](https://arxiv.org/abs/1607.06450)</sup> The method was built on batch normalization, introduced by Sergey Ioffe and Christian Szegedy in 2015 as \"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift\".<sup>[8](https://ar5iv.labs.arxiv.org/html/1502.03167)</sup> The transposition was motivated by batch normalization's limits outside large-batch supervised settings: layer normalization performs exactly the same computation at training and test times and can be applied to recurrent neural networks by computing the normalization statistics separately at each time step.<sup>[1](https://arxiv.org/abs/1607.06450)</sup>\n\n## Variants\n\n**RMSNorm.** [Root mean square](https://www.edgepedia.org/root-mean-square) layer normalization hypothesizes that the re-centering invariance in LayerNorm is dispensable, and regularizes the summed inputs to a neuron according to the root mean square only, retaining re-scaling invariance and implicit learning-rate adaptation.<sup>[5](https://proceedings.neurips.cc/paper_files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf)</sup> It was introduced by Biao Zhang and Rico Sennrich in \"Root Mean Square Layer Normalization\" (arXiv, 2019).<sup>[5](https://proceedings.neurips.cc/paper_files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf)</sup> When the mean of the summed inputs is zero, RMSNorm is exactly equal to LayerNorm.<sup>[5](https://proceedings.neurips.cc/paper_files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf)</sup> Partial RMSNorm (pRMSNorm) estimates the RMS from p% of the summed inputs without breaking the re-scaling invariance properties.<sup>[5](https://proceedings.neurips.cc/paper_files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf)</sup>\n\n**LayerNorm-simple.** A version without the bias and gain outperforms standard LayerNorm on four datasets and achieves state-of-the-art performance on En-Vi translation.<sup>[10](https://dl.acm.org/doi/10.5555/3454287.3454681)</sup>\n\n**Weight normalization.** Weight normalization, introduced by Tim Salimans and [Diederik P. Kingma](https://www.edgepedia.org/durk-kingma) (NeurIPS, 2016), reparameterizes weights rather than activations; for a layer with whitened inputs, normalizing pre-activations with batch normalization is equivalent to normalizing the weights with weight normalization, using μ = 0 and σ = ||v||.<sup>[11](https://papers.neurips.cc/paper_files/paper/2016/file/ed265bc903a5a097f61d3ec064d96d2e-Paper.pdf)</sup> It can be viewed as a cheaper and less noisy approximation to batch normalization, and its deterministic nature and independence from the mini-batch input make it applicable where batch statistics are problematic.<sup>[11](https://papers.neurips.cc/paper_files/paper/2016/file/ed265bc903a5a097f61d3ec064d96d2e-Paper.pdf)</sup>\n\n## Applications\n\nLayer normalization is used most prominently in transformers. Post-LN transformers apply normalization after the residual addition and suffer unstable gradient flow; Pre-LN transformers apply normalization before the self-attention and feed-forward layers, written as \\( x' = x + \\mathrm{MHSA}(\\mathrm{LN1}(x)) \\) instead of \\( x' = \\mathrm{LN1}(x + \\mathrm{MHSA}(x)) \\), stabilizing training and achieving faster convergence, which makes Pre-LN the preferred choice in modern architectures such as GPT, Llama, and Vision Transformers.<sup>[4](https://proceedings.neurips.cc/paper_files/paper/2025/file/6f7d90b1198fec96defd80b5ebd5bc81-Paper-Conference.pdf)</sup> Earlier analysis had already shown that placing layer normalization inside the residual blocks yields well-behaved gradients at initialization, motivating removal of the learning-rate warm-up stage for training Pre-LN transformers.<sup>[12](https://proceedings.mlr.press/v119/xiong20b.html)</sup> RMSNorm, a variant of LayerNorm that does not remove the component along the uniform vector, has moved from variant to default and is used to train the latest Llama models.<sup>[6](https://arxiv.org/html/2409.12951v1)</sup>\n\n## Limitations and alternatives\n\nThe affine transformation mechanism carries a risk of over-fitting: LayerNorm achieves lower training loss (or BPC) but higher validation loss than LayerNorm-simple on En-Vi and Enwiki8.<sup>[10](https://dl.acm.org/doi/10.5555/3454287.3454681)</sup> On cost, the efficiency gain from faster and more stable training (in number of training steps) is counter-balanced by an increased computational cost per training step, and LayerNorm's computational overhead becomes severe as networks grow larger and deeper.<sup>[5](https://proceedings.neurips.cc/paper_files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf)</sup> A 2024 analysis adds a structural limitation: layer normalization is irreversible, because information along the uniform vector is lost during normalization and cannot be recovered by the learnable scale-and-shift parameters, so those parameters cannot represent an identity transformation.<sup>[6](https://arxiv.org/html/2409.12951v1)</sup> Among alternatives in transformer NLP, PowerNorm outperforms LayerNorm by 0.4/0.6 BLEU on IWSLT14/WMT14 and 5.6/3.0 PPL on PTB/[WikiText-103](https://www.edgepedia.org/wikitext-103).<sup>[13](https://proceedings.mlr.press/v119/shen20e.html)</sup>\n\n## References\n\n1. [Layer Normalization](https://arxiv.org/abs/1607.06450)\n2. [Layer Normalization - Awesome AI Papers](https://awesome.papernotes.org/en/era2_deep_renaissance/2016_layer_norm/)\n3. [torch.nn.LayerNorm, PyTorch documentation](http://docs.pytorch.org/docs/main/generated/torch.nn.LayerNorm.html)\n4. [Impact of Layer Norm on Memorization and Generalization in Transformers (NeurIPS 2025)](https://proceedings.neurips.cc/paper_files/paper/2025/file/6f7d90b1198fec96defd80b5ebd5bc81-Paper-Conference.pdf)\n5. [Root Mean Square Layer Normalization (NeurIPS 2019)](https://proceedings.neurips.cc/paper_files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf)\n6. [Re-Introducing LayerNorm: Geometric Meaning, Irreversibility and a Comparative Study with RMSNorm (arXiv 2024)](https://arxiv.org/html/2409.12951v1)\n7. [Geometric Interpretation of Layer Normalization and a Comparative Analysis with RMSNorm (EACL 2026 Findings)](https://aclanthology.org/2026.findings-eacl.20/)\n8. [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://ar5iv.labs.arxiv.org/html/1502.03167)\n9. [Transformers without Tears: Improving the Normalization of Self-Attention (IWSLT 2019)](https://aclanthology.org/2019.iwslt-1.17.pdf)\n10. [Understanding and Improving Layer Normalization (NeurIPS 2019)](https://dl.acm.org/doi/10.5555/3454287.3454681)\n11. [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (NeurIPS 2016)](https://papers.neurips.cc/paper_files/paper/2016/file/ed265bc903a5a097f61d3ec064d96d2e-Paper.pdf)\n12. [On Layer Normalization in the Transformer Architecture (ICML 2020)](https://proceedings.mlr.press/v119/xiong20b.html)\n13. [PowerNorm: Rethinking Batch Normalization in Transformers (ICML 2020)](https://proceedings.mlr.press/v119/shen20e.html)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation › Neural networks and deep learning › Titles I to Z*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n",
 "same_as": [],
 "url": "https://www.edgepedia.org/layer-normalization",
 "markdown_url": "https://www.edgepedia.org/layer-normalization.md",
 "license": {
  "name": "Edgepedia Community License 1.0",
  "url": "https://www.edgepedia.org/edgepedia/license",
  "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.",
  "spdx": "LicenseRef-Edgepedia-Community-1.0"
 },
 "credit": "\"Layer normalization\", Edgepedia (EdgeChat), https://www.edgepedia.org/layer-normalization. Edgepedia Community License 1.0.",
 "credit_md": "\"[Layer normalization](https://www.edgepedia.org/layer-normalization)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/layer-normalization](https://www.edgepedia.org/layer-normalization). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).",
 "credit_html": "\"<a href=\"https://www.edgepedia.org/layer-normalization\">Layer normalization</a>\", Edgepedia (EdgeChat), <a href=\"https://www.edgepedia.org/layer-normalization\">https://www.edgepedia.org/layer-normalization</a>. <a href=\"https://www.edgepedia.org/edgepedia/license\">Edgepedia Community License 1.0</a>.",
 "speakable": "Layer normalization is a deep learning technique that standardizes a neural network layer's activations per example, introduced by Jimmy Lei Ba and Geoffrey Hinton in 2016 and widely used in transformers."
}