{ "id": "epcezvn50p", "slug": "self-attention-model", "title": "Self-attention model", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation" }, { "id": "technology.computing.ai.neural_deep", "label": "Neural networks and deep learning", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.neural_deep" }, { "id": "technology.computing.ai.ensembles_boosting.attention_transformer_training", "label": "Attention and transformer training topics", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ensembles_boosting.attention_transformer_training" } ], "geo": [ { "id": "geo.nongeo.t2001.technology.computing.ai.neural_deep", "label": "Non-geographic · 2001 to 2020: Neural networks and deep learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.neural_deep", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology" }, { "id": "geo.nongeo.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing" }, { "id": "geo.nongeo.t2001.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t2001.technology.computing.ai.neural_deep", "label": "Neural networks and deep learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.neural_deep" } ] } ], "excerpt": "Self-attention is a neural network mechanism that computes each sequence element's representation by comparing it with every other element, capturing long-range dependencies without recurrence as the core of the Transformer.", "snippet": "Self-attention is a neural network mechanism that computes each sequence element's representation by comparing it with every other element, capturing long-range dependencies without recurrence as the core of the Transformer.", "node": "technology.computing.ai.ensembles_boosting.attention_transformer_training", "markdown": "# Self-attention model\n\nSelf-attention is a neural network mechanism that computes a representation of each element of a sequence or set by comparing it with every other element and averaging their vectors with weights derived from those comparisons. To compute a new representation for a word, the model compares it to every other word in the sentence and uses the resulting scores as weights in a weighted average of all words' representations.[1](https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/) It captures long-range dependencies without recurrence or convolution, and it is the computational core of the [Transformer](https://www.edgepedia.org/transformer), whose authors describe it as the first transduction model relying entirely on self-attention to compute representations of its input and output without sequence-aligned RNNs or convolution.[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) [Attention](https://www.edgepedia.org/attention) was originally introduced as an extension to recurrent networks; the Transformer demonstrated that the attention mechanism alone suffices to build a state-of-the-art model.[3](https://ar5iv.labs.arxiv.org/html/2203.14263)\n\n| Property | Value |\n|---|---|\n| Core computation | Weighted average of value vectors; weights are softmaxed, scaled dot products of queries with keys[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) |\n| Formula | \\( \\mathrm{Attention}(Q,K,V) = \\mathrm{softmax}\\!\\left( \\frac{Q \\cdot K^{\\top}}{\\sqrt{d_{k}}} \\right) V \\)[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) |\n| Per-layer complexity | \\( O(n^{2} \\cdot d) \\) time with \\( O(1) \\) sequential operations, versus \\( O(n \\cdot d^{2}) \\) and \\( O(n) \\) sequential operations for recurrence[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) |\n| Original configuration | \\( h = 8 \\) heads, \\( d_{k} = d_{v} = 64 \\), \\( d_{\\mathrm{model}} = 512 \\), stacks of \\( N = 6 \\) layers[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) |\n| Memory with FlashAttention | Linear in sequence length, up to 20× more memory-efficient than exact attention baselines[4](https://doi.org/10.48550/arxiv.2205.14135) |\n| Nearest linear-time alternative | Mamba: 5× higher inference throughput than Transformers and linear scaling in sequence length[5](https://doi.org/10.48550/arxiv.2312.00752) |\n| KV cache at long context | On the order of 40 GB of GPU memory for a 70B model at 200K context[6](https://neurals.ca/deep-dive/algorithms/attention/) |\n\n## How it works\n\nSelf-attention relates different positions of a single sequence to compute a representation of the sequence.[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) Each input element \\( x_{i} \\) is projected into a query \\( q_{i} = Q \\cdot x_{i} \\), a key \\( k_{j} = K \\cdot x_{j} \\), and a value \\( v_{j} = V \\cdot x_{j} \\). The output for position \\( i \\) is a weighted sum of the values, with softmax weights \\( \\alpha_{ij} = \\exp(q_{i}^{\\top} k_{j} / \\sqrt{d_{k}}) / \\sum_{j'} \\exp(q_{i}^{\\top} k_{j'} / \\sqrt{d_{k}}) \\), summed over unmasked positions.[7](https://web.stanford.edu/class/cs224n/readings/cs224n-self-attention-transformers-2023_draft.pdf) Because every position attends to every other position in one step, a self-attention layer connects all positions with a constant number of sequentially executed operations, which is what lets it model long-range dependencies that recurrent layers accumulate over \\( O(n) \\) steps.[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\n\n**Why divide by \\( \\sqrt{d_{k}} \\)?** For large \\( d_{k} \\) the dot products grow large in magnitude, pushing the softmax into regions with extremely small gradients; scaling by \\( 1/\\sqrt{d_{k}} \\) counteracts this.[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) A variance argument makes the growth concrete: if the components of \\( q \\) and \\( k \\) are independent with mean 0 and variance 1, the dot product \\( q \\cdot k = \\sum_{i=1}^{d_{k}} q_{i} \\cdot k_{i} \\) has mean 0 and variance \\( d_{k} \\).[8](https://nlp.seas.harvard.edu/2018/04/01/attention.html)\n\n**Multi-head attention** runs \\( h \\) attention functions in parallel on learned linear projections of the inputs, then concatenates: \\( \\mathrm{MultiHead}(Q,K,V) = \\mathrm{Concat}(\\mathrm{head}_{1}, \\ldots, \\mathrm{head}_{h}) \\cdot W^{O} \\), where \\( \\mathrm{head}_{i} = \\mathrm{Attention}(Q \\cdot W_{i}^{Q}, K \\cdot W_{i}^{K}, V \\cdot W_{i}^{V}) \\). This lets the model attend to different representation subspaces.[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) Because each head operates at dimension \\( d_{\\mathrm{model}}/h \\), the total computational cost is similar to single-head attention at full dimensionality.[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\n\n## How it is done\n\nA practitioner runs five steps: project inputs to queries, keys, and values; compute the \\( n \\times n \\) score matrix \\( Q \\cdot K^{\\top} / \\sqrt{d_{k}} \\); apply a mask; softmax the scores; and multiply by the values.\n\nMasking details matter numerically. The decoder sets softmax inputs for illegal (future) connections to \\( -\\infty \\) to preserve the autoregressive property.[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) In practice a large finite negative constant is often used instead, because infinity can produce NaNs in float16 and library behavior with infinite inputs is not uniformly defined; a large enough negative constant still sets the attention weight to exactly zero in finite precision.[7](https://web.stanford.edu/class/cs224n/readings/cs224n-self-attention-transformers-2023_draft.pdf) The original encoder stacks \\( N = 6 \\) identical layers, each pairing multi-head self-attention with a position-wise feed-forward network, residual connections, and layer normalization; the decoder adds a third sub-layer attending over the encoder output.[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\n\n## Origin\n\nModern attention is commonly traced to Bahdanau, Cho, and Bengio's 2014 machine translation model, which used attention to address structural issues of recurrent neural networks; the same slides credit multiplicative (bilinear) attention to Luong, Pham, and Manning in 2015.[3](https://ar5iv.labs.arxiv.org/html/2203.14263) • [9](https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1244/slides/cs224n-2024-lecture08-transformers.pdf) Self-attention, sometimes called intra-attention, had already been used in reading comprehension, abstractive summarization, textual entailment, and task-independent sentence representations before 2017.[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) A related step was the decomposable attention model of Parikh and colleagues, reported in 2016 on arXiv, which applied attention-style comparison to feedforward networks for natural language inference.[10](https://doi.org/10.48550/arxiv.1606.01933)\n\nThe paper \"Attention Is All You Need\" discarded recurrence entirely.[1](https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/) It names ByteNet, ConvS2S, and the Extended Neural GPU as convolution-based precursors whose operations to relate two positions grow with distance (logarithmically for ByteNet, linearly for ConvS2S).[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\n\n## Variants\n\n**Sparse attention** computes a limited selection of the pairwise similarity scores instead of all of them; named methods include Sparse Transformers, Longformers, Routing Transformers, Reformers, and [Big Bird](https://www.edgepedia.org/big-bird).[11](https://research.google/blog/rethinking-attention-with-performers/)\n\n**Linear and kernelized attention** approximates the softmax. The Performer uses the FAVOR+ algorithm (Fast Attention Via Positive Orthogonal Random Features) for scalable, low-variance, unbiased estimation with linear time and space.[11](https://research.google/blog/rethinking-attention-with-performers/)\n\n**IO-aware exact kernels** keep the mathematics unchanged and reorganize the computation. FlashAttention, reported by Dao and colleagues in 2022 on arXiv, requires \\( \\Theta(N^{2} d^{2} M^{-1}) \\) HBM accesses versus \\( \\Theta(Nd + N^{2}) \\) for standard attention, making memory linear in sequence length and up to 20× more memory-efficient.[4](https://doi.org/10.48550/arxiv.2205.14135) FlashAttention-2, reported by Dao in 2023 on arXiv, improves parallelism and work partitioning.[12](https://doi.org/10.48550/arxiv.2307.08691)\n\n**KV-cache-reducing heads.** [Grouped-query attention](https://www.edgepedia.org/grouped-query-attention), reported by Ainslie and colleagues in 2023 on arXiv, shares keys and values across multiple query heads to shrink the [KV cache](https://www.edgepedia.org/kv-cache); multi-query attention is named alongside it in production model families such as Llama-3, Gemma, and Mistral.[13](https://doi.org/10.48550/arxiv.2305.13245) • [6](https://neurals.ca/deep-dive/algorithms/attention/)\n\n## Applications\n\nModern neural attention was popularized in machine translation, while early self-attention appeared in tasks such as reading comprehension and sentence representation learning, and the mechanism later spread to image processing, video processing, and recommender systems.[3](https://ar5iv.labs.arxiv.org/html/2203.14263) In vision, replacing all spatial convolutions in a ResNet with local self-attention outperforms the baseline on ImageNet classification with 12% fewer FLOPS and 29% fewer parameters, and on COCO object detection a pure self-attention model matches RetinaNet's mAP with 39% fewer FLOPS and 34% fewer parameters.[14](https://proceedings.neurips.cc/paper/2019/file/3416a75f4cea9109507cacd8e2f2aefc-Paper.pdf)\n\n**Architecture families** use the mechanism differently. Decoder-only autoregressive language models include GPT-2, GPT-3, and BLOOM, a 176B-parameter open-access multilingual model.[7](https://web.stanford.edu/class/cs224n/readings/cs224n-self-attention-transformers-2023_draft.pdf) • [15](https://doi.org/10.4230/oasics.commit2data.3) Encoder-decoder models such as T5 can outperform decoder-only models at modest scale, and cross-attention connects the two by drawing keys and values from encoder outputs with queries from the decoder.[7](https://web.stanford.edu/class/cs224n/readings/cs224n-self-attention-transformers-2023_draft.pdf) • [9](https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1244/slides/cs224n-2024-lecture08-transformers.pdf)\n\n## Limitations and alternatives\n\nStandard attention requires quadratic computation time and quadratic memory to compute and store all pairwise similarity scores.[11](https://research.google/blog/rethinking-attention-with-performers/) This cost is not incidental: the time complexity of dense dot-product self-attention is necessarily quadratic in input length unless the Strong Exponential Time Hypothesis (SETH) is false, even approximately; sparse, windowed, and kernelized methods change the computation and can have subquadratic costs.[16](https://ar5iv.labs.arxiv.org/html/2209.04881) The root cause is that attention does not compress context: autoregressive inference must store an entire KV cache that grows with context length, while dense attention has quadratic sequence-length computation during training independently of the inference cache.[5](https://doi.org/10.48550/arxiv.2312.00752) The cache scales as \\( 2 \\cdot n \\cdot \\mathrm{layers} \\cdot \\mathrm{heads} \\cdot d_{\\mathrm{head}} \\), on the order of 40 GB for a 70B model at 200K context.[6](https://neurals.ca/deep-dive/algorithms/attention/)\n\n**Positional information** is a structural weakness. Because the model contains no recurrence or convolution, information about token position must be injected, in the original design as sinusoidal positional encodings added to the input embeddings.[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\n\n**Against recurrence and state-space models.** Recurrent layers need \\( O(n) \\) sequential operations and \\( O(n \\cdot d^{2}) \\) compute per layer, while self-attention needs \\( O(1) \\) sequential operations, and the Transformer sped up training by up to an order of magnitude on parallel hardware.[2](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) • [1](https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/) Mamba, reported by Gu and Dao in 2023 on arXiv, makes its parameters functions of the input, allowing selective propagation or forgetting along the sequence; it achieves 5× higher inference throughput than [Transformers](https://www.edgepedia.org/transformers) with linear scaling, and Mamba-3B outperforms same-size Transformers while matching Transformers twice its size.[5](https://doi.org/10.48550/arxiv.2312.00752) Despite these challengers, in practice almost no large Transformer language models use anything but quadratic-cost attention, because cheaper methods tend not to work as well at scale.[9](https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1244/slides/cs224n-2024-lecture08-transformers.pdf)\n\n## References\n\n1. [Transformer: A Novel Neural Network Architecture for Language Understanding (Google Research blog, Aug 31, 2017)](https://research.google/blog/transformer-a-novel-neural-network-architecture-for-language-understanding/)\n2. [Attention Is All You Need (NeurIPS 2017 proceedings)](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\n3. [A General Survey on Attention Mechanisms in Deep Learning](https://ar5iv.labs.arxiv.org/html/2203.14263)\n4. [Dao, Tri and colleagues (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2205.14135)\n5. [Gu, Albert, Dao, Tri (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2312.00752)\n6. [Attention & Self-Attention, Tutorial (neurals.ca)](https://neurals.ca/deep-dive/algorithms/attention/)\n7. [CS 224n Note 10: Self-Attention & Transformers (Stanford, 2023)](https://web.stanford.edu/class/cs224n/readings/cs224n-self-attention-transformers-2023_draft.pdf)\n8. [The Annotated Transformer (Harvard NLP)](https://nlp.seas.harvard.edu/2018/04/01/attention.html)\n9. [CS224N Lecture 8 slides: Transformers (Stanford, 2024)](https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1244/slides/cs224n-2024-lecture08-transformers.pdf)\n10. [Parikh, Ankur P. and colleagues (2016). A Decomposable Attention Model for Natural Language Inference. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1606.01933)\n11. [Rethinking Attention with Performers (Google Research blog)](https://research.google/blog/rethinking-attention-with-performers/)\n12. [Dao, Tri (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2307.08691)\n13. [Ainslie, Joshua and colleagues (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2305.13245)\n14. [Stand-Alone Self-Attention in Vision Models (NeurIPS 2019)](https://proceedings.neurips.cc/paper/2019/file/3416a75f4cea9109507cacd8e2f2aefc-Paper.pdf)\n15. [Workshop, BigScience and colleagues (2022). BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. arXiv (Cornell University).](https://doi.org/10.4230/oasics.commit2data.3)\n16. [On the Computational Complexity of Self-Attention](https://ar5iv.labs.arxiv.org/html/2209.04881)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation › Neural networks and deep learning › Attention and transformer training topics*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/self-attention-model", "markdown_url": "https://www.edgepedia.org/self-attention-model.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Self-attention model\", Edgepedia (EdgeChat), https://www.edgepedia.org/self-attention-model. Edgepedia Community License 1.0.", "credit_md": "\"[Self-attention model](https://www.edgepedia.org/self-attention-model)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/self-attention-model](https://www.edgepedia.org/self-attention-model). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Self-attention model\", Edgepedia (EdgeChat), https://www.edgepedia.org/self-attention-model. Edgepedia Community License 1.0.", "speakable": "Self-attention is a neural network mechanism that computes each sequence element's representation by comparing it with every other element, capturing long-range dependencies without recurrence as the core of the Transformer." }