{ "id": "epnk41eej3", "slug": "graph-attention-network", "title": "Graph attention network", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation" }, { "id": "technology.computing.ai.neural_deep", "label": "Neural networks and deep learning", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.neural_deep" }, { "id": "technology.computing.ai.nn_architectures", "label": "Neural network architectures", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.nn_architectures" }, { "id": "technology.computing.ai.nn_architectures.graph_networks", "label": "Graph neural network architectures", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.nn_architectures.graph_networks" } ], "geo": [ { "id": "geo.nongeo.t2001.technology.computing.ai.neural_deep", "label": "Non-geographic · 2001 to 2020: Neural networks and deep learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.neural_deep", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology" }, { "id": "geo.nongeo.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing" }, { "id": "geo.nongeo.t2001.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t2001.technology.computing.ai.neural_deep", "label": "Neural networks and deep learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.neural_deep" } ] } ], "excerpt": "A graph attention network (GAT) is a neural network architecture for machine learning on graphs that weights neighbor features with learned attention, introduced at ICLR 2018.", "snippet": "A graph attention network (GAT) is a neural network architecture for machine learning on graphs that weights neighbor features with learned attention, introduced at ICLR 2018.", "node": "technology.computing.ai.nn_architectures.graph_networks", "markdown": "# Graph attention network\n\nA graph attention network (GAT) is a neural network architecture for machine learning on graphs that computes a node's representation as a weighted combination of its neighbors' features, with the weights produced by an attention mechanism rather than fixed normalization. It is used for node classification, link prediction, and graph-level prediction tasks.\n\n| Key fact | Detail |\n|---|---|\n| Introduced by | Veličković, Cucurull, Casanova, Romero, Liò, and Bengio, ICLR 2018 (arXiv 2017) [1](https://arxiv.org/abs/1710.10903) |\n| Core operation | Masked self-attention over each node's neighborhood, softmax-normalized [1](https://arxiv.org/abs/1710.10903) |\n| Complexity per head | \\( O(|V|F \\cdot F' + |E|F') \\), on par with graph convolutional networks [1](https://arxiv.org/abs/1710.10903) |\n| Transductive accuracy | 83.0 ± 0.7% (Cora), 72.5 ± 0.7% (Citeseer), 79.0 ± 0.3% (Pubmed) [1](https://arxiv.org/abs/1710.10903) |\n| Inductive PPI micro-F1 | 0.975 ± 0.006 (DGL implementation) vs 0.509 ± 0.025 for GCN [2](https://docs.dgl.ai/en/0.4.x/tutorials/models/1_gnn/9_gat.html) |\n| Main limitation | Static attention: the ranking of attention scores does not depend on the query node [3](https://arxiv.org/abs/2105.14491v3) |\n| Main fix | GATv2, strictly more expressive dynamic attention at the same time complexity [3](https://arxiv.org/abs/2105.14491v3) |\n\n## How it works\n\nEach GAT layer transforms node features with a shared linear map \\( W \\), then computes an un-normalized attention score for every edge between node \\( i \\) and neighbor \\( j \\):\n\n\\[ e_{ij} = \\mathrm{LeakyReLU}\\left( \\vec{a}^{\\top} [ W \\cdot h_i \\| W \\cdot h_j ] \\right) \\]\n\nwhere \\( \\vec{a} \\in \\mathbb{R}^{2F'} \\) is a learned weight vector, \\( \\| \\) denotes concatenation, and the nonlinearity is LeakyReLU with negative slope 0.2.[1](https://arxiv.org/abs/1710.10903) • [4](https://ar5iv.labs.arxiv.org/html/2206.02849) Scores are normalized with a softmax over the neighborhood,\n\n\\[ \\alpha_{ij} = \\frac{\\exp(e_{ij})}{\\sum_{k \\in \\mathcal{N}_i} \\exp(e_{ik})} \\]\n\nand the new representation aggregates the transformed neighbor features:\n\n\\[ h_i' = \\sigma\\left( \\sum_{j \\in \\mathcal{N}_i} \\alpha_{ij} \\, W \\cdot h_j \\right) \\]\n\n[1](https://arxiv.org/abs/1710.10903) • [5](https://petar-v.com/GAT/) This is additive attention, contrasted with the dot-product attention of the [Transformer](https://www.edgepedia.org/transformer).[2](https://docs.dgl.ai/en/0.4.x/tutorials/models/1_gnn/9_gat.html) • [6](https://link.springer.com/article/10.1186/s40537-023-00876-4) The layer is \"masked\" because scores are computed only for edges that exist, so the model never sees the full graph structure upfront.[1](https://arxiv.org/abs/1710.10903)\n\nStability comes from multi-head attention: \\( K \\) independent attention mechanisms run the transformation and their outputs are concatenated in intermediate layers, while the final layer averages the heads.[1](https://arxiv.org/abs/1710.10903) • [7](https://www.mdpi.com/1999-5903/16/9/318) PyTorch Geometric's GATConv adds self-loops by default, supports optional edge features in the attention logits, and can return the per-edge attention weights [8](https://pytorch-geometric.readthedocs.io/en/stable/generated/torch_geometric.nn.conv.GATConv.html); Both PyTorch Geometric's and DGL's GATConv support attention dropout and residual connections (PyG via 'dropout' on the normalized attention coefficients and 'residual'; DGL via 'attn_drop' and 'residual'), and DGL raises an error on zero-in-degree nodes unless configured otherwise.[9](https://docs.dgl.ai/generated/dgl.nn.pytorch.conv.GATConv.html)\n\nThe mathematical difference from a graph convolutional network (GCN) is the weighting rule. GCNs scale neighbor contributions by a non-parametric, degree-based normalization coefficient, whereas GATs learn the scaling factors through attention.[7](https://www.mdpi.com/1999-5903/16/9/318) • [2](https://docs.dgl.ai/en/0.4.x/tutorials/models/1_gnn/9_gat.html)\n\n## How it is done\n\nThe original transductive model stacks two GAT layers with \\( K = 8 \\) heads of 8 features each, applies dropout \\( p = 0.6 \\) on input features and normalized attention coefficients, and uses \\( L^2 \\) regularization with \\( \\lambda = 0.0005 \\).[1](https://arxiv.org/abs/1710.10903) The inductive protein-protein interaction (PPI) model uses three layers with skip connections, \\( K = 4 \\) heads of 256 features in the first two layers and \\( K = 6 \\) heads of 121 features in the final layer, trained with batch size 2 graphs.[1](https://arxiv.org/abs/1710.10903) Both models use Glorot initialization, the [Adam optimizer](https://www.edgepedia.org/adam-optimizer) at learning rate 0.005, cross-entropy loss on training nodes, and early stopping with patience of 100 epochs.[1](https://arxiv.org/abs/1710.10903)\n\nTwo practical caveats matter. First, GPU frameworks parallelize softmax efficiently only for same-sized neighborhoods, so the original implementation used the masked approach with a \\( -\\infty \\) bias, which raises memory to \\( O(V) \\) intermediates; a sparse-matrix version reduces storage to linear in nodes and edges but the framework used supports only rank-2 sparse multiplication, limiting batching.[1](https://arxiv.org/abs/1710.10903) Second, nodes with zero in-degree produce invalid outputs unless the layer is configured to allow them.[9](https://docs.dgl.ai/generated/dgl.nn.pytorch.conv.GATConv.html)\n\n## Origin\n\nGraph attention networks were reported in \"Graph Attention Networks\", published at ICLR.[1](https://arxiv.org/abs/1710.10903) • [5](https://petar-v.com/GAT/) The paper builds on earlier graph neural networks, which Scarselli, Gori, Tsoi, Hagenbuchner, and Monfardini analyzed in IEEE Transactions on Neural Networks in 2008 [10](https://doi.org/10.1109/tnn.2008.2005141), and on graph convolutional networks introduced by Thomas N. Kipf and [Max Welling](https://www.edgepedia.org/max-welling) in 2016.[11](https://doi.org/10.48550/arxiv.1609.02907) The attentional setup follows the additive attention used in recurrent sequence models, and the multi-head scheme follows the Transformer's multi-head attention.[1](https://arxiv.org/abs/1710.10903)\n\n## Variants\n\nNamed variants differ mainly in how attention is computed or what structure it attends over:\n\n- **GATv2**, by Shaked Brody, Uri Alon, and Eran Yahav (posted 2021, published at ICLR 2022), reorders the operations so the learned vector \\( a \\) is applied after the LeakyReLU nonlinearity, making the scoring an MLP per query-key pair rather than a collapsible linear layer.[3](https://arxiv.org/abs/2105.14491v3) It outperforms GAT across 12 benchmarks of node-, link-, and graph-prediction at matched parametric cost and the same time complexity [3](https://arxiv.org/abs/2105.14491v3), and is available in PyTorch Geometric, DGL, and TensorFlow GNN.[12](https://github.com/tech-srl/how_attentive_are_gats)\n- **GaAN**, by Jiani Zhang and colleagues (2018), inserts gating mechanisms into the multi-head system so different heads contribute different values, and adopts key-value dot-product attention.[13](https://doi.org/10.48550/arxiv.1803.07294) • [4](https://ar5iv.labs.arxiv.org/html/2206.02849)\n- **HAN**, by [Xiao Wang](https://www.edgepedia.org/xiao-wang) and colleagues (2019), handles heterogeneous graphs with hierarchical node-level attention over meta-path based neighbors and semantic-level attention over meta-paths.[14](https://doi.org/10.48550/arxiv.1903.07293)\n- **Signed graph attention networks**, by Junjie Huang and colleagues (2019), and **relational graph attention networks**, by Dan Busbridge and colleagues (2019), adapt attention to signed edges and typed relations respectively.[15](https://doi.org/10.48550/arxiv.1906.10958) • [16](https://doi.org/10.48550/arxiv.1904.05811)\n- **Graph transformer networks**, by Seongjun Yun and colleagues (2019), and **DeepInf**, by Jiezhong Qiu and colleagues (2018), apply attention-style architectures to meta-path graph construction and social influence prediction.[17](https://doi.org/10.48550/arxiv.1911.06455) • [18](https://doi.org/10.48550/arxiv.1807.05560)\n- Post-2023 work includes **PD-GATv2**, a residual edge-weighted GATv2 for directed weighted graphs.[19](https://link.springer.com/article/10.1007/s10489-024-05432-y)\n\n## Applications\n\nOn the citation benchmarks, the original paper reports 83.0 ± 0.7% on Cora, 72.5 ± 0.7% on Citeseer, and 79.0 ± 0.3% on Pubmed, improving on GCN by 1.5% and 1.6% on Cora and Citeseer.[1](https://arxiv.org/abs/1710.10903) On the inductive PPI task, GAT reaches micro-F1 0.975 ± 0.006 versus 0.509 ± 0.025 for GCN.[2](https://docs.dgl.ai/en/0.4.x/tutorials/models/1_gnn/9_gat.html)\n\nFor link prediction with 85/5/10 edge splits, GAT achieves AUC of 0.9027 ± 0.0023 on Cora, 0.9084 ± 0.0035 on CiteSeer, 0.9179 ± 0.0002 on PubMed, and 0.9293 ± 0.0014 on Wiki-CS, performing similarly to GCN, GraphSAGE, and VGAE.[20](https://ar5iv.labs.arxiv.org/html/2102.12557) On ogbn-proteins, GATv2 reaches 80.63 ± 0.70 ROC-AUC versus 78.63 for 8-head GAT.[3](https://arxiv.org/abs/2105.14491v3) With modern tuning (residual connections, normalization, dropout), GAT* reaches 84.46 ± 0.55% on Cora, 72.22 ± 0.84% on Citeseer, 80.28 ± 0.64% on Pubmed, and 94.09 ± 0.37% on ogbn-arxiv.[21](https://www.proceedings.com/content/079/079017-3098open.pdf)\n\n## Limitations and alternatives\n\n**Static attention.** Brody, Alon, and Yahav prove that GAT computes static attention: the ranking of attention scores is unconditioned on the query node, because \\( W \\) and \\( a \\) applied consecutively collapse into a single linear layer.[3](https://arxiv.org/abs/2105.14491v3) GATv2 restores dynamic attention and is more robust to edge noise, since noisy edges receive decaying attention [3](https://arxiv.org/abs/2105.14491v3); the ESA authors note this comes at the price of doubling the parameter count and memory consumption.[22](https://www.nature.com/articles/s41467-025-60252-z)\n\n**Uniform attention and the hard regime.** [Attention](https://www.edgepedia.org/attention) distributions learned by GAT are near uniform across heads and layers on Cora, Citeseer, and Pubmed, becoming more concentrated only on PPI.[23](https://rlgm.github.io/papers/62.pdf) Theoretical work via contextual stochastic block models shows that in the \"hard regime\", where class-mean distance in feature space is small relative to the standard deviation, every attention architecture fails to distinguish inter-class from intra-class edges with high probability and most attention coefficients become uniform, while in the easy regime attention maintains the weights of important edges.[24](https://jmlr.org/papers/volume24/22-125/22-125.pdf)\n\n**Over-smoothing.** A NeurIPS 2023 analysis shows the graph attention mechanism cannot prevent over-smoothing and loses expressive power exponentially, though potentially at a slower rate than GCNs, based on 128-layer single-head experiments.[25](https://proceedings.neurips.cc/paper_files/paper/2023/file/6e4cdfdd909ea4e34bfc85a12774cba0-Paper-Conference.pdf) Mitigations include skip connections, normalization layers, and Jumping Knowledge aggregation.[26](https://dl.acm.org/doi/full/10.1145/3816725)\n\n**Heterophily and when attention helps.** Attentional networks tend to outperform convolutional ones on low-homophily (heterophilous) graphs, at the cost of fewer parameters and lower expressiveness than full message-passing networks.[26](https://dl.acm.org/doi/full/10.1145/3816725) A 2025 ICML analysis via contextual stochastic block models finds graph attention improves classification when structure noise exceeds feature noise, whereas simpler graph convolutions are more effective when feature noise predominates.[27](https://proceedings.mlr.press/v267/ma25w.html)\n\n**Cost and alternatives.** A single GAT head costs \\( O(|V|F \\cdot F' + |E|F') \\), on par with GCN, with multi-head attention multiplying parameters and storage by \\( K \\).[1](https://arxiv.org/abs/1710.10903) In timing experiments reported in 2025, GCN is usually the fastest method while GAT and GATv2 are among the slowest.[22](https://www.nature.com/articles/s41467-025-60252-z) Against graph transformers, a 2024 reassessment found that tuned classic GNNs (GCN, [GraphSAGE](https://www.edgepedia.org/graphsage), GAT) rank first on 17 of 18 node-classification datasets, with tuned GAT* improving ogbn-proteins by an absolute 12.99% and surpassing SGFormer by 5.09%.[21](https://www.proceedings.com/content/079/079017-3098open.pdf) • [28](https://proceedings.neurips.cc/paper_files/paper/2024/file/b10ed15ff1aa864f1be3a75f1ffc021b-Paper-Datasets_and_Benchmarks_Track.pdf) The 2025 Edge-Set Attention architecture, which treats graphs as edge sets and interleaves masked and vanilla self-attention, outperforms fine-tuned message-passing baselines and recent graph transformers on graph-level tasks, though its node-level variant does not beat GCN on Cora accuracy.[22](https://www.nature.com/articles/s41467-025-60252-z)\n\n## References\n\n1. [Graph Attention Networks (Veličković et al., ICLR 2018; arXiv:1710.10903)](https://arxiv.org/abs/1710.10903)\n2. [DGL GAT tutorial](https://docs.dgl.ai/en/0.4.x/tutorials/models/1_gnn/9_gat.html)\n3. [How Attentive are Graph Attention Networks? (Brody, Alon, Yahav; GATv2; excerpts from the arXiv PDF merged here)](https://arxiv.org/abs/2105.14491v3)\n4. [A Bird's-Eye Tutorial of Graph Attention Architectures](https://ar5iv.labs.arxiv.org/html/2206.02849)\n5. [Graph Attention Networks, official author page (Petar Veličković)](https://petar-v.com/GAT/)\n6. [A review of graph neural networks (Journal of Big Data)](https://link.springer.com/article/10.1186/s40537-023-00876-4)\n7. [Graph Attention Networks: A Comprehensive Review of Methods and Applications (Future Internet, MDPI, 2024)](https://www.mdpi.com/1999-5903/16/9/318)\n8. [torch_geometric.nn.conv.GATConv documentation](https://pytorch-geometric.readthedocs.io/en/stable/generated/torch_geometric.nn.conv.GATConv.html)\n9. [DGL GATConv API documentation](https://docs.dgl.ai/generated/dgl.nn.pytorch.conv.GATConv.html)\n10. [F. Scarselli and colleagues (2008). Computational Capabilities of Graph Neural Networks. IEEE Transactions on Neural Networks.](https://doi.org/10.1109/tnn.2008.2005141)\n11. [Kipf, Thomas N., Welling, Max (2016). Semi-Supervised Classification with Graph Convolutional Networks. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1609.02907)\n12. [Official GATv2 implementation repository (tech-srl)](https://github.com/tech-srl/how_attentive_are_gats)\n13. [Zhang, Jiani and colleagues (2018). GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1803.07294)\n14. [Wang, Xiao and colleagues (2019). Heterogeneous Graph Attention Network. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1903.07293)\n15. [Huang, Junjie and colleagues (2019). Signed Graph Attention Networks. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1906.10958)\n16. [Busbridge, Dan and colleagues (2019). Relational Graph Attention Networks. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1904.05811)\n17. [Yun, Seongjun and colleagues (2019). Graph Transformer Networks. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1911.06455)\n18. [Qiu, Jiezhong and colleagues (2018). DeepInf: Social Influence Prediction with Deep Learning. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.1807.05560)\n19. [PD-GATv2: positive difference second generation graph attention network (Applied Intelligence, 2024)](https://link.springer.com/article/10.1007/s10489-024-05432-y)\n20. [Benchmarking Graph Neural Networks on Link Prediction](https://ar5iv.labs.arxiv.org/html/2102.12557)\n21. [Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification (NeurIPS proceedings)](https://www.proceedings.com/content/079/079017-3098open.pdf)\n22. [An end-to-end attention-based approach for learning on graphs (Edge-Set Attention; Nature Communications, 2025; arXiv-version excerpts merged here)](https://www.nature.com/articles/s41467-025-60252-z)\n23. [Statistical Characterization of Attentions in Graph Neural Networks (ICLR 2019 workshop)](https://rlgm.github.io/papers/62.pdf)\n24. [Graph Attention Retrospective (JMLR)](https://jmlr.org/papers/volume24/22-125/22-125.pdf)\n25. [Demystifying Oversmoothing in Attention-Based Graph Neural Networks (NeurIPS 2023)](https://proceedings.neurips.cc/paper_files/paper/2023/file/6e4cdfdd909ea4e34bfc85a12774cba0-Paper-Conference.pdf)\n26. [Introduction to Graph Neural Networks for Machine Learning Engineers (ACM Computing Surveys)](https://dl.acm.org/doi/full/10.1145/3816725)\n27. [Graph Attention is Not Always Beneficial: A Theoretical Analysis via Contextual Stochastic Block Models (ICML 2025, PMLR v267)](https://proceedings.mlr.press/v267/ma25w.html)\n28. [Revisiting the Power of GNNs for Node Classification (NeurIPS 2024 Datasets & Benchmarks)](https://proceedings.neurips.cc/paper_files/paper/2024/file/b10ed15ff1aa864f1be3a75f1ffc021b-Paper-Datasets_and_Benchmarks_Track.pdf)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation › Neural networks and deep learning › Neural network architectures › Graph neural network architectures*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/graph-attention-network", "markdown_url": "https://www.edgepedia.org/graph-attention-network.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Graph attention network\", Edgepedia (EdgeChat), https://www.edgepedia.org/graph-attention-network. Edgepedia Community License 1.0.", "credit_md": "\"[Graph attention network](https://www.edgepedia.org/graph-attention-network)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/graph-attention-network](https://www.edgepedia.org/graph-attention-network). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Graph attention network\", Edgepedia (EdgeChat), https://www.edgepedia.org/graph-attention-network. Edgepedia Community License 1.0.", "speakable": "A graph attention network is a neural network architecture for machine learning on graphs that weights neighbor features with learned attention, introduced at ICLR 2018." }