{ "id": "ep1p8gbk91", "slug": "modular-neural-network", "title": "Modular neural network", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation" }, { "id": "technology.computing.ai.neural_deep", "label": "Neural networks and deep learning", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.neural_deep" }, { "id": "technology.computing.ai.neural_deep.alpha-i-to-z", "label": "Titles I to Z", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.neural_deep.alpha-i-to-z" } ], "geo": [ { "id": "geo.nongeo.t1946.technology.computing.ai.neural_deep", "label": "Non-geographic · 1946 to 2000: Neural networks and deep learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.neural_deep", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1946", "label": "Non-geographic · 1946 to 2000", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946" }, { "id": "geo.nongeo.t1946.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology" }, { "id": "geo.nongeo.t1946.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing" }, { "id": "geo.nongeo.t1946.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai" }, { "id": "geo.nongeo.t1946.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t1946.technology.computing.ai.neural_deep", "label": "Neural networks and deep learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.technology.computing.ai.neural_deep" } ] } ], "excerpt": "A modular neural network is a machine learning architecture that decomposes computation into specialized modules, combined by a routing or gating mechanism to solve the overall task.", "snippet": "A modular neural network is a machine learning architecture that decomposes computation into specialized modules, combined by a routing or gating mechanism to solve the overall task.", "node": "technology.computing.ai.neural_deep.alpha-i-to-z", "markdown": "# Modular neural network\n\nA modular neural network is a neural network architecture in machine learning that decomposes computation into separate specialized modules, whose outputs are combined by a routing or gating mechanism to solve the overall task. In the general sense, modularity is the property of an entity whereby it can be broken down into a number of sub-entities, called modules, with properties such as functional specialization, reusability, and replaceability.[1](https://ar5iv.labs.arxiv.org/html/2310.01154) In neural networks, modular methods involve three ingredients: modules that can be updated locally without affecting the rest of the parameters, a routing function that chooses a subset of modules per example or task, and an aggregation function that combines the outputs of the active modules.[2](https://arxiv.org/html/2302.11529) The mixture-of-experts architecture is one instance, in which expert networks compete to learn training patterns and a gating network mediates the competition.[3](https://proceedings.neurips.cc/paper/1990/file/f74909ace68e51891440e4da0b65a70c-Paper.pdf)\n\n| Key fact | Detail |\n|---|---|\n| Defining ingredients | Modules, a routing function, and an aggregation function[2](https://arxiv.org/html/2302.11529) |\n| Classic combination rule | Gating network with softmax outputs: nonnegative activations summing to one[3](https://proceedings.neurips.cc/paper/1990/file/f74909ace68e51891440e4da0b65a70c-Paper.pdf) |\n| Motivation | Avoids the strong interference effects that arise when one back-propagation network learns different subtasks[4](https://proceedings.neurips.cc/paper/1990/file/432aca3a1e345e339f35a30c8f65edce-Paper.pdf) |\n| Scale result | Sparsely-gated MoE reported over 1000x model-capacity gains with minor losses in computational efficiency, and a 137-billion-parameter MoE between LSTM layers[5](https://ar5iv.labs.arxiv.org/html/1701.06538) |\n| Systematic review | Of 86 comparative studies, nearly two-thirds report accuracy improvements and 82% report efficiency benefits[6](https://link.springer.com/chapter/10.1007/978-3-031-66459-5_2) |\n| Main failure modes | Module collapse, shrinking batch size, training instability, and overfitting[1](https://ar5iv.labs.arxiv.org/html/2310.01154) • [2](https://arxiv.org/html/2302.11529) |\n| Recent LLM use | Mixtral routes each token to 2 of 8 experts per layer; DeepSeekMoE 16B matches 7B dense models at about 40% of the computation[7](https://arxiv.org/pdf/2401.04088) • [8](https://aclanthology.org/2024.acl-long.70/) |\n\n## How it works\n\nSeveral expert networks, each a separate network, learn to handle a subset of the training cases, while a gating network decides which expert should be used for each case.[9](http://www.cs.toronto.edu/~hinton/absps/jjnh91.pdf) • [4](https://proceedings.neurips.cc/paper/1990/file/432aca3a1e345e339f35a30c8f65edce-Paper.pdf) The gating network has as many output units as there are experts, and its activations must be nonnegative and sum to one; this is implemented with the softmax activation function, so the gate outputs form a soft assignment over the experts, akin to a partition of unity, rather than a disjoint partition of the input space.[3](https://proceedings.neurips.cc/paper/1990/file/f74909ace68e51891440e4da0b65a70c-Paper.pdf) • [10](https://link.springer.com/chapter/10.1007/978-3-662-43505-2_28) Concretely, for an input \\( x \\), the \\( n \\)-th expert produces an output vector, and the gating network produces \\( N \\) scalar outputs that weight or select the experts.[10](https://link.springer.com/chapter/10.1007/978-3-662-43505-2_28)\n\nThe design responds to a specific failure of monolithic networks. If back-propagation trains a single multilayer network to perform different subtasks on different occasions, strong interference effects lead to slow learning and poor generalization; a [mixture of experts](https://www.edgepedia.org/mixture-of-experts) plus a gate reduces this interference.[4](https://proceedings.neurips.cc/paper/1990/file/432aca3a1e345e339f35a30c8f65edce-Paper.pdf) The 1991 procedure can be viewed either as a modular version of a multilayer supervised network or as an associative version of competitive learning.[9](http://www.cs.toronto.edu/~hinton/absps/jjnh91.pdf)\n\n## How it is done\n\nBuilding a modular network involves four decisions, which a 2023 survey of modular deep learning organizes along four dimensions: how modules are implemented, how routing selects active modules, how module outputs are aggregated, and how modules are trained with the rest of the model.[2](https://arxiv.org/html/2302.11529) In the classic form, the steps are: define the expert networks, attach a gating network whose softmax outputs assign each input a distribution over experts, combine expert outputs under this weighting, and train the whole system so that experts specialize on different subsets of cases while the gate learns the assignment.[3](https://proceedings.neurips.cc/paper/1990/file/f74909ace68e51891440e4da0b65a70c-Paper.pdf) • [9](http://www.cs.toronto.edu/~hinton/absps/jjnh91.pdf) In sparse modern forms, the gate selects only a small subset of experts per example, which is what allows parameter count to grow while time complexity stays constant.[2](https://arxiv.org/html/2302.11529)\n\nThree training regimes are used. In the classic and sparse MoE forms, all parts of the network, experts and gate alike, are trained jointly by back-propagation.[5](https://ar5iv.labs.arxiv.org/html/1701.06538) For stacked MoE, alternative training schemes explored in the literature include generalized Viterbi EM, multi-agent reinforcement learning, and genetic algorithms.[1](https://ar5iv.labs.arxiv.org/html/2310.01154) A third regime is modular training, proposed as an alternative to end-to-end training, in which modules are composed so that they can be trained independently and then kept for future use.[11](https://dl.acm.org/doi/abs/10.1016/j.neunet.2021.03.034) More broadly, modules can be trained jointly with the base model in multi-task learning, added sequentially in continual learning, or integrated post-hoc into a frozen pre-trained model.[2](https://arxiv.org/html/2302.11529) Routing itself splits into soft routing, which is amenable to vanilla gradient descent but highly inefficient, and hard routing, which requires approximate inference but enables conditional computation and module specialization.[2](https://arxiv.org/html/2302.11529)\n\n## Origin\n\nThe mixture-of-experts architecture appears in the literature with the 1991 Neural Computation paper \"Adaptive Mixtures of Local Experts\" by [Robert A. Jacobs](https://www.edgepedia.org/robert-a-jacobs), [Michael I. Jordan](https://www.edgepedia.org/michael-i-jordan), [Steven J. Nowlan](https://www.edgepedia.org/steven-j-nowlan), and Geoffrey E. Hinton, which presented a new supervised learning procedure for systems composed of many separate networks.[12](https://doi.org/10.1162/neco.1991.3.1.79) • [9](http://www.cs.toronto.edu/~hinton/absps/jjnh91.pdf) A companion NeurIPS paper states that the architecture combines earlier work on learning task decompositions in a modular architecture by Jacobs, Jordan, and Barto with the mixture-models view of competitive learning advocated by Nowlan and by Hinton and Nowlan.[3](https://proceedings.neurips.cc/paper/1990/file/f74909ace68e51891440e4da0b65a70c-Paper.pdf) The companion Cognitive Science paper compared the architecture with two multilayer networks on the \"what\" and \"where\" vision tasks and noted that function decomposition is an underconstrained problem.[13](https://onlinelibrary.wiley.com/doi/10.1207/s15516709cog1502_2)\n\n## Variants\n\nSeveral named variants exist. The hierarchical mixture of experts recursively decomposes the problem, with gating at multiple levels, trained with the EM algorithm in the 1994 formulation.[10](https://link.springer.com/chapter/10.1007/978-3-662-43505-2_28) The reference-work lineage records the 1994 Neural Computation paper \"Hierarchical mixture of experts and the EM algorithm\" (6(2), 181-214) as the canonical hierarchical extension.[10](https://link.springer.com/chapter/10.1007/978-3-662-43505-2_28) Stacked MoE uses multiple mixture-of-experts layers, each with its own gating network.[1](https://ar5iv.labs.arxiv.org/html/2310.01154) The sparsely-gated MoE layer, introduced by [Noam Shazeer](https://www.edgepedia.org/noam-shazeer) and colleagues in 2017 on arXiv, scales the idea to up to thousands of feed-forward expert sub-networks with a trainable gate selecting a sparse combination per example.[5](https://ar5iv.labs.arxiv.org/html/1701.06538) Neural module networks treat visual question answering as a highly-multitask setting in which each problem instance is a novel task whose identity is expressed noisily in language.[14](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Andreas_Neural_Module_Networks_CVPR_2016_paper.pdf) Switch Transformers, the 2021 arXiv work by William Fedus, Barret Zoph, and Noam Shazeer, scales sparse routing to trillion-parameter models with simple and efficient sparsity.[15](https://doi.org/10.48550/arxiv.2101.03961) The 2023 modular deep learning survey unifies these and adapter-style methods along the four dimensions of module implementation, routing, aggregation, and training.[2](https://arxiv.org/html/2302.11529)\n\n## Applications\n\nModular networks are applied wherever tasks decompose or models must serve many purposes. Early evaluations covered a \"what\" and \"where\" vision task and a multi-payload robotics task.[3](https://proceedings.neurips.cc/paper/1990/file/f74909ace68e51891440e4da0b65a70c-Paper.pdf) On a complex but low-dimensional vowel recognition task, the modular architecture of competing experts showed consistently better generalization across many task variations than a comparable single back-propagation network, and the learning procedure divided the vowel discrimination task into appropriate subtasks.[4](https://proceedings.neurips.cc/paper/1990/file/432aca3a1e345e339f35a30c8f65edce-Paper.pdf) • [9](http://www.cs.toronto.edu/~hinton/absps/jjnh91.pdf) Modern uses include multi-task and continual learning, and post-hoc modularization of frozen pre-trained models, where fine-tuning for a task requires storing only a modular adapter rather than a separate copy of the entire model; modules can also be added or removed on the fly, and only the selected modules are executed for a given input, an ability known as conditional computation.[2](https://arxiv.org/html/2302.11529) Visual question answering is a recurring testbed for neural module networks.[14](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Andreas_Neural_Module_Networks_CVPR_2016_paper.pdf) In large language models, sparse MoE layers are the dominant form: Mixtral, reported in 2024 by Albert Q. Jiang and colleagues on arXiv, is a decoder-only sparse mixture-of-experts network whose feedforward block picks from 8 distinct groups of parameters, with a router choosing two of these experts for every token at every layer,[7](https://arxiv.org/pdf/2401.04088) and DeepSeekMoE, reported in 2024 by Damai Dai and colleagues on arXiv, adds fine-grained expert segmentation (splitting \\( N \\) experts into \\( m \\cdot N \\) and activating \\( m \\cdot K \\)) and \\( K_{s} \\) shared experts that capture common knowledge and reduce redundancy among routed experts.[8](https://aclanthology.org/2024.acl-long.70/)\n\n## Limitations and alternatives\n\nLearned routing poses training instability, module collapse, and overfitting.[2](https://arxiv.org/html/2302.11529) Module collapse means under-utilization of modules or lack of module diversity: because the gating network's behavior is self-reinforcing during training, premature modules may be selected and thus trained even more.[1](https://ar5iv.labs.arxiv.org/html/2310.01154) A second MoE-specific failure mode is shrinking batch size, where the batch is reduced for each conditionally activated module, hurting hardware efficiency.[1](https://ar5iv.labs.arxiv.org/html/2310.01154) Benchmark work finds standard modular systems often sub-optimal both in focusing on the right information and in their ability to specialize, suggesting the need for additional inductive biases, and provides rule-based tasks and metrics to quantify collapse and specialization.[16](https://papers.nips.cc/paper_files/paper/2022/file/b8d1d741f137d9b6ac4f3c1683791e4a-Paper-Conference.pdf) The systematic review adds that publication bias can favor modular networks and that most studies were conducted in laboratory environments on focused tasks and static requirements.[6](https://link.springer.com/chapter/10.1007/978-3-031-66459-5_2)\n\nCompared with a monolithic deep network, a modular network trades a simpler single-objective training procedure for conditional computation, local updatability, and potential reuse, at the cost of routing machinery and its failure modes; the accuracy advantage is conditional, appearing in about two-thirds of comparative studies and mainly when tasks contain many underlying rules.[6](https://link.springer.com/chapter/10.1007/978-3-031-66459-5_2) • [16](https://papers.nips.cc/paper_files/paper/2022/file/b8d1d741f137d9b6ac4f3c1683791e4a-Paper-Conference.pdf)\n\n## References\n\n1. [Modularity in Deep Learning: A Survey (arXiv 2310.01154)](https://ar5iv.labs.arxiv.org/html/2310.01154)\n2. [Modular Deep Learning (survey, arXiv 2302.11529)](https://arxiv.org/html/2302.11529)\n3. [A Competitive Modular Connectionist Architecture (NeurIPS 1990)](https://proceedings.neurips.cc/paper/1990/file/f74909ace68e51891440e4da0b65a70c-Paper.pdf)\n4. [Evaluation of Adaptive Mixtures of Competing Experts (NeurIPS 1990)](https://proceedings.neurips.cc/paper/1990/file/432aca3a1e345e339f35a30c8f65edce-Paper.pdf)\n5. [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al., 2017)](https://ar5iv.labs.arxiv.org/html/1701.06538)\n6. [On Modularity of Neural Networks: Systematic Review and Open Challenges](https://link.springer.com/chapter/10.1007/978-3-031-66459-5_2)\n7. [Mixtral of Experts](https://arxiv.org/pdf/2401.04088)\n8. [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (ACL 2024)](https://aclanthology.org/2024.acl-long.70/)\n9. [Adaptive Mixtures of Local Experts (Neural Computation, 1991)](http://www.cs.toronto.edu/~hinton/absps/jjnh91.pdf)\n10. [Deep and Modular Neural Networks (Springer Handbook chapter)](https://link.springer.com/chapter/10.1007/978-3-662-43505-2_28)\n11. [Design and independent training of composable and reusable neural modules (Neural Networks, 2021)](https://dl.acm.org/doi/abs/10.1016/j.neunet.2021.03.034)\n12. [Robert A. Jacobs and colleagues (1991). Adaptive Mixtures of Local Experts. Neural Computation.](https://doi.org/10.1162/neco.1991.3.1.79)\n13. [Task Decomposition Through Competition in a Modular Connectionist Architecture: The What and Where Vision Tasks (Cognitive Science, 1991)](https://onlinelibrary.wiley.com/doi/10.1207/s15516709cog1502_2)\n14. [Neural Module Networks (Andreas et al., CVPR 2016)](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Andreas_Neural_Module_Networks_CVPR_2016_paper.pdf)\n15. [Fedus, William, Zoph, Barret, Shazeer, Noam (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2101.03961)\n16. [Is a Modular Architecture Enough? (NeurIPS 2022)](https://papers.nips.cc/paper_files/paper/2022/file/b8d1d741f137d9b6ac4f3c1683791e4a-Paper-Conference.pdf)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation › Neural networks and deep learning › Titles I to Z*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [ "http://www.cs.toronto.edu/~hinton/absps/jjnh91.pdf" ], "url": "https://www.edgepedia.org/modular-neural-network", "markdown_url": "https://www.edgepedia.org/modular-neural-network.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Modular neural network\", Edgepedia (EdgeChat), https://www.edgepedia.org/modular-neural-network. Edgepedia Community License 1.0.", "credit_md": "\"[Modular neural network](https://www.edgepedia.org/modular-neural-network)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/modular-neural-network](https://www.edgepedia.org/modular-neural-network). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Modular neural network\", Edgepedia (EdgeChat), https://www.edgepedia.org/modular-neural-network. Edgepedia Community License 1.0.", "speakable": "A modular neural network is a machine learning architecture that decomposes computation into specialized modules, combined by a routing or gating mechanism to solve the overall task." }