# Tensor factorization

Tensor factorization decomposes a multiway array (a tensor) into a small number of lower-dimensional component factors, and is used for dimensionality reduction, estimation, and recovery of latent structure in high-dimensional data. The two central models are the CP (CANDECOMP/PARAFAC) decomposition, which writes a tensor as a sum of rank-one terms, and the [Tucker decomposition](https://www.edgepedia.org/tucker-decomposition), a higher-order form of principal component analysis; both generalize the matrix singular value decomposition (SVD) to arrays with three or more modes.<sup>[1](https://epubs.siam.org/doi/10.1137/07070111X)</sup>

| Key fact | Detail |
|---|---|
| CP model | A tensor is written as a sum of R rank-one tensors, each an outer product of one vector per mode<sup>[1](https://epubs.siam.org/doi/10.1137/07070111X)</sup> |
| Tucker model | A core tensor multiplied by a matrix along each mode<sup>[1](https://epubs.siam.org/doi/10.1137/07070111X)</sup> |
| Parameter savings | CP reduces a \( D_{\mathrm{th}} \)-order tensor from \( \prod_{d=1}^{D} p_d \) free parameters to \( R \cdot \sum_{d=1}^{D} p_d \)<sup>[2](https://web.ics.purdue.edu/~sun244/Tensor_survey.pdf)</sup> |
| Uniqueness | CP is essentially unique under mild conditions; Kruskal's sufficient condition is stated in terms of the k-ranks of the factor matrices<sup>[3](https://doi.org/10.1016/0024-3795%2877%2990069-6)</sup> |
| Workhorse algorithm | Alternating least squares: fix all factor matrices but one, solve a linear least-squares subproblem, repeat<sup>[1](https://epubs.siam.org/doi/10.1137/07070111X)</sup> |
| Main failure mode | The best rank-R approximation may not exist, producing degenerate solutions with diverging components<sup>[4](https://pmc.ncbi.nlm.nih.gov/articles/PMC2792364/)</sup> |
| Rank choice | Tensor rank is NP-hard to compute, so in practice one fits several ranks and compares fit<sup>[5](https://ar5iv.labs.arxiv.org/html/1711.10781)</sup> |

## How it works

A rank-one third-order tensor of size \( I \times J \times K \) is the outer product of three vectors, \( X(i,j,k) = a(i)\,b(j)\,c(k) \).<sup>[6](http://www.cs.ucr.edu/~epapalex/papers/tsp17-tensors.pdf)</sup> The rank of a tensor is the smallest number of such rank-one terms that sum to it. Unlike matrix rank, tensor rank can differ depending on whether the entries are treated as real or complex, and computing tensor rank is NP-hard, so exact determination can be intractable for large or general inputs.<sup>[1](https://epubs.siam.org/doi/10.1137/07070111X)</sup>

The CP decomposition approximates a tensor as

\[ X \approx \sum_{r=1}^{R} a_r \circ b_r \circ c_r, \]

where each triple of vectors forms one component and \( R \) is the chosen rank. The Tucker decomposition instead writes

\[ X \approx G \times_1 A \times_2 B \times_3 C, \]

a core tensor \( G \) transformed by a factor matrix along each mode; it is a form of higher-order PCA.<sup>[1](https://epubs.siam.org/doi/10.1137/07070111X)</sup>

Uniqueness is where tensors depart most sharply from matrices. A matrix factorization is unique only under extra assumptions such as the orthogonality imposed by the SVD, and matrix factor analysis suffers from a rotation problem; tensor decompositions are unique under much milder conditions.<sup>[7](https://oar.princeton.edu/bitstream/88435/pr1w814/1/TensorDecompositionsUniquenessThreshold.pdf)</sup> Harshman's 1970 theorem gives uniqueness, with a polynomial-time algorithm, when two factor matrices are linearly independent and no two columns of the third are collinear.<sup>[7](https://oar.princeton.edu/bitstream/88435/pr1w814/1/TensorDecompositionsUniquenessThreshold.pdf)</sup> Kruskal's 1977 theorem gives uniqueness conditions expressed in terms of the k-ranks of the factor matrices, where the k-rank of a matrix is the largest k such that any k columns are linearly independent.<sup>[3](https://doi.org/10.1016/0024-3795%2877%2990069-6)</sup> The Tucker decomposition, by contrast, is generally not unique because its core tensor can be arbitrarily structured.<sup>[5](https://ar5iv.labs.arxiv.org/html/1711.10781)</sup>

## How it is done

For CP, the workhorse is alternating least squares (ALS): fix two factor matrices, solve a linear least-squares problem for the third (the update can be written \( T_{(n)} \approx A_{(n)} \cdot V_{\{n\}}^{\mathrm{T}} \) via Khatri–Rao products of the other factors), and cycle through the modes until a convergence criterion is met.<sup>[1](https://epubs.siam.org/doi/10.1137/07070111X)</sup><sup> • </sup><sup>[8](https://ftp.esat.kuleuven.be/pub/stadius/sistakulak/reports/algorithms_for_tensor_decompositions.pdf)</sup> ALS is simple to implement but can take many iterations, is not guaranteed to converge even to a stationary point, and depends heavily on the starting guess, so practitioners run it from multiple random initializations.<sup>[1](https://epubs.siam.org/doi/10.1137/07070111X)</sup>

Because tensor rank cannot be computed directly, one fits models at several ranks and picks the best approximation; the CORCONDIA consistency diagnostic compares different numbers of components.<sup>[5](https://ar5iv.labs.arxiv.org/html/1711.10781)</sup><sup> • </sup><sup>[1](https://epubs.siam.org/doi/10.1137/07070111X)</sup> Spectral initialization from the [HOSVD](https://www.edgepedia.org/higher-order-singular-value-decomposition), computed from leading singular vectors of mode matricizations (unfolding a tensor into a matrix), is widely used, and truncating to the HOSVD before ALS improves convergence speed provided the reduced dimensions are not smaller than the tensor rank; the HOOI algorithm refines an HOSVD initialization by ALS.<sup>[9](https://arxiv.org/html/2405.18412)</sup><sup> • </sup><sup>[10](https://www.gipsa-lab.grenoble-inp.fr/~pierre.comon/FichiersPdf/ComoLA09-jchemo.pdf)</sup><sup> • </sup><sup>[5](https://ar5iv.labs.arxiv.org/html/1711.10781)</sup> Numerical comparisons find nonlinear least-squares methods (Gauss–Newton, Levenberg–Marquardt) less sensitive to initialization, more efficient, and less prone to swamps, at higher memory and time cost per iteration; all-at-once optimizers (CP-OPT, CP-NLS) and direct computation via simultaneous diagonalization are the main alternatives.<sup>[8](https://ftp.esat.kuleuven.be/pub/stadius/sistakulak/reports/algorithms_for_tensor_decompositions.pdf)</sup><sup> • </sup><sup>[1](https://epubs.siam.org/doi/10.1137/07070111X)</sup><sup> • </sup><sup>[11](https://users.wfu.edu/ballard/pdfs/tensor_textbook.pdf)</sup> A hybrid initialization, TASD, compresses the tensor by Tucker decomposition (via HOOI) before applying simultaneous diagonalization to the low-dimensional core, improving robustness to noise.<sup>[12](https://doi.org/10.48550/arxiv.2505.23046)</sup> Degeneracy is monitored by watching the smallest singular values of the factor matrices together with the factor norms.<sup>[4](https://pmc.ncbi.nlm.nih.gov/articles/PMC2792364/)</sup> Standard software includes the N-way Toolbox, Tensor Toolbox, and Multilinear Engine.<sup>[1](https://epubs.siam.org/doi/10.1137/07070111X)</sup>

## Origin

Hitchcock proposed expressing a tensor as a finite sum of rank-one terms, the polyadic form, in 1927.<sup>[13](https://doi.org/10.1002/sapm192761164)</sup> Cattell stated the principle of parallel proportional profiles for choosing factors in 1944.<sup>[14](https://doi.org/10.1007/bf02288739)</sup> Tucker introduced the three-mode factor analysis model, with a core matrix relating three factor matrices, in work culminating in his 1966 paper.<sup>[15](https://doi.org/10.1007/bf02289464)</sup> In 1970 the same trilinear model was proposed twice independently: Harshman presented PARAFAC, generalizing Cattell's principle and proving uniqueness under stated conditions,<sup>[16](https://psychology.uwo.ca/faculty/harshman/wpppfac0.pdf)</sup> while Carroll and Chang presented CANDECOMP, an N-way generalization of the Eckart–Young decomposition, from which they developed INDSCAL.<sup>[17](https://doi.org/10.1007/bf02310791)</sup> Kruskal supplied the classical uniqueness theorem in 1977.<sup>[3](https://doi.org/10.1016/0024-3795%2877%2990069-6)</sup>

## Variants

The Tucker model is computed in its SVD-like constrained form by the HOSVD, which takes the mode-q factor matrices to be the left singular matrices of the mode-q matricization; truncating this constrained version usually gives a good approximation, though not the optimal one.<sup>[9](https://arxiv.org/html/2405.18412)</sup><sup> • </sup><sup>[18](https://ftp.esat.kuleuven.be/pub/stadius/delathauwer/reports/ldl-09-34.pdf)</sup> The block term decomposition, introduced by De Lathauwer in 2008, unifies the two models: with a single term it is a Tucker decomposition, and with scalar cores it is CP.<sup>[19](https://doi.org/10.1137/070690729)</sup> Nonnegative variants constrain the factor matrices to be nonnegative, following the interpretability arguments made for nonnegative matrix factorization; unlike unconstrained CP, nonnegative CP is always well defined.<sup>[20](https://psycnet.apa.org/doi/10.1145/2915921)</sup><sup> • </sup><sup>[21](https://ar5iv.labs.arxiv.org/html/1412.1885)</sup> Other named models in the family include PARAFAC2, INDSCAL, CANDELINC, DEDICOM, and PARATUCK2.<sup>[1](https://epubs.siam.org/doi/10.1137/07070111X)</sup> For very high-order tensors, the Tensor-Train decomposition, proposed by Oseledets in 2011, imposes a parsimonious model to counter the curse of dimensionality.<sup>[22](https://doi.org/10.1137/090752286)</sup> A 2024 algorithm based on Koszul–Young flattenings decomposes generic \( n \times n \times n \) tensors and certifies uniqueness up to rank \( (2-\epsilon)n \). A 2014 paper proved identifiability of a specific \( 5 \times 5 \times 5 \) tensor of rank 7; this exceeds Kruskal's exact sufficient bound for those dimensions, but not the simplified \( r \le 3n/2 \) bound.<sup>[23](https://arxiv.org/html/2411.14344v1)</sup><sup> • </sup><sup>[25](https://arxiv.org/abs/1403.4157)</sup>

## Applications

Psychometrics and chemometrics historically drove the theory and algorithms; signal processing followed in the 1990s, and computer science adopted tensors roughly a decade before 2017.<sup>[6](http://www.cs.ucr.edu/~epapalex/papers/tsp17-tensors.pdf)</sup> [Signal processing](https://www.edgepedia.org/signal-processing) uses CP for unsupervised separation of speech mixtures, code-division communication signals without knowledge of the codes, and emitter localization for radar and passive sensing.<sup>[6](http://www.cs.ucr.edu/~epapalex/papers/tsp17-tensors.pdf)</sup> In machine learning, tensor decompositions support temporal pattern discovery, knowledge-graph relation inference, and method-of-moments estimation for Gaussian mixture and topic models.<sup>[5](https://ar5iv.labs.arxiv.org/html/1711.10781)</sup> Tucker approximation is used for dimensionality reduction of large tensor datasets, subspace estimation, harmonic retrieval, and classification.<sup>[18](https://ftp.esat.kuleuven.be/pub/stadius/delathauwer/reports/ldl-09-34.pdf)</sup> Neural-network compression applies tensor-train, block-term, and Tucker decompositions to fully connected layers, which hold about 80% of VGG-19's parameters.<sup>[2](https://web.ics.purdue.edu/~sun244/Tensor_survey.pdf)</sup>

## Limitations and alternatives

The sharpest contrast with matrices concerns existence. For matrices, the Eckart–Young theorem gives the best low-rank approximation by truncated SVD; for tensors no such theorem exists, and the best rank-k approximation may not exist at all, a prevalent phenomenon rather than a pathological exception.<sup>[10](https://www.gipsa-lab.grenoble-inp.fr/~pierre.comon/FichiersPdf/ComoLA09-jchemo.pdf)</sup><sup> • </sup><sup>[9](https://arxiv.org/html/2405.18412)</sup> De Silva and Lim showed that for \( 2 \le R \le \min(I,J,K) \) there always exists a three-way array of rank \( R+1 \) with no optimal CP solution.<sup>[4](https://pmc.ncbi.nlm.nih.gov/articles/PMC2792364/)</sup> When no optimum exists, any sequence of factors decreasing the CP criterion to its infimum becomes nearly rank deficient while some factor norms tend to infinity, the signature of degeneracy; constraining one factor matrix to be column-wise orthonormal, or imposing nonnegativity on all factors, guarantees an optimal solution.<sup>[4](https://pmc.ncbi.nlm.nih.gov/articles/PMC2792364/)</sup> For generic \( I \times J \times 2 \) arrays, diverging components can be handled by computing a generalized [Schur decomposition](https://www.edgepedia.org/schur-decomposition) instead, which always exists and splits into a nondiverging CP part plus a sparse Tucker3 part.<sup>[24](https://epubs.siam.org/doi/10.1137/070692121)</sup>

CP algorithms also converge to local solutions, especially when bottlenecks exist in a factor matrix or the model is not unique, and the loss surface has exponentially many local minima when the signal-to-noise ratio is weak.<sup>[21](https://ar5iv.labs.arxiv.org/html/1412.1885)</sup><sup> • </sup><sup>[12](https://doi.org/10.48550/arxiv.2505.23046)</sup> Spurious local minima appear when the signal strength \( \lambda \) is much smaller than the dimension \( d \), while power iteration initialized by the HOSVD yields consistent estimation when \( \lambda \gtrsim d^{3/4} \).<sup>[9](https://arxiv.org/html/2405.18412)</sup> Tang, Chhor, Klopp, and Zhang established non-asymptotic, minimax-optimal error bounds for ALS under general CP models with arbitrary rank and order, matched by lower bounds, and proved a two-phase convergence behavior: a quadratic phase under low coherence followed by a linear phase near the truth.<sup>[12](https://doi.org/10.48550/arxiv.2505.23046)</sup> Choosing between the models is a trade-off: CP gives the most compact, essentially unique representation but can be ill-posed and fit poorly, while Tucker is not unique but often fits better and has been shown generally more effective at estimating missing values.<sup>[21](https://ar5iv.labs.arxiv.org/html/1412.1885)</sup><sup> • </sup><sup>[20](https://psycnet.apa.org/doi/10.1145/2915921)</sup> A common rule of thumb is CP for latent parameter estimation and Tucker for subspace estimation, compression, and dimensionality reduction.<sup>[5](https://ar5iv.labs.arxiv.org/html/1711.10781)</sup> Flattening a tensor and applying PCA is statistically inefficient and lacks interpretability, and the matrix PCA/SVD equivalence does not carry over to multiway PCA.<sup>[9](https://arxiv.org/html/2405.18412)</sup>

## References

1. [Tensor Decompositions and Applications (Kolda & Bader, SIAM Review 2009)](https://epubs.siam.org/doi/10.1137/07070111X)
2. [An Overview of Tensor Analysis in Modern Statistical Learning (survey, author-hosted)](https://web.ics.purdue.edu/~sun244/Tensor_survey.pdf)
3. [Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics (Linear Algebra and its Applications, 1977)](https://doi.org/10.1016/0024-3795%2877%2990069-6)
4. [On the Non-Existence of Optimal Solutions and the Occurrence of "Degeneracy" in the CANDECOMP/PARAFAC Model (Psychometrika; PMC copy)](https://pmc.ncbi.nlm.nih.gov/articles/PMC2792364/)
5. [Introduction to Tensor Decompositions and their Applications in Machine Learning (Rabanser, Shchur, Günnemann)](https://ar5iv.labs.arxiv.org/html/1711.10781)
6. [Tensor Decomposition for Signal Processing and Machine Learning (Sidiropoulos, De Lathauwer, Fu, Huang, Papalexakis, Faloutsos; IEEE TSP 2017)](http://www.cs.ucr.edu/~epapalex/papers/tsp17-tensors.pdf)
7. [Open Problem: Tensor Decompositions: Algorithms up to the Uniqueness Threshold? (Bhaskara, Charikar, Moitra, Vijayaraghavan, COLT 2014)](https://oar.princeton.edu/bitstream/88435/pr1w814/1/TensorDecompositionsUniquenessThreshold.pdf)
8. [Algorithms for the Computation of the Canonical Polyadic and Block Term Decompositions (SIAM Optimization / KU Leuven report)](https://ftp.esat.kuleuven.be/pub/stadius/sistakulak/reports/algorithms_for_tensor_decompositions.pdf)
9. [Tensor Methods in High Dimensional Data Analysis: Opportunities and Challenges (2024 survey)](https://arxiv.org/html/2405.18412)
10. [Tensor Decompositions, Alternating Least Squares and other Tales (Comon, Luciani & De Almeida, J. Chemometrics 2009)](https://www.gipsa-lab.grenoble-inp.fr/~pierre.comon/FichiersPdf/ComoLA09-jchemo.pdf)
11. [Tensor Decompositions: a textbook (Ballard, Wake Forest)](https://users.wfu.edu/ballard/pdfs/tensor_textbook.pdf)
12. [Tang, Runshi and colleagues (2025). Revisit CP Tensor Decomposition: Statistical Optimality and Fast Convergence. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2505.23046)
13. [Frank L. Hitchcock (1927). The Expression of a Tensor or a Polyadic as a Sum of Products. Studies in Applied Mathematics.](https://doi.org/10.1002/sapm192761164)
14. [Raymond B. Cattell (1944). “Parallel Proportional Profiles” and other Principles for Determining the Choice of Factors by Rotation. Psychometrika.](https://doi.org/10.1007/bf02288739)
15. [Ledyard R Tucker (1966). Some Mathematical Notes on Three-Mode Factor Analysis. Psychometrika.](https://doi.org/10.1007/bf02289464)
16. [Foundations of the PARAFAC procedure: Models and conditions for an 'explanatory' multi-modal factor analysis (Harshman, UCLA Working Papers in Phonetics 16, 1970)](https://psychology.uwo.ca/faculty/harshman/wpppfac0.pdf)
17. [J. Douglas Carroll, Jih-Jie Chang (1970). Analysis of Individual Differences in Multidimensional Scaling Via an N-way Generalization of “Eckart-Young” Decomposition. Psychometrika.](https://doi.org/10.1007/bf02310791)
18. [A Survey of Tensor Methods (De Lathauwer, KU Leuven report)](https://ftp.esat.kuleuven.be/pub/stadius/delathauwer/reports/ldl-09-34.pdf)
19. [Lieven De Lathauwer (2008). Decompositions of a Higher-Order Tensor in Block Terms, Part II: Definitions and Uniqueness. SIAM Journal on Matrix Analysis and Applications.](https://doi.org/10.1137/070690729)
20. [Tensors for Data Mining and Data Fusion: Models, Applications, and Scalable Algorithms (Papalexakis, Faloutsos & Sidiropoulos, ACM TIST 2016/2017; author copy excerpts from cs.ucr.edu merged here)](https://psycnet.apa.org/doi/10.1145/2915921)
21. [Decomposition of Big Tensors With Low Multilinear Rank (Zhang, Zhou, et al.)](https://ar5iv.labs.arxiv.org/html/1412.1885)
22. [I. V. Oseledets (2011). Tensor-Train Decomposition. SIAM Journal on Scientific Computing.](https://doi.org/10.1137/090752286)
23. [Overcomplete Tensor Decomposition via Koszul–Young Flattenings (2024)](https://arxiv.org/html/2411.14344v1)
24. [A Method to Avoid Diverging Components in the Candecomp/Parafac Model for Generic I×J×2 Arrays (SIAM J. Matrix Anal. Appl.)](https://epubs.siam.org/doi/10.1137/070692121)
25. [An algorithm for generic and low-rank specific identifiability of complex tensors](https://arxiv.org/abs/1403.4157)

---
*Topic: Encyclopedia › Physical world and mathematics › Mathematics and statistics › Statistics and probability › Multivariate association and dimension reduction › Titles Mo to W*

*Initially written Sep 29, 2026 · Reviewed: Sep 30, 2026 · Edited: Sep 30, 2026 · Last review: Sep 30, 2026*

*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*

License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license