Tensor Decomposition in Deep Networks

With the help of available big data and powerful computing units, deep learning has achieved remarkable performance in a series of data processing applications, such as speech recognition, computer vision, and natural language processing. Most of the existing neural networks take a lot of matrix computation operators in their architecture, which requires that the multidimensional input or output must be transformed into matrix form and the multilinear structure information may inevitably be lost. In order to avoid the performance degeneration from the structure loss, tensor computation operators have been introduced to extend the matrix-based neural networks. In this chapter, we first give a brief introduction of classical deep neural networks. Tensor computation is used to extend deep neural networks mainly from three ways in existing works. The first one extends the matrix product in each neural layer to tensor counterparts, inspired by different tensor decompositions, including t-product connected deep neural network, mode-n product-based tensorized neural network, etc. In the second way, different tensor decompositions have been used to compress the whole network parameters by low-rank tensor approximation. The third way builds up the connection between tensor decompositions and neural networks, which helps to theoretically analyze deep learning. To demonstrate the effectiveness of tensor analysis for deep neural networks, experimental results on low-rank tensor approximation for network compression show that the computational complexity and storage requirement can be largely reduced, while the image classification accuracy can be maintained.