A Fully Trainable Network with RNN-based Pooling

Abstract Pooling is an important component in convolutional neural networks (CNNs) for aggregating features and reducing computational burden. Compared with other components such as convolutional layers and fully connected layers which are completely learned from data, the pooling component is still handcrafted such as max pooling and average pooling. This paper proposes a learnable pooling function using recurrent neural networks (RNN) so that the pooling can be fully adapted to data and other components of the network, leading to an improved performance. Such a network with learnable pooling function is referred to as a fully trainable network (FTN). Experimental results demonstrate that the proposed RNN based pooling can well approximate the existing pooling functions with just one neuron, thus making it appropriate to be used as pooling function in a network with its rich representation capability. Furthermore, experiments have shown that the proposed FTN can achieve better performance than the existing pooling methods under similar network architectures for image classification.