BitCluster: Fine-Grained Weight Quantization for Load-Balanced Bit-Serial Neural Network Accelerators

Convolutional neural network (CNN) has demonstrated great success in pattern recognition scenarios at the cost of nearly billions of parameters and consequent convolution operations. Various dedicated hardware designs are proposed to accelerate the CNN computation in more energy-efficient manners. Especially, the bit-serial accelerator (BSA) is one of the most effective approaches on resource-limited platforms by eliminating zero-bit computations. However, the irregular distribution and varying number of effectual (nonzero) bits in weights significantly cause hardware underutilization, impeding further performance improvement of state-of-the-art BSAs. To address this issue, BitCluster, a hardware-friendly quantization method, is proposed to make each weight with the identical number of effectual bits for load-balanced computation. Considering distinct sensitivities to weight precision in different neural layers, layer-level BitCluster is proposed to design further for fine-grained weight quantization. It systematically determines the layerwise quantization configurations, which significantly improve the overall performance with <1% accuracy loss. BitCluster is comprehensively evaluated on a BitCluster-compatible BSA design by taking six mainstream CNN models as benchmarks. The experimental results show that the BitCluster-based BSA achieves $1.6\times $ higher hardware utilization and $3.4\times $ speedup on average than state-of-the-art BSAs, with $5\times $ better energy efficiency on average.