Optimization Based Layer-Wise Pruning Threshold Method for Accelerating Convolutional Neural Networks

Abstract

Among various network compression methods, network pruning has developed rapidly due to its superior compression performance. However, the trivial pruning threshold limits the compression performance of pruning. Most conventional pruning threshold methods are based on well-known hard or soft techniques that rely on time-consuming handcrafted tests or domain experience. To mitigate these issues, we propose a simple yet effective general pruning threshold method from an optimization point of view. Specifically, the pruning threshold problem is formulated as a constrained optimization program that minimizes the size of each layer. More importantly, our pruning threshold method together with conventional pruning works achieves a better performance across various pruning scenarios on many advanced benchmarks. Notably, for the L1-norm pruning algorithm with VGG-16, our method achieves higher FLOPs reductions without utilizing time-consuming sensibility analysis. The compression ratio boosts from 34% to 53%, which is a huge improvement. Similar experiments with ResNet-56 reveal that, even for compact networks, our method achieves competitive compression performance even without skipping any sensitive layers.