Improving knowledge distillation performance for acoustic scene classification

Abstract

Acoustic Scene Classification (ASC) enables devices to recognise environmental contexts, with applications in safety and context-aware services. However, state-of-the-art ASC models often demand significant computational resources, limiting their deployment in low-complexity devices. Knowledge Distillation (KD) addresses this by transferring knowledge from complex teacher models to lightweight student models, but challenges remain due to teacher-student capacity disparities and poor knowledge quality, especially in data scarce scenarios. This work proposes a robust hybrid Transfer Learning-Knowledge Distillation framework to enhance knowledge transfer. Central to this framework is the Slow learner for Incremental Transfer learning (SIT) approach, which enriches teacher models by incorporating an ASC-relevant external dataset via an intermediate training stage. This process improves the feature diversity and generalisation, resulting in higher-quality knowledge for KD. The enriched teachers further strengthen multi-teacher ensemble distillation, while logit selection techniques help mitigate capacity mismatches and optimise knowledge transfer. Experimental results demonstrate a 1.484 percentage-point improvement in accuracy over the state-of-the-art ensemble distillation setup, highlighting the efficacy of the SIT approach in teacher ensemble enrichment to improve ensemble KD efficiency, particularly in low-resource ASC scenarios.