Space-time video super-resolution based on temporal feature refinement network

Abstract

Space-time video super-resolution( STVSR) enhances video quality across both temporal and spatial dimensions, enabling real-time presentation of high-resolution and high-frame-rate videos despite limitations in video capture devices, transmission, or storage, thus meeting the demand for ultra-high-definition image quality. Compared to two-stage methods, one-stage approaches achieve frame interpolation at the feature level rather than the pixel level,significantly outperforming in terms of inference speed and computational complexity. Some existing one-stage STVSR methods employ pixel hallucination-based feature interpolation, which struggles to predict fast-moving objects between frames. To address this, a pyramid encoder-decoder network based on optical flow for temporal feature interpolation is proposed, to achieve rapid bidirectional optical flow estimation and more realistic texture synthesis. This network structure, termed temporal feature refinement network(TFRnet), enhances efficiency while mitigating the instability of optical flow estimation for large motions. Additionally, the spatial module incorporates sliding window-based local propagation and bidirectional propagation based on recurrent networks to strengthen frame alignment. To further exploit TFRnet′s potential, spatial super-resolution is prioritized over temporal super-resolution(space-first approach).Experiments on several widely used data benchmarks and evaluation metrics demonstrate the excellent performance of our proposed method, TFRnet-sf. While improving overall peak signal-to-noise ratio(PSNR) and structural similarity index(SSIM), it also enhances PSNR and SSIM for inserted intermediate frames, alleviating to some extent the issue of significant disparities in PSNR and SSIM between inserted intermediate frames and original frames.