Energy-Efficient Interactive 360° Video Streaming with Real-Time Gaze Tracking on Mobile Devices

Abstract

360° videos are becoming one of the major media in recent years, providing immersive experience for viewers with more interactions compared to traditional videos. Most of today's implementations rely on bulky Head-Mounted Displays (HMDs) or require touch screen operations for interactive display, which are not only expensive but also inconvenient for viewers. In this paper, we demonstrate that interactive 360° video streaming can be done with hints from gaze movement detected by the front camera of today's mobile devices (e.g., a smartphone). We design a lightweight real-time gaze point tracking method for this purpose. Using only the front camera, our solution detects the users' faces by a lightweight Haar-like cascaded classifier, measures the user's face-to-screen distance and sight angle, and then derives the location of the user's gaze point following a customized triangularity model. We integrate it with streaming module and apply a dynamic margin adaption algorithm to minimize the overall energy consumption for battery-constrained mobile devices. Our experiments on state-of-the-art smartphones show the feasibility of our solution and its energy efficiency toward cost-effective real-time 360° video streaming.