Computational Media Intelligence: Human-Centered Machine Analysis of Media

Abstract

Media is created by humans for humans to tell stories. There exists a natural and imminent need for creating human-centered media analytics to illuminate the stories being told and to understand their impact on individuals and society at large. An objective understanding of media content has numerous applications for different stakeholders, from creators to decision-/policy-makers to consumers. Advances in multimodal signal processing and machine learning (ML) can enable detailed and nuanced characterization of media content (of who, what, how, where, and why) at scale. They can also aid our understanding of the impact of media on a range of issues, including individual experiences, behavioral, cultural, and societal trends, and commercial outcomes. Modern deep learning models combined with audiovisual signal processing can analyze entertainment media, such as Film & TV content to quantify gender, age, and race representations. This creates awareness in an objective way that was hitherto impossible. On the other hand, text mining and natural language processing allow nuanced understanding of language use and spoken interactions in media, such as News to track patterns and trends across different contexts. Moreover, advances in human sensing have enabled us to directly measure the influence of media on an individual’s physiology (and brain), while social media analysis enables tracking the societal impact of media content on different cross sections of the society. This article reviews representative methodologies and algorithms, tools, and systems advancing human-centered media understanding through ML in the pursuit of developing computational media intelligence.