On Evaluating Video-based Generative Adversarial Networks (GANs)

Abstract

We study the problem of evaluating video-based Generative Adversarial Networks (GANs) by applying existing image quality assessment methods to the explicit evaluation of videos generated by state-of-the-art frameworks [1]-[3]. Specifically, we provide results and discussion on using quantitative methods such as the Fréchet Inception Distance [4], the Multi-scale Structural Similarity Measure (MS-SSIM) [5], as well as the Birthday Paradox inspired test [6] and compare these to the prevalent performance evaluation methods in the literature. We summarize that current testing methodologies are not sufficient for quality assurance in video-based GAN frameworks, and that methods based on the image-based GAN literature can be useful to consider. The results of our experiments and a discussion on evaluating video-based GANs provide key insight that may be useful in generating new measures of quality assurance in future work.