[1603.08155] Perceptual Losses for Real-Time Style Transfer and Super-Resolution

We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a \emph{per-pixel} loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing \emph{perceptual} loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results.

arXiv:1603.08155v1 [cs.CV] 27 Mar 2016 Perceptual Losses for Real-Time Style Transfer and Super-Resolution Justin Johnson, Alexandre Alahi, Li Fei-Fei {jcjohns, alahi, feifeili}@cs.stanford.edu Department of Computer Science, Stanford University Abstract. We consider image transformation…

related reading

• Feature-wise transformationsdistill.pub
• [1606.09282] Learning without Forgettingarxiv.org
• ImageNet Classification with Deep Convolutional Neural Networksproceedings.neurips.cc
• [1810.13128] The Effect of Learning Strategy versus Inherent Architecture Properties on the Ability of Convolutional Neural Networks to Develop Transformation Invariancearxiv.org
• d89a66c7c80a29b1bdbab0f2a1a94af8-Paper.pdfproceedings.neurips.cc
• [2203.09795] Three things everyone should know about Vision Transformersarxiv.org
• [2006.11239] Denoising Diffusion Probabilistic Modelsarxiv.org
• [2205.13147] Matryoshka Representation Learningarxiv.org
• https://arxiv.org/pdf/2006.11239arxiv.org
• Stand-Alone Self-Attention in Vision Models - NeurIPS-2019-stand-alone-self-attention-in-vision-models-Paper.pdfpapers.nips.cc
• [2103.03230] Barlow Twins: Self-Supervised Learning via Redundancy Reductionarxiv.org
• Superquadrics Revisited: Learning 3D Shape Parsing beyond Cuboidsarxiv.org