CLIPasso: Semantically-Aware Object Sketching

Abstraction is at the heart of sketching due to the simple and minimal nature of line drawings. Abstraction entails identifying the essential visual properties of an object or scene, which requires semantic understanding and prior knowledge of high-level concepts. Abstract depictions are therefore challenging for artists, and even more so for machines. We present an object sketching method that can achieve different levels of abstraction, guided by geometric and semantic simplifications. While sketch generation methods often rely on explicit sketch datasets for training, we utilize the remarkable ability of CLIP (Contrastive-Language-Image-Pretraining) to distill semantic concepts from sketches and images alike. We define a sketch as a set of Bézier curves and use a differentiable rasterizer to optimize the parameters of the curves directly with respect to a CLIP-based perceptual loss. The abstraction degree is controlled by varying the number of strokes. The generated sketches demonst

CLIPasso: Semantically-Aware Object Sketching --> CLIPasso: Semantically-Aware Object Sketching CLIPasso: Semantically-Aware Object Sketching --> Yael Vinker 1,2 , Ehsan Pajouheshgar 1 , Jessica Y. Bo 1 , Roman Bachmann 1 , Amit Haim Bermano 2 , Daniel Cohen-Or 2 , Amir Zamir 1 , Ariel Shamir 3 1 Swiss Federal Institute of Technology (EPFL), 2 Tel Aviv University, 3 Reichman University * Indicates Equal Contribution --> SIGGRAPH 2022 (Best Paper Award) * Indicates Equal Contribution --> arXiv ACM Paper Code Colab Demo --> --> Our work converts an image of an object to a sketch, allowing for va

related reading

• [2106.14843] CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encodersarxiv.org
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversiontextual-inversion.github.io
• [2304.06718] Segment Everything Everywhere All at Oncearxiv.org
• Zero-Shot Object Re-Identification in Egocentric Kitchen Videos via Multi-Stage SAM3 Feature Fusionarxiv.org
• Language-Grounded Indoor 3D Semantic Segmentation in the Wildarxiv.org
• Learning 3D object-centric representation through predictionarxiv.org
• Telling Left from Right: Identifying Geometry-Aware Semantic Correspondencetelling-left-from-right.github.io
• DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation | PDFarxiv.org
• VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Modelsajayj.com
• Superquadrics Revisited: Learning 3D Shape Parsing beyond Cuboidsarxiv.org
• Video models are zero-shot learners and reasonersarxiv.org