[2106.14843] CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders

This work presents CLIPDraw, an algorithm that synthesizes novel drawings based on natural language input. CLIPDraw does not require any training; rather a pre-trained CLIP language-image encoder is used as a metric for maximizing similarity between the given description and a generated drawing. Crucially, CLIPDraw operates over vector strokes rather than pixel images, a constraint that biases drawings towards simpler human-recognizable shapes. Results compare between CLIPDraw and other synthesis-through-optimization methods, as well as highlight various interesting behaviors of CLIPDraw, such as satisfying ambiguous text in multiple ways, reliably producing drawings in diverse artistic styles, and scaling from simple to complex visual representations as stroke count is increased. Code for experimenting with the method is available at: this https URL

CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders Kevin Frans1,2 , L. B. Soros1 and Olaf Witkowski1,3,4 1 Cross Labs, Cross Compass Ltd., Tokyo, Japan 2…

related reading

• Replicate - Run AI with an APIreplicate.com
• Surya Narreddisurya.website
• The Illustrated Stable Diffusion – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• CLIP · Hugging Facehuggingface.co
• An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversiontextual-inversion.github.io
• VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Modelsajayj.com
• CLIPasso: Semantically-Aware Object Sketchingclipasso.github.io
• Editing Text in Images with AI | Towards Data Sciencetowardsdatascience.com
• ⭐️ Diffusion Modelsandrewkchan.dev
• Teaching Machines to Draw | 大トロblog.otoro.net
• [2301.13823] Grounding Language Models to Images for Multimodal Generationarxiv.org
• GitHub - CompVis/stable-diffusion: A latent text-to-image diffusion modelgithub.com