CATLAB

Recent Publications

1. ICCV

   LatentCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions

   Oguz Kaan Yuksel*, Enis Simsar*, Ezgi Gulperi Er, and Pinar Yanardag

   International Conference on Computer Vision (ICCV) , 2021

   Abstract PDF Code Project Page

   Recent research has shown great potential for finding interpretable directions in the latent spaces of pre-trained Generative Adversarial Networks (GANs). These directions provide controllable generation and support a wide range of semantic editing operations such as zoom or rotation. The discovery of such directions is often performed in a supervised or semi-supervised fashion and requires manual annotations, limiting their applications in practice. In comparison, unsupervised discovery enables finding subtle directions a priori hard to recognize. In this work, we propose a contrastive-learning-based approach for discovering semantic directions in the latent space of pretrained GANs in a self-supervised manner. Our approach finds semantically meaningful dimensions compatible with state-of-the-art methods.

2. WACV

   StyleMC: Multi-Channel Based Fast Text-Guided Image Generation and Manipulation

   Umut Kocasari, Alara Dirik, Mert Tiftikci, and Pinar Yanardag

   Winter Conference on Applications of Computer Vision (WACV) , 2021

   Abstract COMING SOON

   Pre-trained GANs have shown great potential for interpretable directions in the latent space. The discovery of such directions is often done in a supervised or self-supervised manner and requires manual annotations which limits their application in practice. On the other hand, unsupervised approaches provide a way to discover interpretable directions without any supervision, but no fine-grained attribute can be discovered. Recent work such as StyleCLIP aims to overcome this limitation by leveraging the power of CLIP, a joint representational model for text and images, for text-driven image manipulation. While promising, these methods take several hours of pre-processing or training time, and require multiple text prompts. In this work, we propose a fast and efficient method for text-guided image generation and manipulation by leveraging the power of StyleGAN2 and CLIP. Our method uses a CLIP-based loss and an identity loss to manipulate images via user-supplied text prompts without changing any of the irrelevant attributes. Unlike previous work, our method requires only 12 seconds of optimization per text prompt and can be used with any pre-trained StyleGAN2 model. We demonstrate the effectiveness of our method with extensive results and comparisons to state-of-the-art approaches.

3. ICCV Workshop

   Graph2Pix: A Graph-Based Image to Image Translation Framework

   Dilara Gokay*, Enis Simsar*, Efehan Atici, Atif Emre Yilmaz, Alper Ahmetoglu, and Pinar Yanardag

   Advances of Image Manipulation (ICCV Workshop) , 2021

   Abstract PDF Code Project Page

   In this paper, we propose a graph-based image-to-image translation framework for generating images. We use rich data collected from the popular creativity platform Artbreeder, where users interpolate multiple GAN-generated images to create artworks. This unique approach of creating new images leads to a tree-like structure where one can track historical data about the creation of a particular image. Inspired by this structure, we propose a novel graph-to-image translation model called Graph2Pix, which takes a graph and corresponding images as input and generates a single image as output. Our experiments show that Graph2Pix is able to outperform several image-to-image translation frameworks on benchmark metrics, including LPIPS (with a 25% improvement) and human perception studies (n=60), where users preferred the images generated by our method 81.5% of the time.

4. NeurIPS Workshop

   Exploring Latent Dimensions of Crowd-sourced Creativity

   Umut Kocasari*, Alperen Bag*, Efehan Atici, and Pinar Yanardag

   Machine Learning for Creativity and Design (NeurIPS Workshop) , 2021

   Abstract COMING SOON

   Recent research showed that it is possible to find directions in the latent spaces of pre-trained GANs. These directions provide controllable generation and support a wide range of semantic editing operations such as zoom-in or rotation. While existing works focus on discovering directions for semantic image editing, we focus on an abstract property: Creativity. Can we manipulate an image to make it more or less creative? We build our work on the largest AI-based creativity platform Artbreeder where users are able to generate unique images using pre-trained GAN models. We explore the latent dimensions of the images generated on this platform and present a novel framework for manipulating images to make them more creative.

5. NeurIPS Workshop

   Controlled Cue Generation for Play Scripts

   Alara Dirik*, Hilal Donmez*, and Pinar Yanardag

   Controllable Generative Modeling in Language and Vision (NeurIPS Workshop) , 2021

   Abstract COMING SOON

   We propose the novel task of theatrical cue generation from dialogues. Using our play scripts dataset, which consists of 775K lines of dialogue and 277K cues, we approach the problem of cue generation as a controlled text generation task and show how cues can be used to amplify the impact of dialogue using a language model conditioned on a dialogue/cue discriminator. In addition, we explore the use of topic keywords and emotions to drive cue generation. Extensive quantitative and qualitative experiments show that language models can be successfully used to generate plausible and attribute-controlled text in highly specialized domains such as theater play scripts.