Make It So: Steering StyleGAN for Any Image Inversion and Editing

Anand Bhattad, Viraj Shah, Derek Hoiem, David A. Forsyth
University of Illinois Urbana-Champaign
Paper arXiv Video Code

Make It So can edit diverse out-of-domain images by using a StyleGAN model trained on bedroom images. This includes images of people, cars, animals, and outdoor scenes. Unlike PTI, a leading inversion method that tends to generate unrealistic bedroom images after editing, Make It So produces more realistic results (use the slider to compare).

Abstract

StyleGAN's disentangled style representation enables powerful image editing by manipulating the latent variables, but accurately mapping real-world images to their latent variables (GAN inversion) remains a challenge. Existing GAN inversion methods struggle to maintain editing directions and produce realistic results.

To address these limitations, we propose Make It So, a novel GAN inversion method that operates in the Z (noise) space rather than the typical W (latent style) space. Make It So preserves editing capabilities, even for out-of-domain images. This is a crucial property that was overlooked in prior methods.

Our quantitative evaluations demonstrate that Make It So outperforms the state-of-the-art method Pivotal Tuning Inversion by a factor of five in inversion accuracy and achieves ten times better edit quality for complex indoor scenes.

Qualitative Results

(1) Target Scene (2) Make It So (3) Relighted (4) Resurfacing
(1) Target Scene (2) Make It So (3) Relighted (4) Resurfacing
(1) Target Scene (2) Make It So (3) Relighted (4) Resurfacing
Target Image Make It So Edit-1 (Relighting) Edit-2 (Recoloring)

Inversion and Editing of Complex Indoor Scenes




Target Image Make It So Edited Edited


Inversion and Editing of Faces

Differences between Make It So and Previous Methods

Image description

Make It So inverts in the noise space, which is different from previous methods that invert in the latent style space of StyleGAN.

BibTeX

@article{bhattad2023makeitso,
  author    = {Bhattad, Anand and Shah, Viraj and Hoeim, Derek and Forsyth, D.A.},
  title     = {Make It So: Steering StyleGAN for Any Image Inversion and Editing},
  journal   = {arXiv},
  year      = {2023},
}