Image	Relit-1(w++d1)	Relit-2(w++d2)	Relit-3(w++d3)	Relit-4(w++d4)	Relit-5(w++d5)

Our approach, StyLitGAN, searches for a set of directions (d_i) in StyleGAN’s W style space that when added to w+ style code can change a generated image’s lighting while maintaining its albedo. Our method does not require a per-image search or fine-tuning. The first column shows images generated by a vanilla StyleGAN2, while the other columns show the same scenes relit with the same applied direction. The relighting directions (di) are obtained from a forward selection process to promote diversity in relighting and are not cherry-picked. The illumination of the scene changes significantly, while the geometry and albedo remain unchanged, resulting in realistic images. It is also worth noting that the effects of the directions are consistent across scenes

Abstract

We propose a novel method, StyLitGAN, for relighting and resurfacing generated images in the absence of labeled data. Our approach generates images with realistic lighting effects, including cast shadows, soft shadows, inter-reflections, and glossy effects, without the need for paired or CGI data.

StyLitGAN uses an intrinsic image method to decompose an image, followed by a search of the latent space of a pre-trained StyleGAN to identify a set of directions. By prompting the model to fix one component (e.g., albedo) and vary another (e.g., shading), we generate relighted images by adding the identified directions to the latent style codes. Quantitative metrics of change in albedo and lighting diversity allow us to choose effective directions using a forward selection process. Qualitative evaluation confirms the effectiveness of our method.

Latent Interpolation Videos

User Controllable Relighting

Interactive Relighting

Choice of Decomposition Matters

Real Image Relighting with Make It So

Paper

@InProceedings{StyLitGAN,
                title = {StyLitGAN: Image-based Relighting via Latent Control},
                author = {Bhattad, Anand and Soole, James and Forsyth, D.A.},
                booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
                year = {2024}
            }

Code

Acknowledgements

We thank Aniruddha Kembhavi, Derek Hoiem, Min Jin Chong, and Shenlong Wang for their feedback and suggestions. This material is based upon work supported by the National Science Foundation under Grant No. 2106825 and by a gift from the Boeing Corporation. This webpage template was adpated from some colorful folks.