Given only a real single-view image collection of "revolutionary" (i.e., solid of revolution) artefacts with known silhouettes as training data (left), our framework learns to de-render a single image into shape, albedo and complex lighting and material components, allowing for novel-view synthesis and relighting (right).

* This work was primarily done while Shangzhe Wu was interning at Google Research.

Abstract

Recent works have shown exciting results in unsupervised image de-rendering -- learning to decompose 3D shape, appearance, and lighting from single-image collections without explicit supervision. However, many of these assume simplistic material and lighting models. We propose a method, termed RADAR^† (Revolutionary Artefact De-rendering And Re-rendering), that can recover environment illumination and surface materials from real single-image collections, relying neither on explicit 3D supervision, nor on multi-view or multi-light images. Specifically, we focus on rotationally symmetric artefacts that exhibit challenging surface properties including specular reflections, such as vases. We introduce a novel self-supervised albedo discriminator, which allows the model to recover plausible albedo without requiring any ground-truth during training. In conjunction with a shape reconstruction module exploiting rotational symmetry, we present an end-to-end learning framework that is able to de-render the world's revolutionary artefacts. We conduct experiments on a real vase dataset and demonstrate compelling decomposition results, allowing for applications including free-viewpoint rendering and relighting.

^† Note the name itself is a "revolutionary" palindrome.

Video

Results

De-rendering from a single image

Novel view rendering and relighting from a single image

images from The Metropolitan Museum of Art Collection

images from Open Images Dataset

Paper

@InProceedings{wu2021derender,
    author={Shangzhe Wu and Ameesh Makadia and Jiajun Wu and Noah Snavely and Richard Tucker and Angjoo Kanazawa},
    title={De-rendering the World's Revolutionary Artefacts},
    booktitle = {CVPR},
    year = {2021}
}

Acknowledgements

We would like to thank Christian Rupprecht, Soumyadip Sengupta, Manmohan Chandraker and Andrea Vedaldi for insightful discussions. The webpage template was adapted from Richard Zhang's and Jason Zhang's templates.