RGB$\leftrightarrow$X: Image decomposition and synthesis using material- and lighting-aware diffusion models

arXiv:2405.00666v1 Announce Type: new
Abstract: The three areas of realistic forward rendering, per-pixel inverse rendering, and generative image synthesis may seem like separate and unrelated sub-fields of graphics and vision. However, recent work has demonstrated improved estimation of per-pixel intrinsic channels (albedo, roughness, metallicity) based on a diffusion architecture; we call this the RGB$\rightarrow$X problem. We further show that the reverse problem of synthesizing realistic images given intrinsic channels, X$\rightarrow$RGB, can also be addressed in a diffusion framework.
Focusing on …

abstract architecture arxiv channels cs.cv cs.gr diffusion diffusion models fields generative graphics however image intrinsic inverse rendering lighting material per pixel rendering synthesis type vision work