Farm3D: Learning Articulated 3D Animals by Distilling 2D Diffusion

    Tomas Jakab*, Ruining Li*, Shangzhe Wu, Christian Rupprecht, Andrea Vedaldi
    University of Oxford
    (* equal contribution)
    In 3DV 2024

    Monocular 3D reconstruction

    Input image

    3D shape

    Textured

    Controllable synthesis

    Animation

    Relighting

    Texture swapping

    Farm3D learns an articulated object category entirely from "free" virtual supervision from a 2D diffusion-based image generator. We propose a framework that employs an image generator, such as Stable Diffusion, to produce training data for learning a reconstruction network from the ground up. Additionally, the diffusion model is incorporated as a scoring mechanism to further improve learning. Our method yields a monocular reconstruction network capable of generating controllable 3D assets from a single input image, whether real or generated, in a matter of seconds.

    Approach

    We prompt Stable Diffusion for virtual views of an object category that are then used to train a monocular articulated object reconstruction model that factorises the input image of an object instance into articulated shape, appearance (albedo and diffuse and ambient intensities), viewpoint, and light direction. During training, we also sample synthetic instance views that are then "critiqued" by Stable Diffusion to guide the learning.

    Single-view 3D Reconstruction

    Farm3D reconstructs the shape of a wide range of categories to their fine details such as legs and ears despite not being trained on any real images.

    Controllable 3D Shape Synthesis

    Our method enables the generation of controllable 3D assets from either a real image or an image synthesised using Stable Diffusion. Once generated, we have the ability to adjust lighting, swap textures between models of the same category, and even animate the shape.

    Animation

    Input image

    Animation

    Input image

    Animation

    Input image

    Animation

    Relighting

    Input image

    Relit

    Input image

    Relit

    Input image

    Relit

    Texture Swapping

    Input image

    Texture image

    Texture swapping

    Input image

    Texture image

    Texture swapping

    Animodel - 3D Articulated Animals Dataset and Benchmark

    We introduce a new 3D Articulated Animals Dataset to directly evaluate the quality of single-view 3D reconstruction of articulated animals. This dataset includes textured 3D meshes of articulated animals, including horses, cows, and sheep, crafted by a professional 3D artist, and is accompanied by realistic articulated poses.

    BibTeX

    @Article{jakab2023farm3d,
        title={{Farm3D}: Learning Articulated 3D Animals by Distilling 2D Diffusion},
        author={Jakab, Tomas and Li, Ruining and Wu, Shangzhe and Rupprecht, Christian and Vedaldi, Andrea},
        journal={arXiv preprint arXiv:2304.10535},
        year={2023},
    }
    
    主站蜘蛛池模板: 精品国产亚洲第一区二区三区| 日韩AV无码一区二区三区不卡 | 亚洲国产av一区二区三区丶| 日本精品一区二区久久久| 99精品一区二区三区| 国产一区二区福利久久| 韩国理伦片一区二区三区在线播放| 国产福利电影一区二区三区,日韩伦理电影在线福 | 亚洲A∨无码一区二区三区| 无码日韩人妻AV一区免费l | 精品乱码一区内射人妻无码| 亚洲AV色香蕉一区二区| 日韩有码一区二区| 日本免费一区二区三区最新| 中文乱码字幕高清一区二区| 亚洲国产美女福利直播秀一区二区| 手机看片一区二区| 国产精品污WWW一区二区三区| 精品一区二区三区AV天堂| 久久精品日韩一区国产二区| 国产精品无码一区二区三区不卡 | 久久99国产精品一区二区| 爆乳熟妇一区二区三区| 亚洲乱码国产一区网址| 日韩熟女精品一区二区三区 | 亚洲av乱码一区二区三区| 日韩在线不卡免费视频一区| 国产精品女同一区二区久久| 在线观看国产一区| 亚洲av无码一区二区三区网站| 亚洲爆乳精品无码一区二区三区 | 在线播放偷拍一区精品| 韩国一区二区三区| 国产A∨国片精品一区二区| 在线观看免费视频一区| 无码人妻精品一区二区三区99仓本| 人妻无码久久一区二区三区免费| 欲色aV无码一区二区人妻 | 国产日韩一区二区三免费高清| 鲁丝片一区二区三区免费| 亚洲av乱码一区二区三区香蕉|