Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation

Hunyuan3D 2.0: High-Resolution Textured 3D Asset Generation

Introduction

Briefing Document: Hunyuan3D 2.0

Source: Hunyuan3D 2.0 Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation

Purpose: This document summarizes the key aspects of the Hunyuan3D 2.0 system, a large-scale 3D synthesis system for generating high-resolution textured 3D assets.

Authors: Hunyuan3D Team (Multiple contributors listed in the source).

Publication Date: Appears to be January 2025, based on the arXiv identifier (2501.12202).

Core Mission: The Hunyuan3D Team’s mission statement is “Living out everyone’s imagination on creating and manipulating 3D assets.”

What Problem is Addressed?

Hunyuan3D 2.0 is presented as a comprehensive solution to the challenges of creating high-quality, detailed, and usable 3D assets from 2D images, improving upon previous methods through novel architectural designs and strategic pipelines, while also contributing to the open-source community.

Here are the key problems addressed:

1. Generating High-Resolution, High-Quality Textured 3D Assets from Images: The overarching problem is the need for an advanced, large-scale system capable of reliably creating detailed 3D models with vibrant, high-resolution textures based on a single condition image.
2. Limitations of Existing 3D Generative Models: Previous state-of-the-art models, both open-source and closed-source, have limitations in terms of achieving high fidelity in geometry details, accurate alignment with the condition image, and overall texture quality. Hunyuan3D 2.0 aims to surpass these models in these key areas.
3. Capturing High-Frequency Geometric Details: In shape generation, simply using uniform sampling from a mesh surface often fails to reconstruct the intricate details of complex objects, such as edges and corners, due to variations in surface complexity. The Hunyuan3D-ShapeVAE addresses this by incorporating an importance sampling strategy to extract this high-frequency detail information.
4. Generating Seamless and Multi-View Consistent Textures: Creating texture maps that are consistent across multiple viewpoints and seamlessly cover the entire mesh is a significant challenge. Previous methods, particularly training-free approaches leveraging models like Stable Diffusion, often suffer from issues like the Janus problem and multi-view inconsistency, resulting in artifacts. Hunyuan3D-Paint tackles this through its architecture, including a multi-task attention mechanism to ensure consistency across generated views.
5. Preventing Illumination Baking in Textures: Input images often contain significant and varied illumination and shadows, which can be inadvertently baked into the generated texture maps if used directly for multi-view generation. Hunyuan3D-Paint includes an Image Delighting Module to preprocess the input image and remove illumination and shadows, enabling the synthesis of illumination-invariant texture maps.
6. Reducing the Burden of Post-Generation Texture Inpainting: Self-occlusion during the multi-view image generation stage can lead to holes in the generated textures, requiring extensive post-processing for inpainting. Hunyuan3D-Paint alleviates this “burden” by facilitating dense-view inference during the multi-view generation stage, which is supported by a view dropout training strategy.
7. Filling the Gap in Open-Source Large-Scale 3D Foundation Models: The field of 3D generation has lacked robust, large-scale open-source foundational models comparable to those available for image or language generation (like Stable Diffusion or LLaMA). The open-source release of Hunyuan3D 2.0, including its code and pre-trained weights, is specifically intended to address this gap and contribute a valuable resource to the community.

Key Themes and Concepts:

• Large-Scale 3D Synthesis System: Hunyuan3D 2.0 is presented as a comprehensive system for creating 3D assets. It is designed for large-scale applications and focuses on generating high-resolution, textured assets.

• Two Foundation Components: The system is built upon two main models:

  • Hunyuan3D-DiT (Shape Generation Model): This model is responsible for generating the geometry (bare mesh) of a 3D asset based on a given condition image. It utilizes a scalable flow-based diffusion transformer architecture.

  • Hunyuan3D-Paint (Texture Synthesis Model): This model focuses on creating high-resolution and vibrant texture maps for either generated or existing meshes. It leverages geometric and diffusion priors.

• Hunyuan3D-Studio (Production Platform): Beyond the core models, the system includes a user-friendly platform called Hunyuan3D-Studio. This platform aims to simplify the 3D asset creation process and allows users to manipulate and animate their meshes.

• Performance and Comparison: The authors claim that Hunyuan3D 2.0 surpasses previous state-of-the-art models (both open-source and closed-source) in key areas such as geometry details, condition alignment, and texture quality. Numerical comparisons using various metrics (FIDCLIP, CMMD, CLIP-score, LPIPS for texture; V-IoU, S-IoU, ULIP-I, Uni3D-I, ULIP-T, Uni3D-T for shape) are presented to support these claims.

• Open-Source Release: A significant aspect of the project is the public release of Hunyuan3D 2.0, including the code and pre-trained weights, to contribute to the open-source 3D community. The GitHub repository is provided: https://github.com/Tencent/Hunyuan3D-2.

Detailed Breakdown of Key Components and Ideas:

Shape Generation (Hunyuan3D-DiT):

• Employs a latent diffusion model architecture, drawing inspiration from its success in image and video generation.

• Consists of an autoencoder (Hunyuan3D-ShapeVAE) to compress 3D mesh shapes into a sequence of continuous tokens.

• Uses a flow-based diffusion model (Hunyuan3D-DiT) trained in the latent space to predict object tokens from an input image.

• The architecture utilizes both double- and single-stream transformer blocks to enhance interaction between shape and image modalities (Figure 4).

• A key innovation in the ShapeVAE is the use of importance sampling alongside uniform sampling to capture high-frequency details like edges and corners from the input mesh surface (Figure 3).

• The decoder reconstructs the 3D shape using a neural field and the marching cubes algorithm to produce a triangle mesh.

Texture Map Synthesis (Hunyuan3D-Paint):

• Aims to generate high-resolution and seamless texture maps for a given mesh and image prompt.

• Requires the texture map to align with the prompt, be multi-view consistent, and harmonize with the mesh.

• Follows a three-stage framework: pre-processing, multi-view image synthesis, and texture baking. (Figure 5)

• Pre-processing: Includes an Image Delighting Module to remove illumination and shadows from the input image, ensuring illumination-invariant texture synthesis. This is achieved by training on paired images rendered under HDRI environmental maps and even white light.

• View Selection Strategy: Employs a geometry-aware viewpoint selection strategy to efficiently generate texture with minimal viewpoints, aiming for maximum surface coverage. It starts with 4 orthogonal views and iteratively adds more using a greedy search (Algorithm 1).

• Multi-task Attention Mechanism: Introduces two additional attention modules (reference attention and multi-view attention) in parallel to the original self-attention within the diffusion model (Equation 4). * Reference attention integrates the input reference image. * Multi-view attention ensures consistency across generated views.

• Geometry and View Conditioning: Geometry conditions (canonical normal maps and canonical coordinate maps) are concatenated with noise and fed into the model. A learnable camera embedding is also used to provide viewpoint information.

• Texture Baking: Employs dense-view inference during the multi-view generation stage to reduce the need for inpainting holes caused by self-occlusion. A view dropout strategy during training helps the model generalize and support this dense inference.

User-Centric Features:

• Hunyuan3D-Studio: Designed to be a versatile and user-friendly platform, accessible to both professionals and amateurs for manipulating and animating 3D assets.

• Sketch-to-3D Module: The system includes a module capable of generating detailed and realistic 3D assets from simple sketches, described as a “low-barrier creation platform for ordinary users.”

• Low-polygon Stylization Module: A module to convert dense meshes into low-polygon meshes, crucial for many CG pipelines and reducing computational costs. This involves geometric editing and texture preservation.

Evaluation and Results:

• The paper presents quantitative results showing Hunyuan3D 2.0’s superior performance compared to baselines across various metrics.

• A user study is mentioned (Figure 10), suggesting evaluation of the generated assets by users based on criteria like overall quality and image following.

• Visual comparisons (Figure 11) highlight the model’s ability to synthesize detailed geometry (surface bumps, intricate structures) and vivid, correct texture maps, even for complex actions.

Key Strengths and Contributions:

• Comprehensive System: Combines shape and texture generation with a user-friendly platform.

• High-Resolution Output: Focuses on generating detailed and high-resolution assets.

• Improved Geometry and Texture Quality: Demonstrates superior performance through quantitative and visual evaluations.

• Novel Architectural Designs: Includes innovations like importance sampling in ShapeVAE and the multi-task attention mechanism in Hunyuan3D-Paint.

• Illumination-Invariant Texture: The image delighting module addresses a common challenge in texture generation.

• Dense-View Inference for Texture Baking: Reduces dependence on post-generation inpainting.

• Open-Source Contribution: Provides valuable resources to the 3D generation community.

• User Accessibility: Aims to make 3D asset creation more accessible through the Studio and Sketch-to-3D features.

Important Quotes:

• “Living out everyone’s imagination on creating and manipulating 3D assets.” (Hunyuan3D Team motto)

• “We present Hunyuan3D 2.0, an advanced large-scale 3D synthesis system for generating high-resolution textured 3D assets.”

• “The shape generative model, built on a scalable flow-based diffusion transformer, aims to create geometry that properly aligns with a given condition image…”

• “The texture synthesis model, benefiting from strong geometric and diffusion priors, produces high-resolution and vibrant texture maps…”

• “Furthermore, we build Hunyuan3D-Studio – a versatile, user-friendly production platform that simplifies the re-creation process of 3D assets.”

• “Hunyuan3D 2.0 is publicly released in order to fill the gaps in the open-source 3D community for large-scale foundation generative models.”

• “Instead of only using uniform sampling on mesh surface, We have developed an importance sampling strategy to extract high-frequency detail information from the input mesh surface, such as edges and cor-ners.” (Regarding ShapeVAE)

• “To address this issue, we leverage a delighting procedure on the input image via an image-to-image approach before multi-view generation.” (Regarding Texture Synthesis pre-processing)

• “To mitigate potential conflicts arising from these multi-functionalities, we design the additional two distinct attention modules in a parallel structure…” (Regarding Multi-task Attention)

• “In our texture-map synthesis framework, we alleviate the burden on the second stage of inpainting by facilitating dense-view inference during the multi-view generation stage.” (Regarding Texture Baking)

Limitations and Future Work (Not explicitly detailed in these excerpts, but common in research papers): While the document highlights strengths, a full paper would likely discuss limitations, potential biases in training data, computational requirements, and areas for future improvement.

Evaluation Metrics

General Text- and Image-to-Texture

	3DShape2VecSet [111]	Michelangelo [118]	Direct3D [98]	Hunyuan3D-ShapeVAE (Ours)
V-IoU(↑)	87.88%	84.93%	88.43%	93.6%
S-IoU(↑)	80.66%	76.27%	81.55%	89.16%

3D Shape Generation

ULIP-T(↑)	ULIP-I(↑)	Uni3D-T(↑)	Uni3D-I(↑)
Michelangelo [118]	0.0752	0.1152	0.2133	0.2611
Craftsman	1.5	[49]	0.0745	0.1296	0.2375	0.2987
Trellis [100]	0.0769	0.1267	0.2496	0.3116
Shape_Model	1	0.0799	0.1181	0.2469	0.3064
Shape_Model	2	0.0741	0.1308	0.2464	0.3106
Shape_Model	3	0.0746	0.1284	0.2516	0.3131
Hunyuan3D-DiT (Ours)	0.0771	0.1303	0.2519	0.3151

Hunyuan3D-Paint

CMMD(↓)	FIDCLIP (↓)	CLIP-score(↑)	LPIPS(↓)
TEXTure [73]	3.047	35.75	0.8499	0.0076
Text2Tex [9]	2.811	31.72	0.8680	0.0071
SyncMVD [59]	2.584	29.93	0.8751	0.0063
Paint3D [110]	2.810	30.29	0.8724	0.0063
TexPainter [112]	2.483	28.83	0.8789	0.0062
Hunyuan3D-Paint (Ours)	2.318	26.44	0.8893	0.0059

Conclusion:

Hunyuan3D 2.0 represents a significant advancement in the field of 3D asset generation. By combining robust shape and texture synthesis models with a user-friendly platform and releasing its components as open-source, it aims to lower the barrier to entry for 3D content creation and contribute to the broader community. The system demonstrates state-of-the-art performance and incorporates novel techniques to improve the quality and usability of generated 3D assets.