Spaces:
Running
on
Zero
Running
on
Zero
File size: 4,128 Bytes
600759a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
# Data Processing
This is the data processing pipeline for 3D shape and texture generation.
**Notes**:
1. This implementation is a simplified version of our industrial pipeline.
2. The rendering script is based on [TRELLIS](https://github.com/microsoft/TRELLIS/blob/main/dataset_toolkits/blender_script/render.py).
## Rendering
### Motivation
The rendering script `render/render.py` serves three main purposes:
1. Converting complex 3D formats to PLY files using Blender for further processing.
2. Rendering condition images for DiT training.
3. Rendering orthogonal images, PBR materials, and conditional signals (world-space normals and positions) for texture generation.
### Requirements
The rendering scripts are executed with Blender 4.1. You need to install `opencv`, `OpenEXR`, and `Imath` using Blender's Python. Here is an example for a Macbook:
```bash
/Applications/Blender.app/Contents/Resources/4.1/python/bin/python3.11 -m pip install OpenEXR Imath opencv-python
```
### Execution
The first two purposes can be executed with a single command:
```bash
$BLENDER_PATH -b -P render/render.py -- \
--object ${INPUT_FILE} --geo_mode --resolution 512 \
--output_folder $OUTPUT_FOLDER
```
For the third purpose, simply remove the `--geo_mode` flag.
## Watertight Mesh Processing and Sampling
### Motivation
To learn an SDF representation for 3DShape2VecSets, we require a watertight input mesh. This pipeline processes raw triangle meshes to generate three essential data types:
1. **Surface samples** - Input points for the encoder.
2. **Volume samples** - Query points for SDF evaluation in the decoder.
3. **Volume SDFs** - Ground-truth signed distance values for VAE training.
### Execution
Process a triangle mesh (OBJ/OFF format) to generate:
1. Watertight mesh (`${OUTPUT_NAME}_watertight.obj`).
2. Surface point samples (`${OUTPUT_NAME}_surface.npz`).
3. Volume samples with SDFs (`${OUTPUT_NAME}_sdf.npz`).
**Command:**
```bash
python3 watertight/watertight_and_sample.py \
--input_obj ${INPUT_MESH} \
--output_prefix ${OUTPUT_NAME}
```
### Output Data Format
#### 1. Surface Samples (`${OUTPUT_NAME}_surface.npz`)
Contains two point cloud arrays in numpy NPZ format:
| Key | Shape | Format | Description |
|-----------------|----------|----------|---------------------------------|
| `random_surface` | `(N, 6)` | `float16`| Uniform point samples on surface |
| `sharp_surface` | `(M, 6)` | `float16`| Samples near sharp mesh edges |
#### 2. Volume SDF Samples (`${OUTPUT_NAME}_sdf.npz`)
Contains three sample types stored as array pairs. For each type `${type}`:
| Sample Type | Points Array | SDF Labels Array | Shape | Format | Description |
|-----------------|----------------------|----------------------|----------|----------|-------------------------|
| `vol` | `vol_points` | `vol_label` | `(P, 3)/(P,)` | `float16`| Random spatial samples |
| `random_near` | `random_near_points` | `random_near_label` | `(Q, 3)/(Q,)` | `float16`| Samples near surface |
| `sharp_near` | `sharp_near_points` | `sharp_near_label` | `(R, 3)/(R,)` | `float16`| Samples near sharp edges |
**Data Specifications**:
- All point coordinates (`*_points` arrays) contain 3D positions stored as `float16` values.
- All SDF values (`*_label` arrays) are `float16` scalars representing:
- **Positive values**: Outside the surface.
- **Negative values**: Inside the surface.
- **Zero values**: On the surface.
- Array dimensions:
- `N`, `M`, `P`, `Q`, `R` represent sample counts (vary per shape).
- `3` indicates XYZ coordinates.
- `6` indicates XYZ/Normal coordinates.
- All arrays are stored uncompressed in numpy's NPZ format.
## Overall Script
Modify the first four variables in `pipeline.sh`:
1. **INPUT_FILE** The path to each 3D data file.
2. **OUTPUT_FOLDER** The overall path for the output dataset.
3. **NAME** The naming for the output path of each data.
4. **BLENDER_PATH** The executable path for Blender.
Then run the following script:
```bash
bash pipeline.sh
``` |