Spaces:
Runtime error
Runtime error
Stanislaw Szymanowicz
commited on
Commit
·
15f992e
1
Parent(s):
11ffd30
Add examples
Browse files- app.py +58 -12
- demo_examples/01_bigmac.png +0 -0
- demo_examples/02_hydrant.jpg +0 -0
- demo_examples/03_spyro.png +0 -0
- demo_examples/04_lysol.png +0 -0
- demo_examples/05_pinapple_bottle.png +0 -0
- demo_examples/06_unsplash_broccoli.png +0 -0
- demo_examples/07_objaverse_backpack.png +0 -0
- demo_examples/08_unsplash_chocolatecake.png +0 -0
- demo_examples/09_realfusion_cherry.png +0 -0
- demo_examples/10_triposr_teapot.png +0 -0
app.py
CHANGED
|
@@ -95,22 +95,23 @@ def main():
|
|
| 95 |
|
| 96 |
return ply_out_path
|
| 97 |
|
| 98 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
gr.Markdown(
|
| 100 |
"""
|
|
|
|
| 101 |
|
| 102 |
-
|
| 103 |
-
[Splatter Image](https://github.com/szymanowiczs/splatter-image) (CVPR 2024) is a fast, super cheap to train method for object 3D reconstruction from a single image.
|
| 104 |
The model used in the demo was trained on **Objaverse-LVIS on 2 A6000 GPUs for 3.5 days**.
|
| 105 |
-
|
| 106 |
-
Upload an image of an object to see how the Splatter Image does.
|
| 107 |
-
|
| 108 |
-
**Comments:**
|
| 109 |
-
1. The first example you upload should take about 4.5 seconds (with preprocessing, saving and overhead), the following take about 1.5s.
|
| 110 |
-
2. The model does not work well on photos of humans.
|
| 111 |
-
3. The 3D viewer shows a .ply mesh extracted from a mix of 3D Gaussians. Artefacts might show - see video for more faithful results.
|
| 112 |
-
4. Best results are achieved on the datasets described in the [repository](https://github.com/szymanowiczs/splatter-image) using that code. This demo is experimental.
|
| 113 |
-
5. Our model might not be better than some state-of-the-art methods, but it is of comparable quality and is **much** cheaper to train and run.
|
| 114 |
"""
|
| 115 |
)
|
| 116 |
with gr.Row(variant="panel"):
|
|
@@ -131,6 +132,26 @@ def main():
|
|
| 131 |
)
|
| 132 |
with gr.Row():
|
| 133 |
submit = gr.Button("Generate", elem_id="generate", variant="primary")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
with gr.Column():
|
| 135 |
with gr.Row():
|
| 136 |
with gr.Tab("Reconstruction"):
|
|
@@ -140,6 +161,31 @@ def main():
|
|
| 140 |
interactive=False
|
| 141 |
)
|
| 142 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 143 |
submit.click(fn=check_input_image, inputs=[input_image]).success(
|
| 144 |
fn=preprocess,
|
| 145 |
inputs=[input_image, preprocess_background],
|
|
|
|
| 95 |
|
| 96 |
return ply_out_path
|
| 97 |
|
| 98 |
+
css = """
|
| 99 |
+
h1 {
|
| 100 |
+
text-align: center;
|
| 101 |
+
display:block;
|
| 102 |
+
}
|
| 103 |
+
"""
|
| 104 |
+
|
| 105 |
+
with gr.Blocks(css=css) as demo:
|
| 106 |
gr.Markdown(
|
| 107 |
"""
|
| 108 |
+
# Splatter Image
|
| 109 |
|
| 110 |
+
**Splatter Image (CVPR 2024)** [[code](https://github.com/szymanowiczs/splatter-image), [project page](https://szymanowiczs.github.io/splatter-image)] is a fast, super cheap-to-train method for object 3D reconstruction from a single image.
|
|
|
|
| 111 |
The model used in the demo was trained on **Objaverse-LVIS on 2 A6000 GPUs for 3.5 days**.
|
| 112 |
+
Locally, on an NVIDIA V100 GPU, reconstruction (forward pass of the network) can be done at 38FPS and rendering (with Gaussian Splatting) at 588FPS.
|
| 113 |
+
Upload an image of an object or click on one of the provided examples to see how the Splatter Image does.
|
| 114 |
+
For best results clone the [main repository](https://github.com/szymanowiczs/splatter-image) and run the demo locally.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 115 |
"""
|
| 116 |
)
|
| 117 |
with gr.Row(variant="panel"):
|
|
|
|
| 132 |
)
|
| 133 |
with gr.Row():
|
| 134 |
submit = gr.Button("Generate", elem_id="generate", variant="primary")
|
| 135 |
+
|
| 136 |
+
with gr.Row(variant="panel"):
|
| 137 |
+
gr.Examples(
|
| 138 |
+
examples=[
|
| 139 |
+
'./demo_examples/01_bigmac.png',
|
| 140 |
+
'./demo_examples/02_hydrant.jpg',
|
| 141 |
+
'./demo_examples/03_spyro.png',
|
| 142 |
+
'./demo_examples/04_lysol.png',
|
| 143 |
+
'./demo_examples/05_pinapple_bottle.png',
|
| 144 |
+
'./demo_examples/06_unsplash_broccoli.png',
|
| 145 |
+
'./demo_examples/07_objaverse_backpack.png',
|
| 146 |
+
'./demo_examples/08_unsplash_chocolatecake.png',
|
| 147 |
+
'./demo_examples/09_realfusion_cherry.png',
|
| 148 |
+
'./demo_examples/10_triposr_teapot.png'
|
| 149 |
+
],
|
| 150 |
+
inputs=[input_image],
|
| 151 |
+
cache_examples=False,
|
| 152 |
+
label="Examples",
|
| 153 |
+
examples_per_page=20,
|
| 154 |
+
)
|
| 155 |
with gr.Column():
|
| 156 |
with gr.Row():
|
| 157 |
with gr.Tab("Reconstruction"):
|
|
|
|
| 161 |
interactive=False
|
| 162 |
)
|
| 163 |
|
| 164 |
+
gr.Markdown(
|
| 165 |
+
"""
|
| 166 |
+
## Comments:
|
| 167 |
+
1. If you run the demo online, the first example you upload should take about 4.5 seconds (with preprocessing, saving and overhead), the following take about 1.5s.
|
| 168 |
+
2. The 3D viewer shows a .ply mesh extracted from a mix of 3D Gaussians. This is only an approximations and artefacts might show.
|
| 169 |
+
3. Known limitations include:
|
| 170 |
+
- sphere-like artefacts on the object and white halo around it: this is due to how the .ply mesh is extracted and limitations of the Gradio viewer
|
| 171 |
+
- see-through parts of objects, especially on the back: this is due to the model performing less well on more complicated shapes
|
| 172 |
+
- back of objects are blurry: this is a model limiation due to it being deterministic
|
| 173 |
+
4. Our model is of comparable quality to state-of-the-art methods, and is **much** cheaper to train and run.
|
| 174 |
+
|
| 175 |
+
## How does it work?
|
| 176 |
+
|
| 177 |
+
Splatter Image formulates 3D reconstruction as an image-to-image translation task. It maps the input image to another image,
|
| 178 |
+
in which every pixel represents one 3D Gaussian and the channels of the output represent parameters of these Gaussians, including their shapes, colours and locations.
|
| 179 |
+
The resulting image thus represents a set of Gaussians (almost like a point cloud) which reconstruct the shape and colour of the object.
|
| 180 |
+
The method is very cheap: the reconstruction amounts to a single forward pass of a neural network with only 2D operators (2D convolutions and attention).
|
| 181 |
+
The rendering is also very fast, due to using Gaussian Splatting.
|
| 182 |
+
Combined, this results in very cheap training and high-quality results.
|
| 183 |
+
For more results see the [project page](https://szymanowiczs.github.io/splatter-image) and the [CVPR article](https://arxiv.org/abs/2312.13150).
|
| 184 |
+
"""
|
| 185 |
+
)
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
|
| 189 |
submit.click(fn=check_input_image, inputs=[input_image]).success(
|
| 190 |
fn=preprocess,
|
| 191 |
inputs=[input_image, preprocess_background],
|
demo_examples/01_bigmac.png
ADDED
|
demo_examples/02_hydrant.jpg
ADDED
|
demo_examples/03_spyro.png
ADDED
|
demo_examples/04_lysol.png
ADDED
|
demo_examples/05_pinapple_bottle.png
ADDED
|
demo_examples/06_unsplash_broccoli.png
ADDED
|
demo_examples/07_objaverse_backpack.png
ADDED
|
demo_examples/08_unsplash_chocolatecake.png
ADDED
|
demo_examples/09_realfusion_cherry.png
ADDED
|
demo_examples/10_triposr_teapot.png
ADDED
|