Efficient Structure-Aware 3D Gaussians via Lightweight Information Shaping
Mutual Information Maximization; 3D Reconstruction; 3D Editing
Yes
No ethics review needed.
Please refer to the weaknesses section.
Previous approaches embed semantic features directly within each Gaussian primitive, requiring extensive manipulation of individual Gaussians to edit scenes. This paper instead proposes encoding mutual information within an attribute decoding network, allowing scene edits to be made by directly adjusting the network parameters.
This paper presents a method for correlating Gaussians that belong to the same semantic group. It leverages a network to decode feature Gaussians into attributes and approximates mutual information via gradients with respect to network parameters. This approach enables editing operations through adjustments to the network parameters rather than direct manipulation of numerous Gaussians. The mutual information between Gaussians is approximated by shaping activations from perturbed network weights.
The paper is generally challenging to follow, with many points lacking clear explanations. For example, Equation 7 and Line 304 state that "we find that ∂h corresponds to repeated activations σ(h(l−1))." However, this lacks sufficient context, as does Line 317, which mentions that "shaping activations ∂h within the attribute decoding network Φa supports a sequence of editing operations." Further explanation on why shaping these activations enables multiple edits would improve clarity.

The proposed method uses an MLP network to model mutual information between Gaussians, where the Gaussians primarily serve to splat features onto images used as inputs for the MLP. The primary contribution appears to be an enhancement to the JacobiNeRF framework, rather than a significant innovation in the context of 3D Gaussian Splatting (3DGS).

The contributions of Section 3.5 on coarse mask guidance and Section 3.6 on smoothness regularization seem minor, primarily focused on refining supervision masks. These sections could potentially be merged, as similar techniques have been explored in prior work.

The method depends on a 2D video mask tracker for grouping Gaussians. Based on the experimental results, it is difficult to determine if performance improvements arise from the mask tracker or the proposed approach. For instance, in Figure 4, the poor mask quality generated by Gaussian Grouping may stem from suboptimal 2D masks.
Yes
No ethics review needed.
Does the decoding network reduce the rendering efficiency?
- This paper adapts the basic idea of JacobiNeRF to 3DGS and makes it work in terms of scene editing.
- The method achieves good performance in the open vocabulary segmentation task.
- The application of object removal seems have good qualitative performance.
This paper proposes information shaping for the 3D Gaussian field, following NeRF-based work JacobiNeRF. By contrastive learning between Gaussian primitives, the resulting Gaussian primitives on the same entities have a strong correlation while primitives on different entities have a weak correlation. The technical part follows the basic idea of JacobiNeRF and is tailored to 3D Gaussian with a attributes decoding network.  The learned Gaussian field is easier to edit, such as object removal and object movement.
- The technical novelty is limited. The Jacobian-based mutual information learning is the core part of this paper, which mainly follows JacobiNeRF. The authors are suggested to emphasize the unique designs that are different from the JacobiNeRF and demonstrate the effectiveness of the unique designs. The mask guidance and smoothness regularization differ from the DINO adopted in JacobiNeRF. However, these components seem to be incremental techniques, and their effectiveness is not well validated in the experiments.
- The writing should be improved. Many concepts lack a clear definition. For example, what are the exact definitions of perturbation and the sequential of perturbations (e.g., 1st/2nd/3rd perturbation). What are the exact parameters perturbated and how are they perturbated (i.e., perturbation direction and norm).
- The demonstration is weak. The object movement in Figure 6 is not impressive and it seems to be a sample handpicked with effort. It would be better to present more object movement demonstrations, using the main components of a 3D scene, such as the earthmover toy in the kitchen scene. Can you move the earthmover to anywhere else on the desk? Furthermore, the demonstrations have low diversity, mainly adopting the scene of kitchen and bear.
- The method outperforms JacobiNeRF and JacobiGS. However, the authors do not show the reasons via an apple-to-apple comparison with JacobiNeRF and JacobiGS. As I suggested in the first item, the authors are encouraged to show the performance roadmap from the JacobiGS baseline (or other reasonable baseline) and add the unique components one by one, offering a better understanding of the inner workings to readers.
Yes
No ethics review needed.
1. Aside from JacobiGS in the paper, which utilizes the same mutual information shaping loss as JacobiNeRF, why was the original JacobiNeRF not included as a baseline for comparison?


2. Regarding weakness 2, is it possible for the proposed method to edit the color of an object to a textured color? Additionally, for the object movement task, can the method modify the trajectory of a selected object to follow a position-related trajectory, such as rotation?


3. Besides open-vocabulary segmentation tasks, would it be feasible to conduct experiments on semantic or instance segmentation tasks to further enhance the credibility of the findings?
1. The paper is well-written and organized, the authors provide a thorough theoretical analysis of the proposed method.
2. Experiments demonstrate that the proposed method outperforms other baselines while maintaining robustness to incorrect masks.
This paper introduces a mutual information technique based on 3D Gaussian splatting. Building upon JacobiNeRF, the authors present a more efficient training pipeline that optimizes activations instead of Jacobians. This approach ensures that the correlations between Gaussians are correctly shaped and remain consistent through successive parameter changes. Experiments on open-vocabulary segmentation and scene editing demonstrate the efficiency and significant capabilities of the proposed method.
1. Compared to methods that edit the scene after obtaining the labels for each Gaussian, such as Gaussian Grouping, the direct manipulation of network parameters appears less flexible. For instance, while Gaussian Grouping can achieve style transfer, the proposed method seems limited to modifying the colors of selected Gaussians to be uniform.

2. The authors conduct open-vocabulary segmentation in only three scenes, evaluating just one object per scene if I understand correctly. Including more experiments would enhance the robustness and credibility of the results.
@inproceedings{
anonymous2024efficient,
title={Efficient Structure-Aware 3D Gaussians via Lightweight Information Shaping},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Pj2qEVzufH},
note={under review}
}
3D Gaussians, 
as an explicit scene representation, 
typically involve 
thousands to millions of elements per scene. 
This makes it 
challenging to control 
the scene in ways 
that reflect the underlying semantics, 
where the number of independent entities 
is typically much smaller. 
Especially, 
if one wants to animate or edit objects in the scene, 
as this requires coordination among the many Gaussians
involved in representing each object. 
To address this issue, 
we develop a mutual information shaping technique 
that enforces resonance and coordination
between correlated Gaussians 
via a Gaussian attribute decoding network. 
Such correlations can be learned 
from putative 2D object masks in different views. 
By approximating the 
mutual information with 
the gradients concerning the network parameters, 
our method ensures consistency 
between scene elements 
and enables efficient scene editing 
by operating on network parameters rather than massive Gaussians.
In particular, 
we develop an effective contrastive learning pipeline
with lightweight optimization to shape the attribute decoding network,
while ensuring that the shaping (consistency) is maintained during continuous edits, 
avoiding re-shaping after parameter changes. 
Notably, 
our training only touches 
a small fraction of all Gaussians in the scene 
yet attains the desired correlated behavior 
according to the underlying scene structure. 
The proposed technique 
is evaluated on challenging scenes and demonstrates significant performance improvements 
in 3D object segmentation and promoting scene interactions, 
while inducing 
low computation and memory requirements. 
Our code and trained models 
will be made available.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Mutual Information Maximization; 3D Reconstruction; 3D Editing
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a358c4ce4f19221f42ffe32c84a4956995486724.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/3b6703b8d8c9d77109b0866829dc3a96616eab17.pdf
Efficient Structure-Aware 3D Gaussians via Lightweight Information Shaping
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission