Mask inconsistency
The multi-view inconsistent 2D segmentation masks introduce significant ambiguity in both feature lifting and mask prediction.
We address the challenge of lifting 2D visual segmentation to 3D in Gaussian Splatting. Existing methods often suffer from inconsistent 2D masks across viewpoints and produce noisy segmentation boundaries as they neglect these semantic cues to refine the learned Gaussians. To overcome this, we introduce Gaussian Instance Tracing (GIT), which augments the standard Gaussian representation with an instance weight matrix across input views. Leveraging the inherent consistency of Gaussians in 3D, we use this matrix to identify and correct 2D segmentation inconsistencies. Furthermore, since each Gaussian ideally corresponds to a single object, we propose a GIT-guided adaptive density control mechanism to split and prune ambiguous Gaussians during training, resulting in sharper and more coherent 2D and 3D segmentation boundaries. Experimental results show that our method extracts clean 3D assets and consistently improves 3D segmentation in both online (e.g., self-prompting) and offline (e.g., contrastive lifting) settings, enabling applications such as hierarchical segmentation, object extraction, and scene editing.
The multi-view inconsistent 2D segmentation masks introduce significant ambiguity in both feature lifting and mask prediction.
Gaussians nearby the edge often belong to both foreground and background at the same time, making precise 3D segmentation difficult.
We augments each 3D Gaussian with an instance weight matrix across views to capture its association with object instances. Leveraging the 3D consistency of Gaussians, GIT identifies and corrects 2D segmentation inconsistencies. Additionally, GIT-guided adaptive density control mechanism can split or prune ambiguous Gaussians, resuling clean and consistent instance boundaries in both 2D and 3D.
Results on Replica demonstrate that our method not only accurately extracts the target object but also significantly reduces artifacts compared to the baselines.
Our method also achieves higher-quality 2D instance segmentation results on both the Replica and NVOS datasets compared to other baselines. Our improvements in consistent instance maps and the reduction of ambiguous Gaussians yield more precise novel-view instance segmentation and feature maps, even for some tiny structures.
Novel view synthesis of 2D segmentation on Replica dataset. The results of each baseline provide the segmentation on the left and the feature map from PCA on the right. The TP, FP, and FN predictions are color-coded in the segmentation.
Novel view synthesis of 2D segmentation on NVOS dataset. The TP, FP, and FN predictions are color-coded in the segmentation. Our results with GIT produce fewer false positive predictions that are hard to observe from the input view.
The videos below show how our method enables clean 3D segmentation and flexible object extraction, supporting various downstream applications.
@inproceedings{shen2025trace3d,
title={Trace3D: Consistent Segmentation Lifting via Gaussian Instance Tracing},
author={Shen, Hongyu and Ni, Junfeng and Chen, Yixin and Li, Weishuo and Pei, Mingtao and Huang, Siyuan},
booktitle=ICCV,
year={2025}
}