Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding
ECCV 2026
1ShanghaiTech University
2ByteDance
Given a dynamic sequence with complex motions and multiple interacting instances, Director can not only achieve temporally coherent 4D reconstruction and high-fidelity rendering, but also enable instance segmentation and open-vocabulary querying.
Overview Video
We present Director, an instance-aware spatio-temporal Gaussian representation for dynamic scene modeling and understanding, which enables robust human performance tracking, high-fidelity rendering, and instance-level understanding for open-vocabulary queries.
Pipeline
Using temporally consistent SAM3 masks and sentence embeddings, our method decomposes the scene into static background and dynamic foreground, learning language- and instance-aligned features for robust tracking, high-quality rendering, and accurate instance segmentation.
Comparison
Qualitative comparison with state-of-the-art dynamic rendering methods on novel view synthesis on MPEG GSC dataset and ST-NeRF basketball dataset.
Qualitative comparison with 4D segmentation methods on ST-NeRF basketball dataset.
Visualization of open-vocabulary querying results.
Result Gallery
Acknowledgements
The authors would like to thank Chengcheng Guo, Zidong Song and Zhehao Shen from ShanghaiTech University for guidance and assistance. We also thank the reviewers for their feedback. This work was supported by the Science and Technology Commission of Shanghai Municipality under the Science and Technology Program (25511106302), the National Natural Science Foundation of China under Grant W2431046, the Central Guided Local Science and Technology Foundation of China under Grant YDZX20253100001001, the MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University), the Shanghai Frontiers Science Center of Human-centered Artificial Intelligence and HPC Platform of ShanghaiTech University.