PlenoCI: Plenoptic CharacterIstics for View Dependence Aware Classification

Given two 3DGS reconstructions from different times, our method first detects scene changes. To classify changes, we draw on view-dependent plenoptic structures encoded in our novel Plenoptic CharacterIstics (PlenoCI) feature. For example, we correctly detect the sheet of paper and book as changes, and classify them using PlenoCI as occlusion edge evidence (left). Our approach remains robust to underconstrained representations, reporting few false positives in unchanged scenes (right). Mouse over the image to zoom.

Radiance field representations such as NeRFs and 3DGS are inherently underconstrained. Two independent optimizations given the same training images yield different primitive configurations.

We derive closed-form analytic plenoptic derivatives directly from a 3DGS representation. These derivatives capture how a ray's color changes with respect to its origin and direction, and can be rendered efficiently using standard 3DGS rasterization.

We sample analytic plenoptic derivatives over a local light field and compute the plenoptic structure tensor. The third eigenvalue λ₂ forms the basis for PlenoCI, a 5D feature that summarises informative plenoptic structures such as occlusion edges, specular highlights and reflections while ignoring diffuse textures.

Given two 3DGS scene reconstructions from different times, our pipeline registers the captures with SfM, builds instance-aware 3DGS representations, and aggregates per-ray change scores at the instance level. This yields explicit change representations from which dense change masks can be rendered. PlenoCI acts as a proxy for geometry: occlusion edges naturally shift when scene structure changes. PlenoCI enables us to classify changes as geometric or appearance-based.

Our overall scene change detection and classification approach. Given instance-aware 3DGS scene representations G₀ and G₁, we aggregate per-ray change scores δ in the corresponding primitives. This yields a sparse set of anchors ΔG'_t to propagate on changed instances. For change detection, we sample edge rays R_e to find changed Gaussians ΔG_t^all. For change classification, we leverage PlenoCI R to find geometrically changed Gaussians ΔG_t^geo. These explicit representations can be used to render novel change masks with disambiguation between geometric and appearance changes.

Our approach surpasses all baselines on CL-Splats, making a 25.7% mIoU improvement over the strongest competitor, and remains competitive on PASLCD. Our instance-aware pipeline produces high-quality segments for instances with sufficient change evidence, while methods lacking instance information provide incomplete segmentations.

The first, second, and third best performances are highlighted.

Ablating RGB, semantic features, or instance occupancy from our change scoring approach reduces performance, confirming that all three components contribute positively to our results.

Since instance segmentation is upstream and orthogonal to PlenoCI, we directly benefit from improvements to the segmenter. Using Oracle instances derived from ground truth masks allows small objects to be segmented, improving change recall.

Using PlenoCI as a proxy for scene geometry enables classification of geometric and appearance changes. On PASLCD, we achieve 74% balanced accuracy, comparable with GS-Diff. Future work could explore disentangling occlusion edges from specularity and reflections captured in PlenoCI.

Comparing two 3DGS representations independently optimized from the same training views should yield zero change. Our method (row 2) is robust to reconstruction ambiguity, reporting a two orders of magnitude lower false positive rate (FPR) than primitive-space methods (row 1).