Open-Vocabulary 3D Scene Graphs: Grounding Multimodal LLMs in Volumetric Metric Space

Open-Vocabulary 3D Scene Graphs: Grounding Multimodal LLMs in Volumetric Metric Space

By TechIDaily Robotics Perception & Spatial Knowledge Systems · Published 2026-10-10

Large Multimodal Models (LMMs, such as GPT-4o, Gemini 1.5, and Claude 3.5) demonstrate unprecedented semantic reasoning, common sense, and high-level planning. When prompted to "Prepare the dining table for tea", an LMM effortlessly decomposes the goal into sequential subtasks: locate the kettle, find clean mugs, place them on saucers, and arrange spoons.

However, when deployed on physical robots operating in real-world homes, a critical disconnect emerges: the spatial metric grounding barrier. LMMs operate in the realm of 2D images and natural language tokens. They have no innate comprehension of 3D metric geometry:

• They cannot tell if an object is 1.2 meters away or 4.5 meters away.
• They cannot determine if a mug will collide with a shelf during an arm trajectory.
• They cannot evaluate structural relationships, such as whether a bowl is inside a microwave or resting on a counter.

To solve this, modern spatial intelligence pipelines construct Hierarchical Open-Vocabulary 3D Scene Graphs (3D-OSGs). By lifting foundation model features (CLIP / SigLIP) directly into volumetric metric entities, robots ground high-level semantic instructions into actionable 6-DoF coordinates.

1. Hierarchical Spatial Scene Graph Architecture

┌────────────────────────────────────────────────────────────────────────┐
│  HIERARCHICAL OPEN-VOCABULARY 3D SCENE GRAPH GENERATOR                 │
├────────────────────────────────────────────────────────────────────────┤
│  Continuous Spatial Mapping Stream (RGB-D + LiDAR Odometry):           │
│  Incremental Point Cloud Voxelization & Surfel Clustering              │
│                   │                                                    │
│                   ▼                                                    │
│  2D-to-3D Open-Vocabulary Feature Projection:                          │
│  - Segment Anything Model (SAM) Mask Decomposition                     │
│  - CLIP / SigLIP Visual Feature Extraction per Mask                    │
│  - Multi-View Ray Tracing & Point Feature Back-Projection (512-dim)    │
│                   │                                                    │
│                   ▼                                                    │
│  Hierarchical Spatial Graph Synthesizer:                               │
│  ┌──────────────────────────────────────────────────────────────────┐  │
│  │ Layer 1: Topological Rooms (Kitchen, Living Room, Office)        │  │
│  │ Layer 2: Major Structural Layouts (Counter, Wall, Table)        │  │
│  │ Layer 3: Discrete Objects (Mug, Kettle, Spoon, Cutting Board)    │  │
│  │ Layer 4: Grasp Affordance Vertices & Collision Bounding Boxes    │  │
│  └──────────────────────────────────────────────────────────────────┘  │
│                   │                                                    │
│                   ▼                                                    │
│  Natural Language Grounding & LLM Query Interface:                    │
│  Vector Cosine Cos(e_prompt, e_node) ──► Metric 3D Bounding Box Center │
└────────────────────────────────────────────────────────────────────────┘

2. Mathematical Formalism: Feature Back-Projection & Graph Retrieval

Each 3D spatial entity node $u \in \mathcal{V}$ possesses a 3D centroid $p_u \in \mathbb{R}^3$, an oriented bounding box (OBB) $B_u \in \text{SE}(3) \times \mathbb{R}^3$, and an aggregated semantic embedding $F_u \in \mathbb{R}^D$:

F_u = \frac{1}{\sum_{v=1}^V w_{u, v}} \sum_{v=1}^V w_{u, v} \cdot f_{\text{CLIP}}(I_v, \pi_v(p_u))

Where $I_v$ is camera frame $v$, $\pi_v(\cdot)$ is the camera projection matrix, and $w_{u, v}$ represents a view-angle and distance confidence weight.

When an LLM issues a query string (e.g., "the porcelain container with floral patterns"), the query score over all candidate nodes in the active room is computed via normalized cosine similarity:

\text{Score}(u, \text{query}) = \frac{\langle F_u, f_{\text{text}}(\text{query}) \rangle}{\|F_u\|_2 \|f_{\text{text}}(\text{query})\|_2} \cdot \mathbf{1}_{\text{spatial}}(u, \text{constraints})

Below is the Python query interface linking natural language embeddings with 3D spatial scene graph entities:

import numpy as np
from dataclasses import dataclass
from typing import List, Optional

@dataclass
class SceneGraphNode:
    node_id: str
    layer: str                class="tok-comment"># class="tok-string">'room', class="tok-string">'structure', class="tok-string">'object', class="tok-string">'affordance'
    centroid_3d: np.ndarray   class="tok-comment"># [x, y, z] metric coordinates in meters
    bounding_box: np.ndarray  class="tok-comment"># [dx, dy, dz, roll, pitch, yaw]
    embedding: np.ndarray     class="tok-comment"># 512-dim L2-normalized CLIP feature
    parent_id: Optional[str] = None

class OpenVocabularySceneGraphEngine:
    class="tok-string">"""
    In-memory spatial metric scene graph query engine for embodied LLM grounding.
    class="tok-string">"""
    def __init__(self):
        self.nodes = {}

    def register_node(self, node: SceneGraphNode):
        self.nodes[node.node_id] = node

    def query_object_by_text(self, text_feature: np.ndarray, target_room_id: Optional[str] = None) -> List[tuple]:
        results = []
        text_feature = text_feature / np.linalg.norm(text_feature)

        for nid, node in self.nodes.items():
            if node.layer != &class="tok-comment">#39;object':
                continue
            if target_room_id and node.parent_id != target_room_id:
                continue

            class="tok-comment"># Cosine similarity matching
            similarity = float(np.dot(node.embedding, text_feature))
            results.append((nid, node.centroid_3d, similarity))

        class="tok-comment"># Sort descending by semantic confidence
        results.sort(key=lambda item: item[2], reverse=True)
        return results

3. Hierarchical Spatial Relations Topology

graph TD
    Kitchen[Room Node: Kitchen] --> Counter[Structure Node: Granite Countertop]
    Kitchen --> Sink[Structure Node: Stainless Sink]
    Counter -->|Supports| Mug[Object Node: Blue Coffee Mug]
    Counter -->|Supports| Kettle[Object Node: Electric Kettle]
    Mug -->|Affordance| GraspHandle[Grasp Node: Handle Antipodal Grasp]
    Mug -->|Affordance| Rim[Placement Node: Top Rim]

4. Benchmark on ScanNet200 & HM3D Real-World Datasets

Tested across 150 diverse household environments:

5. Key Engineering Insights

1. Hierarchical Pruning Accelerates Inference: Searching for a fork within an entire house is computationally expensive. Searching inside Kitchen -> Cutlery Drawer -> Fork reduces vector similarity candidate evaluations by 98%.
2. Affordance Nodes Enable Zero-Shot Grasping: Labeling not just the object centroid, but antipodal handle grasps directly in 3D metric space bridges the gap to robot arm inverse kinematics.
3. Robust to Dynamic Changes: When an object moves, only the leaf node's metric transform updates, preserving the high-level room topology intact.