Open-Vocabulary 3D Scene Graphs: Grounding Multimodal LLMs in Volumetric Metric Space

Open-Vocabulary 3D Scene Graphs: Grounding Multimodal LLMs in Volumetric Metric Space

By TechIDaily Robotics Perception & Spatial Knowledge Systems · Published 2026-10-10

Large Multimodal Models (LMMs, such as GPT-4o, Gemini 1.5, and Claude 3.5) demonstrate unprecedented semantic reasoning, common sense, and high-level planning. When prompted to "Prepare the dining table for tea", an LMM effortlessly decomposes the goal into sequential subtasks: locate the kettle, find clean mugs, place them on saucers, and arrange spoons.

However, when deployed on physical robots operating in real-world homes, a critical disconnect emerges: the spatial metric grounding barrier. LMMs operate in the realm of 2D images and natural language tokens. They have no innate comprehension of 3D metric geometry:

• They cannot tell if an object is 1.2 meters away or 4.5 meters away.
• They cannot determine if a mug will collide with a shelf during an arm trajectory.
• They cannot evaluate structural relationships, such as whether a bowl is inside a microwave or resting on a counter.

To solve this, modern spatial intelligence pipelines construct Hierarchical Open-Vocabulary 3D Scene Graphs (3D-OSGs). By lifting foundation model features (CLIP / SigLIP) directly into volumetric metric entities, robots ground high-level semantic instructions into actionable 6-DoF coordinates.

1. Hierarchical Spatial Scene Graph Architecture

  │
│                   │                                                    │
│                   ▼                                                    │
│  Natural Language Grounding & LLM Query Interface:                    │
│  Vector Cosine Cos(e_prompt, e_node) ──► Metric 3D Bounding Box Center │
└────────────────────────────────────────────────────────────────────────┘

2. Mathematical Formalism: Feature Back-Projection & Graph Retrieval

Each 3D spatial entity node $u \in \mathcal{V}$ possesses a 3D centroid $p_u \in \mathbb{R}^3$, an oriented bounding box (OBB) $B_u \in \text{SE}(3) \times \mathbb{R}^3$, and an aggregated semantic embedding $F_u \in \mathbb{R}^D$:

F_u = \frac{1}{\sum_{v=1}^V w_{u, v}} \sum_{v=1}^V w_{u, v} \cdot f_{\text{CLIP}}(I_v, \pi_v(p_u))

Where $I_v$ is camera frame $v$, $\pi_v(\cdot)$ is the camera projection matrix, and $w_{u, v}$ represents a view-angle and distance confidence weight.

When an LLM issues a query string (e.g., "the porcelain container with floral patterns"), the query score over all candidate nodes in the active room is computed via normalized cosine similarity:

\text{Score}(u, \text{query}) = \frac{\langle F_u, f_{\text{text}}(\text{query}) \rangle}{\|F_u\|_2 \|f_{\text{text}}(\text{query})\|_2} \cdot \mathbf{1}_{\text{spatial}}(u, \text{constraints})

Below is the Python query interface linking natural language embeddings with 3D spatial scene graph entities:

import numpy as np
from dataclasses import dataclass
from typing import List, Optional

@dataclass
class SceneGraphNode:
    node_id: str
    layer: str                class="tok-comment"># class="tok-string">'room', class="tok-string">'structure', class="tok-string">'object', class="tok-string">'affordance'
    centroid_3d: np.ndarray   class="tok-comment"># [x, y, z] metric coordinates in meters
    bounding_box: np.ndarray  class="tok-comment"># [dx, dy, dz, roll, pitch, yaw]
    embedding: np.ndarray     class="tok-comment"># 512-dim L2-normalized CLIP feature
    parent_id: Optional[str] = None

class OpenVocabularySceneGraphEngine:
    class="tok-string">"""
    In-memory spatial metric scene graph query engine for embodied LLM grounding.
    class="tok-string">"""
    def __init__(self):
        self.nodes = {}

    def register_node(self, node: SceneGraphNode):
        self.nodes[node.node_id] = node
    class="tok-comment"># ... [Implementation detail: 14 lines of low-level hardware/state routines condensed] ...

        class="tok-comment"># Sort descending by semantic confidence
        results.sort(key=lambda item: item[2], reverse=True)
        return results

3. Hierarchical Spatial Relations Topology

4. Benchmark on ScanNet200 & HM3D Real-World Datasets

Tested across 150 diverse household environments:

5. Key Engineering Insights

1. Hierarchical Pruning Accelerates Inference: Searching for a fork within an entire house is computationally expensive. Searching inside Kitchen -> Cutlery Drawer -> Fork reduces vector similarity candidate evaluations by 98%.
2. Affordance Nodes Enable Zero-Shot Grasping: Labeling not just the object centroid, but antipodal handle grasps directly in 3D metric space bridges the gap to robot arm inverse kinematics.
3. Robust to Dynamic Changes: When an object moves, only the leaf node's metric transform updates, preserving the high-level room topology intact.