MultINav ZGCA HMI Lab

Embodied AI · Interactive Navigation

Interactive Navigation Beyond Reachability: Benchmark and Agentic Framework

Delun Li1,2,* Xueyao Wei1,2,* Rui Chen3 Yifei Wang3 Junzhi Yu3 Wenjie Song1 Xingyu Chen2,✉

1Beijing Institute of Technology 2Zhongguancun Academy 3Peking University

*Equal contribution ✉Corresponding author

Paper BibTeX Code Dataset
Overview of MultINav: conventional interactive navigation, the MultINav benchmark, the MultINav agent, and joint navigation and decision making.
Figure 1 · Overview of MultINav. Channel interactions change spatial connectivity while container interactions change target visibility; MultINav-Bench covers both effects and their sequential dependencies.

Abstract

Existing embodied navigation tasks commonly assume that traversability and object states remain fixed during an episode. Yet reaching a goal may require opening a door to access a room and then opening a container to reveal a hidden target. We formulate interactive navigation beyond reachability, unifying channel-induced changes in spatial connectivity, container-induced changes in target visibility, and their sequential dependencies.

We introduce MultINav-Bench, a 3K-episode ObjectGoal benchmark in MolmoSpaces spanning channel, container, and mixed tasks, with interaction-necessity annotations covering required and unnecessary episodes. Its data generation pipeline automates task construction across multiple task types. The evaluation protocol separately measures navigation success, interaction effectiveness, and cost.

We further propose MultINav-Agent, a three-stage agentic framework built on a persistent, state-conditioned interaction graph. Semantic interactive perception grounds entities and interaction states; a vision-language model selects task-relevant actions based on expected changes in connectivity and target observability; and post-interaction verification updates the graph to guide subsequent navigation, exploration, or interaction. Experiments on MultINav-Bench show that MultINav-Agent consistently outperforms adapted baselines across channel, container, and mixed tasks.

01

Interactive navigation beyond reachability

A formulation that unifies channel-induced connectivity changes, container-induced visibility changes, and the sequential order in which they must be resolved.

02

MultINav-Bench

3,000 ObjectGoal episodes across 639 houses — 1,000 channel, 1,000 container, and 1,000 mixed — with interaction-necessity labels and a five-metric protocol.

03

MultINav-Agent

A persistent, state-conditioned interaction graph that connects semantic perception, effect-guided action selection, and post-interaction verification.

MultINav-Bench

A diagnostic ObjectGoal benchmark for navigation under interaction-induced state changes. An episode may require opening a passage door, exposing a target concealed inside a container, or composing both operations in sequence before navigation can succeed.

Table 1 · Comparison of interaction scope and evaluated state effects

Chan., Cont., and Obj. denote channels, containers, and manipulable objects. Reported scale follows the respective publications.

Benchmark Endpoint Chan. Cont. Evaluated state effect Reported scale / envs.
Non-interactive benchmarks (fixed environment state)
Habitat ObjectNavObjectGoalStatic2.8M eps. / 90 envs.
MolmoSpaces Navigate-toObjectGoalStatic2K traj. / 679 envs.
R2RInstructionGoalStatic21.6K instr. / 90 envs.
VLN-CEInstructionGoalStaticR2R annot. / 90 envs.
Interactive benchmarks (environment-changing actions)
Interactive GibsonPointGoal✓Free space + disturbance106 envs.
Interactive Visual Nav.PointGoalFree space15K eps. / 120 envs.
CaMPPointGoalFree space600K eps. / 12K envs.
InterNavPointGoal✓Free space + connectivity150 eval / 15 envs.
ALFREDHousehold✓Object state8K demos / 120 envs.
BEHAVIOR-1KActivity✓Goal predicates1K activities / 50 envs.
MultINav-Bench (ours)ObjectGoal ✓✓ Connectivity + target exposure3K eps. / 639 envs.
3,000frozen episodes
639houses in MolmoSpaces
1,000episodes per split
2,654 / 346required / unnecessary

Channel interactions

Hinged doors govern spatial reachability, including critical closed doors (required), distractor doors (unnecessary), and composite layouts.

Container interactions

Refrigerators, cabinets, and drawers expose occluded targets, enabling the visibility that terminal ObjectGoal success requires.

Mixed episodes

Sequential composition is mandatory — for example, traversing a closed door to reach a closed refrigerator that hides the goal.

Construction and composition of MultINav-Bench: channel and container interaction effects, the four-stage data generation pipeline, required interaction counts, and oracle-conditioned path lengths.
Figure 2 · Construction and composition of MultINav-Bench. Top: channel interactions alter reachability and, in a route-shortening subset, route efficiency, whereas container interactions change target visibility and accessibility. Bottom: benchmark construction and data collection. Right: required-interaction counts and feasible-route lengths for the 3,000 episodes.

Task and state

The embodied state is st = (xt, Mt, zt), combining the robot state, a geometric and semantic map, and the articulation states of interactive entities. Actions combine navigation, which updates the robot state, with physical interaction, which modifies traversability or target visibility.

Terminal ObjectGoal success requires stopping within the distance threshold and observing the target in the head camera. Opening a door without locating the target yields no success, which keeps interaction effects separate from the final navigation outcome.

Generation pipeline

Starting from ProcTHOR-10K houses and NavToObj seeds, candidates are synthesized through counterfactual reachability and visibility checks: closing a channel must sever the shortest route to the goal, while concealing a target inside a container must invalidate initial visibility until it is opened.

Candidates pass articulation readback, target-identity verification, and oracle rollout validation before freezing 1,000 balanced episodes per domain. A full collection mode additionally records multimodal expert demonstrations with synchronized visual streams, base trajectories, and end-effector targets.

Evaluation metrics (paper Section 3.3)

MetricDirectionWhat it measures
SRhigher Terminal ObjectGoal success under standard distance and visibility thresholds.
SPLhigher Success-weighted planar path efficiency relative to a feasible oracle route, reported on [0, 1].
ISRhigher Mean fraction of verified necessary interaction effects on required episodes; valid plans receive partial credit.
IPhigher Category-level interaction precision: the episode-mean fraction of attempts producing a verified physical effect in the target interaction class.
Total Costlower All-episode mean of normalized operation cost, counting navigation, interactions, and attempts without a qualifying effect.

MultINav-Agent

A persistent, state-conditioned interaction graph connects semantic perception, action selection, and post-interaction verification. Verified changes update the graph that drives each subsequent navigation, exploration, or interaction decision.

MultINav-Agent framework: semantic interactive perception, action selection with candidate validation and semantic decision, and interaction verification with graph updates.
Figure 3 · MultINav-Agent framework. Perception builds a persistent, state-conditioned interaction graph; candidate plans are formed; interaction proposals are filtered for reachability, observability, and freshness; a VLM scores admissible plans by task relevance; and post-interaction verification updates the graph to guide subsequent action selection.
Stage 1

Semantic interactive perception

The graph Gt = (Vt, Et) represents rooms, interactive entities, objects, and the robot through connectivity, containment, and visibility relations. Entities are associated across observations while visual attribute inference estimates interaction class and coarse state: open, closed, or uncertain.

Stage 2

Effect-guided action selection

Navigation, frontier exploration, and interaction candidates are generated from the graph and map. Eligibility checks on route feasibility, visual evidence, and state reliability constrain the executable set, and a vision-language model ranks candidates by their expected contribution to reaching or observing the target.

Stage 3

Verification and graph updates

Post-interaction verification reconciles fresh observations with execution feedback. Confident evidence updates the affected entity state, while conflicting or occluded evidence is marked uncertain, and failed attempts stay in the decision history to discourage immediate repetition.

Sequential dependencies. Opening a door enables entry into a room; observations there can introduce a container candidate; a verified container opening exposes objects whose observations support the next navigation decision. Each step updates the graph and the available actions.

Results

MultINav-Agent achieves better aggregate values on all five metrics than the MolmoSpaces adaptation of MoMa-LLM, and remains competitive on non-interactive ObjectGoal navigation.

Table 2 · Non-interactive ObjectGoal (MolmoSpaces NavToObj)

Terminal success rate over 2,000 tasks; † indicates leaderboard results.

MethodTasksSR (%) ↑Domain
MultINav-Agent (ours)2,00053.0ProcTHOR
RING†2,00048.0ProcTHOR
DualVLN†2,00021.0Habitat
SPOC2,00020.15ProcTHOR
VLFM2,00011.90Habitat

Table 3 · Interactive navigation on MultINav-Bench

Each method is evaluated on 3,000 episodes, 1,000 per split. Each method's first row reports the overall result, followed by its task splits. Bold marks the best value for each metric within the same split.

Method / Split Navigation & Task Interaction Overall
SR (%) ↑ SPL ↑ ISR (%) ↑ IP (%) ↑ Cost ↓
MoMa-LLM 10.530.084823.6927.590.9309
Channel 19.600.175854.2840.610.8651
Container 10.300.06852.002.370.9324
Mixed 1.700.010125.3739.810.9951
MultINav-Agent (ours) 41.630.290349.3846.790.7743
Channel 55.800.456760.0954.290.6517
Container 45.000.300846.7038.910.7220
Mixed 24.100.113545.0747.160.9493

MultINav-Agent completes 1,249 of 3,000 episodes and improves SR by 31.10 points, SPL from 0.0848 to 0.2903, and normalized cost from 0.9309 to 0.7743. Channel, container, and mixed SR gains are 36.20, 34.70, and 22.40 points respectively; mixed tasks remain the hardest for both methods.

Table 4 · Ablation on a fixed 200-episode mixed subset

All four variants use the same frozen subset of 165 houses. Bold marks the best value in each column.

Method Navigation & Task Interaction Overall
SR (%) ↑ SPL ↑ ISR (%) ↑ IP (%) ↑ Cost ↓
Full 23.00.10741.840.40.961
Flat object memory 21.00.10043.826.70.963
Greedy selection 22.00.13440.936.90.933
Perception-only update 5.50.03326.636.60.988

Retaining execution-informed graph updates matters most: replacing verification with a perception-only update drops SR from 23.0% to 5.5% on the ablation subset, while graph memory shows its clearest advantage in interaction precision.

Qualitative Results

ObjectGoal rollouts from the supplementary material. Each clip follows the agent as it explores, interacts, and verifies the resulting state change before continuing toward the goal.

Simulated trials in MolmoSpaces

Channel and container effects, resolved as the agent searches for the target object.

Case study: the robot opens two doors to reach a refrigerator and opens it to reveal the target, with evolving semantic maps and interaction graphs.
Figure 4 · Sequential interactions and graph updates. The robot opens two doors to reach a refrigerator, then opens it to reveal the target. Insets show evolving semantic maps and interaction graphs; the final panel shows the resulting graph and trajectory, where blue marks navigation and exploration and orange marks interaction.
Object goal: apple Door and container interactions in sequence.
Object goal: mug Container interaction exposes the target.
Object goal: book Exploration followed by a verified opening.
Object goal: lettuce Visibility change required for terminal success.
Object goal: egg Scene 1: Multi-room house combining reachability and visibility changes.
Object goal: egg Scene 2: Different house environment testing generalization across room topologies.

Real-world trials

A Go2 quadruped in an indoor environment, with human-assisted door and refrigerator opening. Egocentric observations, intermediate interaction graphs, and the semantic map record the evolving scene representation.

Real-world navigation with a Go2 quadruped: starting, opening a door, exploring, navigating, and opening a refrigerator.
Figure 5 · Real-world navigation. The robot explores and navigates to a refrigerator, with human-assisted door and refrigerator opening. Insets show observations and interaction graphs; map markers indicate navigation (blue) and interaction (orange).
Object goal: black bag Door traversal, exploration, and container inspection.
Object goal: watermelon Sequential interactions in an office and kitchen area.

BibTeX

@misc{multinav2026,
  title  = {Interactive Navigation Beyond Reachability: Benchmark and Agentic Framework},
  author = {Li, Delun and Wei, Xueyao and Chen, Rui and Wang, Yifei and
            Yu, Junzhi and Song, Wenjie and Chen, Xingyu},
  year   = {2026},
  note   = {Preprint}
}

The arXiv identifier will be added once it is assigned; code and dataset links are available above.