Large vision-language models (LVLMs) have shown strong potential for embodied vision-language navigation (VLN), yet their performance degrades on long-horizon trajectories due to context-length limits, redundant historical observations, and unstable spatial alignment across time. These issues make it difficult to both compress long navigation histories and reliably reuse past spatial experience. To address these limitations, we propose MemoryVLN, a unified memory-driven navigation system built around memory construction and management. During memory construction, Long Sequence Compression employs forgetting-curve-inspired temporal sampling to preserve dense recent evidence and sparse yet representative distant observations under a limited token budget. Ego-Aware Memory Representation further integrates 2D semantic tokens, 3D geometric tokens, and 6D agent poses, allowing each memory to encode both what was observed and where it was observed. During memory management, Hierarchical Memory Bank maintains in-episode and context-level memories through cross-context warm-start and confidence-controlled write-back. Memory Enhancement then retrieves the Top-K relevant memories according to the current navigation state and fuses them with the current visual tokens. Together, these mechanisms transform raw navigation history into compact, spatially grounded, selectively retrievable, and reusable navigation experience. Extensive experiments on R2R-CE and RxR-CE demonstrate consistent improvements over the state-of-the-art baselines by a substantial margin (i.e., 2% and 6.5% in success rate).
We propose MemoryVLN, a unified memory-driven navigation system organized around memory construction and memory management, covering the complete memory process of compression, representation, storage, update, retrieval, and reuse.
Long Sequence Compression reduces long-history redundancy under a fixed token budget; Ego-Aware Memory Representation builds spatially grounded memories from 2D semantics, 3D geometry, and 6D poses; Hierarchical Memory Bank maintains in-episode and context-level experience; and Memory Enhancement retrieves and fuses relevant memories.
On R2R-CE and RxR-CE, MemoryVLN improves the success rate over state-of-the-art, data-comparable baselines by approximately 2% and 6.5%, respectively, while using only a single RGB stream and standard VLN training trajectories.
Memory construction turns raw observations into structured memory; memory management decides when that memory is stored, retrieved, and updated.
An Ebbinghaus-inspired retention prior combines inter-chunk budget allocation with intra-chunk key-frame selection. Retention grows sublinearly as B(N)=min{N, max[Bmin, ⌊a+b√N⌋]}, preserving dense recent evidence and sparse representative distant landmarks under a bounded visual-token budget.
ViT-based 2D appearance tokens, frozen-VGGT 3D geometric tokens, and 6D agent poses are fused through a geometry residual. Each memory therefore encodes both what was observed and where it was observed, disambiguating visually similar locations.
A persistent Context-Level Bank warm-starts new episodes with the most similar prior and writes back only confidence-gated experience. An In-Episode Bank keeps recent frames in a FIFO short-term pool and consolidates distant history in a cosine-weighted long-term pool.
A frame-level query retrieves the Top-K memories from the joint short- and long-term pools. A softmax-weighted summary is interpolated with the current frame and injected as a residual into the current visual tokens before action prediction.
Comparison with state-of-the-art methods on the R2R-CE and RxR-CE val-unseen splits, ablations, and paired significance tests.
R2R-CE and RxR-CE val-unseen. ★ denotes methods using a waypoint predictor; † denotes models trained only with VLN-CE expert trajectories without additional navigation data. Best results in bold.
| Method | Observation | R2R Val-Unseen | RxR Val-Unseen | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pano. | Odo. | Depth | S.RGB | NE↓ | OS↑ | SR↑ | SPL↑ | NE↓ | SR↑ | SPL↑ | nDTW↑ | |
| HPN+DN★ | ✓ | ✓ | ✓ | -- | 6.31 | 40.0 | 36.0 | 34.0 | -- | -- | -- | -- |
| CMA★ | ✓ | ✓ | ✓ | -- | 6.20 | 52.0 | 41.0 | 36.0 | 8.76 | 26.5 | 22.1 | 47.0 |
| Sim2Sim★ | ✓ | ✓ | ✓ | -- | 6.07 | 52.0 | 43.0 | 36.0 | -- | -- | -- | -- |
| VLN-BERT★ | ✓ | ✓ | ✓ | -- | 5.74 | 53.0 | 44.0 | 39.0 | 8.98 | 27.0 | 22.6 | 46.7 |
| Ego2-Map★ | ✓ | ✓ | ✓ | -- | 5.54 | 56.0 | 47.0 | 41.0 | -- | -- | -- | -- |
| DreamWalker★ | ✓ | ✓ | ✓ | -- | 5.53 | 59.0 | 49.0 | 44.0 | -- | -- | -- | -- |
| GridMM★ | ✓ | ✓ | ✓ | -- | 5.11 | 61.0 | 49.0 | 41.0 | -- | -- | -- | -- |
| Reborn★ | ✓ | ✓ | ✓ | -- | 5.40 | 57.0 | 50.0 | 46.0 | 5.98 | 48.6 | 42.0 | 63.3 |
| InstructNav | ✓ | -- | -- | -- | 6.89 | -- | 31.0 | 24.0 | -- | -- | -- | -- |
| COSMO | ✓ | -- | -- | -- | -- | 56.0 | 47.0 | 40.0 | -- | -- | -- | -- |
| AO-Planner | ✓ | -- | -- | ✓ | 5.55 | 59.0 | 47.0 | 33.0 | 7.06 | 43.3 | 30.5 | 50.1 |
| LAW | -- | ✓ | ✓ | ✓ | 6.83 | 44.0 | 35.0 | 31.0 | 10.90 | 8.0 | 8.0 | 38.0 |
| MapNav | -- | ✓ | ✓ | ✓ | 4.93 | 53.0 | 39.7 | 37.2 | -- | -- | -- | -- |
| g3D-LF | -- | ✓ | ✓ | ✓ | 5.70 | 59.5 | 47.2 | 34.6 | -- | -- | -- | -- |
| Seq2Seq | -- | -- | ✓ | ✓ | 7.77 | 37.0 | 25.0 | 22.0 | 12.10 | 13.9 | 11.9 | 30.8 |
| NaVid-4D | -- | -- | ✓ | ✓ | 5.99 | 55.7 | 43.8 | 37.1 | -- | -- | -- | -- |
| NavMorph | -- | -- | ✓ | ✓ | 5.75 | 56.9 | 47.9 | 33.2 | 8.85 | 30.8 | 22.8 | 44.2 |
| NaVid | -- | -- | -- | ✓ | 5.72 | 49.2 | 41.9 | 36.5 | 5.72 | 45.7 | 38.2 | -- |
| Sim2Real | -- | -- | -- | ✓ | 5.95 | 55.8 | 44.9 | 30.4 | 8.79 | 36.7 | 25.5 | 18.1 |
| Uni-NaVid | -- | -- | -- | ✓ | 5.58 | 53.3 | 47.0 | 42.7 | 6.24 | 48.7 | 40.9 | -- |
| StreamVLN-RGB-only | -- | -- | -- | ✓ | 5.10 | 64.0 | 55.7 | 50.9 | 6.16 | 51.8 | 45.0 | 62.1 |
| NaVILA | -- | -- | -- | ✓ | 5.22 | 62.5 | 54.0 | 49.0 | 6.77 | 49.3 | 44.0 | 58.8 |
| InternVLA-N1 | -- | -- | -- | ✓ | 4.89 | 60.6 | 55.4 | 52.1 | 6.41 | 49.5 | 41.8 | 62.6 |
| JanusVLN† | -- | -- | -- | ✓ | 5.17 | 58.0 | 52.8 | 49.2 | 6.46 | 51.4 | 44.3 | 59.1 |
| MemoryVLN (Ours) | -- | -- | -- | ✓ | 3.52 | 59.98 | 54.76 | 50.26 | 3.33 | 57.84 | 50.42 | 63.52 |
MemoryVLN uses a single RGB stream and standard VLN training trajectories, without DAgger rollouts or additional navigation data. Compared with the strongest data-comparable baseline, SR improves by about 2% on R2R-CE and 6.5% on RxR-CE.
R2R-CE and RxR-CE val-unseen. Removing Long Sequence Compression leads to out-of-memory (OOM) because historical visual tokens grow with trajectory length.
| Dataset | Variant | NE↓ | SR↑ | SPL↑ |
|---|---|---|---|---|
| R2R | w/o Long-Seq. Comp. | OOM | ||
| R2R | w/o Pos-Encoding | 3.66 | 52.75 | 49.25 |
| R2R | w/o In-Episode | 3.57 | 53.34 | 49.06 |
| R2R | w/o Context | 3.55 | 54.16 | 49.65 |
| R2R | Full MemoryVLN | 3.52 | 54.76 | 50.26 |
| RxR | w/o Long-Seq. Comp. | OOM | ||
| RxR | w/o Pos-Encoding | 3.35 | 57.24 | 48.74 |
| RxR | w/o In-Episode | 3.30 | 56.69 | 48.13 |
| RxR | w/o Context | 3.29 | 57.37 | 49.09 |
| RxR | Full MemoryVLN | 3.33 | 57.84 | 50.42 |
The four modules provide complementary benefits in scalability, spatial alignment, in-episode history preservation, and cross-episode reuse.
Episode-level paired tests between JanusVLN and MemoryVLN on the same evaluation episodes. Differences are MemoryVLN minus JanusVLN in percentage points.
| Dataset | Metric | N | JanusVLN | MemoryVLN | Δ (pp) | p |
|---|---|---|---|---|---|---|
| R2R | SR | 1,839 | 52.8 | 54.8 | +2.0 | 0.158 |
| R2R | SPL | 1,839 | 49.0 | 50.3 | +1.2 | 0.327 |
| RxR | SR | 3,669 | 51.0 | 57.2 | +6.2 | 2.7×10-12 |
| RxR | SPL | 3,570 | 43.5 | 48.7 | +5.2 | 1.3×10-12 |
On R2R the differences are positive but not statistically significant; on RxR both SR and SPL differences are significant. Confidence intervals are paired bootstrap 95% intervals.
Under identical instructions, JanusVLN loses instruction-relevant context and fails, whereas MemoryVLN preserves historical and spatial context and reaches the target.
“Walk down the stairs and take a right to go down the next flight of stairs. Stop at the bottom of the stairs.”
JanusVLN · Fail · 110 steps · NE 4.28 m
MemoryVLN · Success · 45 steps · NE 0.03 m
“Walk all the way forward and enter another door. Keep walking forward on the red carpet, and stop where the tiles are on the ground, and the end of the carpet.”
JanusVLN · Fail · 109 steps · NE 32.91 m
MemoryVLN · Success · 41 steps · NE 1.13 m
“Begin facing 2 grey chairs. Turn around and head into the open closet door way to the right of the bed. Head down the length of the closet towards the washroom and you’re done.”
JanusVLN · Fail · 55 steps · NE 8.76 m
MemoryVLN · Success · 36 steps · NE 0.08 m
“You are now facing to the wall, from there take an about turn, start walking straight through the narrow corridor. Search for the brown color in front of you and stand near it.”
JanusVLN · Fail · 40 steps · NE 3.03 m
MemoryVLN · Success · 53 steps · NE 0.08 m
“Right now you are facing toward a stone wall. Make a slight left and get down the stair case in front you, you can see a carpet, walk through the carpet until where you can see a sofa in front of you.”
JanusVLN · Fail · 52 steps · NE 15.95 m
MemoryVLN · Success · 31 steps · NE 0.10 m
Beyond simulation, MemoryVLN executes natural-language instructions on a physical robot in an indoor office environment.
Four natural-language instructions executed end-to-end on a quadruped robot, with navigation-policy inference on a single NVIDIA GeForce RTX 4090 GPU. Landmarks and routes are shown at 4× speed.
“Go out of the gym and move right, enter the conference hall, and stop in front of the podium.”
Task 1 · Conference-hall podium
“Go out and turn right, walk pass the printer and turn right, and stop in front of the coffee machine.”
Task 2 · Coffee machine
“Go out and turn right, walk pass the table and turn right, enter the second room and stop in it.”
Task 3 · Designated room
“Move forward and turn right, walk pass the elevator and stop in front of the water dispenser.”
Task 4 · Water dispenser
Four representative MemoryVLN failures, exposing limitations at the semantic-topological, terminal-grounding, memory-quality, and long-horizon progress levels.
All four trajectories have zero Success, SPL, and Oracle Success. NE is the final navigation error; Oracle NE is the minimum goal distance reached anywhere along the predicted trajectory.
| Dataset | Scene / Episode | Steps | NE | Oracle NE | Coll. | Failure type |
|---|---|---|---|---|---|---|
| R2R | QUCTc6BB5sX / 577 | 78 | 28.61 | 14.13 | 0 | Semantic branch confusion |
| R2R | x8F5xyUWy9e / 231 | 60 | 10.76 | 6.08 | 3 | Final-doorway grounding |
| R2R | EU6Fwq7SyZv / 1206 | 56 | 10.28 | 5.82 | 0 | Failed-context reuse |
| RxR | 2azQ1b91cZZ / 5691 | 44 | 16.40 | 12.49 | 3 | Progress loss / early stop |
@article{memoryvln2027,
title={MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation},
author={Xueyao Wei and Yifei Wang and Delun Li and Shaoan Wang and Junzhi Yu and Wenjie Song and Xingyu Chen},
journal={arXiv preprint},
year={2027}
}