World-Ego Modeling for Embodied Video Generation in Long-Horizon Navigation-Manipulation Tasks

Zuyao Lin1,2,3 Jianhui Zhang3,4 Peidong Jia5 Xiaoguang Zhao1 Shanghang Zhang5 Jingdong Wang3 Xingyu Chen3,✉
1Institute of Automation, Chinese Academy of Sciences 2University of Chinese Academy of Sciences 3Zhongguancun Academy 4Shanghai Jiaotong University 5Peking University
✉Corresponding author
World-ego decomposition for consistent scenes and instruction-aligned interactions across navigation and manipulation

Separate the world and the ego for consistent scenes and accurate instruction following in long-horizon navigation-manipulation tasks.

Abstract

Embodied video world models typically capture both scene evolution and the robot's behavior, which we refer to as the world and the ego, respectively. The world and the ego exhibit different underlying dynamics: world prediction relies primarily on visual history and emphasizes scene stability, whereas ego prediction relies more strongly on the current instruction and emphasizes accurate instruction following. Modeling both components within a single generation stream can entangle these different dependencies, making it difficult to specialize the prediction of either component. Consequently, it becomes difficult to simultaneously maintain scene consistency and accurate instruction following, particularly in long-horizon navigation-manipulation tasks. In this paper, we propose to decompose an embodied video into the world and the ego and disentangle their generation processes. Specifically, we define the world as the background and currently unmanipulated objects, and the ego as the robot and currently manipulated objects. Based on this definition, we develop the World-Ego Model (WEM), which combines a vision-language state predictor using role-conditioned attention (RCA) and asymmetric query budgets with a semantic-routed mixture-of-experts (SR-MoE) diffusion generator. To enable rigorous evaluation, we further construct HTEWorld, a dataset and benchmark for long-horizon embodied video generation with hybrid navigation-manipulation tasks, providing 125K training video clips comprising over 4.5M frames with fine-grained instructions, together with 300 multi-turn evaluation trajectories covering over 2K instructions. Extensive experiments show that WEM achieves state-of-the-art performance on HTEWorld while remaining competitive on existing manipulation-oriented evaluations.

Highlights

01

World-Ego Decomposition

We show that an embodied video can be decomposed into world (i.e., background and currently unmanipulated objects) and ego (i.e., the robot and currently manipulated objects) components, providing an effective formulation for embodied video world models.

02

World-Ego Model

We develop WEM to instantiate world-ego decomposition with a world-ego state predictor and an SR-MoE video generator. We demonstrate its effectiveness for long-horizon, multi-turn generation in hybrid navigation-manipulation tasks.

03

HTEWorld Benchmark

We establish HTEWorld, which provides 125K training clips and 300 evaluation trajectories, as a dataset and benchmark for long-horizon embodied video generation in hybrid navigation-manipulation tasks. Experiments show that WEM achieves SOTA performance on HTEWorld while remaining competitive on existing manipulation-oriented evaluations.

World-Ego Model

Separate conditioning information. Specialize generation. Preserve shared spatial and temporal structure.

World-Ego State Predictor

A vision-language model extracts world and ego states using Role-Conditioned Attention (RCA). World queries attend to historical vision-language context; ego queries focus on the most recent video and current instruction. Asymmetric query budgets allocate more capacity to the longer world history.

SR-MoE Video Generator

A shared Preceding Expert processes all video tokens. A Semantic Head predicts world-ego assignments, routing tokens to specialized World and Ego Experts. Neighbor-Expanded Assignment provides overlapping context near their boundaries before the outputs are recombined.

WEM architecture: role-conditioned world and ego state prediction followed by semantic routing to specialized video-generation experts
Overview of WEM. The state predictor separates conditioning information, and the SR-MoE generator combines shared video processing with role-specific generation. Chunk-wise autoregressive generation extends this process over successive instructions.

HTEWorld

The Hybrid-Task Embodied World Benchmark evaluates long-horizon video generation with interleaved navigation and manipulation. Built from teleoperated demonstrations released for the 2025 BEHAVIOR Challenge in BEHAVIOR-1K, it uses head-camera RGB observations from the Galaxea R1 Pro robot across ten household tasks. Five episodes per task are held out for evaluation.

HTEWorld provides 125K training clips with over 4.5M frames and 300 multi-turn evaluation trajectories covering over 2K instructions. Clips contain 37 frames at 16 FPS and 480 × 480 resolution. The primary evaluation uses WorldArena’s 16 metrics and EWMScore.

HTEWorld statistics: annotation vocabulary, dataset composition, and instruction-round distribution across evaluation trajectories
Statistics of HTEWorld. Left: word-frequency visualization of manipulation, navigation, object, and scene annotations. Middle: dataset composition and annotation statistics. Right: instruction-round distribution and navigation/manipulation proportions across evaluation trajectories.

Quantitative Results

Evaluation across long-horizon navigation-manipulation tasks, tabletop manipulation, and real-world robot videos.

Experiments on HTEWorld

WorldArena metrics and EWMScore. Higher is better; metric abbreviations and column order follow the paper.

ModelEWMScore ↑VisualMotionConsistency3DControlPhysics
IQAQJEPADDFlowMSBCSCPhotoDATAPerspSemActInstInter
WoW-7B53.4464.7249.741.3022.7625.4967.7466.8663.0638.0882.4128.1695.1486.753.4778.4280.90
Cosmos-Predict 2.5-2B54.8364.4050.211.2624.6227.2369.3371.5466.8842.5382.1128.7895.2887.053.5279.6083.02
Cosmos-Predict 2.5-14B55.4162.1450.021.3829.3732.3471.6373.8568.6535.4882.6028.8194.4087.463.5580.2084.70
PAN (Wan2.1)56.5864.8749.011.2039.6544.0077.0182.9975.9529.8779.6426.2695.2386.223.1269.1481.12
PAN (Wan2.2)58.4065.4849.001.7038.1447.4379.4780.2474.7933.1582.3928.7595.0887.674.4280.4086.33
WEM (Wan2.1)60.1165.0249.011.9238.9547.4082.6188.1981.9534.2283.6931.9195.3388.494.0287.7281.32
WEM (Wan2.2)61.4866.8250.302.4941.5249.2182.7087.9282.0735.9584.5534.5197.6090.744.5082.0090.80

WEM reaches 61.48 with Wan2.2 and 60.11 with Wan2.1, improving over the corresponding PAN baselines by 3.08 and 3.53 points.

Experiments on WorldArena

The condition is the control signal supplied in addition to visual context.

ModelConditionEWMScore ↑
Wan2.2Text54.54
WoWText54.88
CogVideoXText57.90
WEM (Wan2.2)Text58.10
Cosmos-Predict 2.5Action55.91
IRASimAction58.12
Ctrl-WorldAction59.70

WEM leads the compared text-conditioned methods. Other scores are reported by WorldArena.

Experiments on DROID

Real-world manipulation videos. PAN and WEM use Wan2.2.

ModelEWMScore ↑
Wan2.246.44
PAN50.83
WEM53.70

WEM improves over PAN by 2.87 points and over Wan2.2 by 7.26 points.

Qualitative Results

Representative rollouts show scene consistency and instruction following across interleaved navigation and manipulation.

Comparison with Baselines

More WEM Outputs

  1. 1Move toward the soda can with a trash can.
  2. 2Continue forward with the trash can.
  3. 3Stop in front of the soda can.
  4. 4Move closer to the soda can.
  5. 5Lower the right arm to the can.
  6. 6Pick up the soda can.
  7. 7Raise the soda can.
  8. 8Drop the soda can into the trash can.
  1. 1Reach for the hinged jar lid.
  2. 2Open the lid while holding the jar.
  3. 3Move left with the open jar.
  4. 4Move forward through the kitchen.
  5. 5Approach the wooden countertop.
  6. 6Continue toward the countertop.
  7. 7Stop beside the countertop with the jar.
  1. 1Move toward the grey trash can.
  2. 2Continue through the room toward the trash can.
  3. 3Lower the left arm to grasp the trash can.
  4. 4Lift the trash can.
  5. 5Turn right while holding the trash can.

BibTeX

@article{wem2026,
  title={World-Ego Modeling for Embodied Video Generation in Long-Horizon Navigation-Manipulation Tasks},
  author={Lin, Zuyao and Zhang, Jianhui and Jia, Peidong and Zhao, Xiaoguang and Zhang, Shanghang and Wang, Jingdong and Chen, Xingyu},
  journal={arXiv preprint arXiv:2605.19957},
  year={2026}
}