Meng Wei*  Chenyang Wan*  Xiqian Yu*  Tai Wang*‡  Yuqiang Yang  Xiaohan Mao  Chenming Zhu  Wenzhe Cai  Hanqing Wang  Yilun Chen  Xihui Liu†  Jiangmiao Pang† 
Shanghai AI Laboratory The University of Hong Kong Zhejiang University Shanghai Jiao Tong University 

StreamVLN generates action outputs from continuous video input in an online, multi-turn dialogue manner. Built on LLaVA-Video as the foundational Video-LLM, we extend it for interleaved vision, language, and action modeling. For both effective context modeling of long sequence and efficient computation for real-time interaction, StreamVLN has: (1) a fast-streaming dialogue context with a sliding-window KV cache; and (2) a slow-updating memory via token pruning.

[2025-09-28] We have updated the checkpoint which is trained on R2R_VLNCE_v1-3, achieving better results: R2R (NE:4.90, OS:63.6, SR:56.4, SPL:50.2) and RxR (NE:5.65, SR:54.4, SPL:45.4, nDTW:63.7). Please switch your training and testing data to R2R_VLNCE_v1-3 if you used R2R_VLNCE_v1 before.

[2025-08-28] We have released the code and guide for real-world deployment on a unitree Go2 robot.

[2025-08-21] We have released the code for the following components: 1) Dagger Data Collection; 2) Stage-Two Co-training with the LLaVA-Video-178K, ScanQA, and MMC4 datasets.

[2025-07-30] We have released the ScaleVLN training data, including a subset of ~150k episodes converted from the discrete environment setting to the VLN-CE format. For usage details, see here.

[2025-07-18] We’ve fixed a bug where num_history was not correctly passed to the model during evaluation, causing it to default to None. This had a significant impact on performance. Please make sure to pull the latest code for correct evaluation.

We test under the following environment:

• Python 3.9
• Pytorch 2.1.2
• CUDA Version 12.4

1. Preparing a conda env with Python3.9 & Install habitat-sim and habitat-lab

   conda create -n streamvln python=3.9
   conda install habitat-sim==0.2.4 withbullet headless -c conda-forge -c aihabitat
   git clone --branch v0.2.4 https://github.com/facebookresearch/habitat-lab.git
   cd habitat-lab
   pip install -e habitat-lab  # install habitat_lab
   pip install -e habitat-baselines # install habitat_baselines

2. Clone this repository

   git clone https://github.com/OpenRobotLab/StreamVLN.git
   cd StreamVLN

To get started, you need to prepare three types of data:

1. Scene Datasets

   • For R2R, RxR and EnvDrop: Download the MP3D scenes from the official project page, and place them under data/scene_datasets/mp3d/.
   • For ScaleVLN: Download the HM3D scenes from the official github page, and place the train split under data/scene_datasets/hm3d/

2. VLN-CE Episodes
   Download the VLN-CE episodes:

   • r2r (Rename R2R_VLNCE_v1/ -> r2r/)
   • rxr (Rename RxR_VLNCE_v0/ -> rxr/)
   • envdrop (Rename R2R_VLNCE_v1-3_preprocessed/envdrop/ -> envdrop/)
   • scalevln (This is a subset of the ScaleVLN dataset, converted to the VLN-CE format. For the original dataset, please refer to the official repository.)

   Extract them into the data/datasets/ directory.

3. Collected Trajectory Data
   We provide pre-collected observation-action trajectory data for training. These trajectories were collected using the training episodes from R2R and RxR under the Matterport3D environment. For the EnvDrop and ScaleVLN subset, please refer to here for instructions on how to collect it yourself. Download the observation-action trajectory data from Hugging Face, and extract it to data/trajectory_data/.

4. Co-training Data Preparation

   Download the respective datasets from their official sources and place them in the data/co-training_data/.

   • LLaVA-Video-178K: Available on Hugging Face at lmms-lab/LLaVA-Video-178K.

   • ScanNet:

     • The main dataset can be downloaded from the official GitHub repository.

     • Download the annotation files scanqa_annotations.json and sqa3d_annotations.json from here. These files are subsets of the LLaVA-3D-DATA.

   • MMC4-core: Available via the official GitHub repository.

Your final folder structure should look like this:

data/
├── datasets/
│   ├── r2r/
│   │   ├── train/
│   │   ├── val_seen/
│   │   │   └── val_seen.json.gz
│   │   └── val_unseen/
│   │       └── val_unseen.json.gz
│   ├── rxr/
│   │   ├── train/
│   │   ├── val_seen/
│   │   │   ├── val_seen_guide.json.gz
│   │   │   └── ...
│   │   └── val_unseen/
│   │       ├── val_unseen_guide.json.gz
│   │       └── ...
│   ├── envdrop/
│   │   ├── envdrop.json.gz
│   │   └── ...
│   └── scalevln/
│       └── scalevln_subset_150k.json.gz
├── scene_datasets/
│   └── hm3d/
│       ├── 00000-kfPV7w3FaU5/
│       ├── 00001-UVdNNRcVyV1/
│       └── ...
│   └── mp3d/
│       ├── 17DRP5sb8fy/
│       ├── 1LXtFkjw3qL/
│       └── ...
├── trajectory_data/
│   ├── R2R/
│   │   ├── images/
│   │   └── annotations.json
│   ├── RxR/
│   │   ├── images/
│   │   └── annotations.json
│   ├── EnvDrop/
│   │   ├── images/
│   │   └── annotations.json
│   └── ScaleVLN/
│       ├── images/
│       └── annotations.json
├── dagger_data/
│   ├── R2R/
│   │   ├── images/
│   │   └── annotations.json
│   ├── RxR/
│   │   ├── images/
│   │   └── annotations.json
│   └── EnvDrop/
│       ├── images/
│       └── annotations.json
└── co-training_data/
    ├── ScanNet/
    │   ├── posed_images/
    │   │   ├── scene0000_00/
    │   │   ├── scene0000_01/
    │   │   └── ...
    │   ├── scanqa_annotations.json
    │   └── sqa3d_annotations.json
    ├── LLaVA-Video-178K/
    │   ├── 0_30_s_academic_v0_1/
    │   │   ├── academic_source/
    │   │   │   ├── Charades/
    │   │   │   ├── NextQA/
    │   │   │   └── ...
    │   │   └── 0_30_s_academic_oe_v0_1_qa_processed.json
    │   ├── 30_60_s_academic_v0_1/
    │   │   ├── academic_source/
    │   │   │   ├── Charades/
    │   │   │   ├── NextQA/
    │   │   │   └── ...
    │   │   └── 30_60_s_academic_oe_v0_1_qa_processed.json
    │   └── 0_30_s_youtube_v0_1/
    │       ├── liwei_youtube_videos/
    │       └── 0_30_s_youtube_oe_v0_1_qa_processed.json
    └── MMC4-core/
            ├── images/
            ├── docs_shard_10000_v3.jsonl
            ├── docs_shard_10001_v3.jsonl
            └── ...

We provide two model checkpoints for different use cases:

• Benchmark Reproduction Use this checkpoint to reproduce results on the VLN-CE benchmark.

• Real-World Deployment This checkpoint is recommended for deployment on physical robots.

  We made two modifications:

  1. Remove redundant initial spinning actions: The initial left/right turns not mentioned in the instructions are removed for better instruction alignment.
  2. Trajectory safety: Enhanced obstacle avoidance ensures more reliable navigation in real-world environments.

1. Stage-one Training

   To perform multi-node multi-GPU training with distributed setup, run:

   sbatch scripts/streamvln_train_slurm.sh

2. Dagger Collection

   To perform multi-GPU collection, simply run:

   sh scripts/streamvln_dagger_collect.sh

3. Stage-two Training

   To perform multi-node multi-GPU training with distributed setup, run:

   sbatch scripts/streamvln_stage_two_train_slurm.sh

To perform multi-GPU evaluation with key-value cache support, simply run:

sh scripts/streamvln_eval_multi_gpu.sh

Please refer to realworld/realworld.md for real-world deployment on a unitree Go2 robot.

• ✅ Release the arXiv paper (Jul. 8, 2025)
• ✅ Provide inference scripts and model checkpoints
• ✅ Release training code and configurations
• ✅ Release training data
• ✅ Support co-training with LLaVA-Video-178K, ScanQA, MMC4
• ✅ Dagger data collection

If you encounter any problems or have questions about StreamVLN, please feel free to open an issue.

If you find our work helpful, please consider starring this repo 🌟 and cite:

@article{wei2025streamvln,
  title={StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling},
  author={Wei, Meng and Wan, Chenyang and Yu, Xiqian and Wang, Tai and Yang, Yuqiang and Mao, Xiaohan and Zhu, Chenming and Cai, Wenzhe and Wang, Hanqing and Chen, Yilun and others},
  journal={arXiv preprint arXiv:2507.05240},
  year={2025}
}

This work is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

This repo is based on LLaVA-NeXT.