Question: Which MMC4-core dataset variant to use for StreamVLN training?

[followingcode]

Hi, thank you so much for your great work on StreamVLN! It's really inspiring and helpful for our research.

I'm currently preparing the datasets for training and have all other data ready. I just need to acquire the MMC4-core dataset to start training.

From the official documentation, I see that the original mmc4-core is no longer available, and only the mmc4-core-ff (Multimodal-C4 core fewer-faces) version can be downloaded now. Could you please confirm if this mmc4-core-ff version is the one we should use for StreamVLN training?

Thanks again for your excellent work and looking forward to your reply!
@kellyiss @mxh1999 @yuqiang-yang

[chchnii]

You can check the details in this issue for reference. #51

[followingcode]

Thanks a lot for your prompt response and the reference link! I've gone through issue #51 carefully.
Just to double-check: should we use the mmc4-core-ff version for StreamVLN training, since the original mmc4-core is no longer available?
Thank you very much!
@chchnii

[chchnii]

Yes, mmc4-core-ff is a subset of mmc4-core and is suitable for StreamVLN training, as we only used a selected subset of mmc4-core (not the full dataset) for the original training.

[followingcode]

Thanks so much for your answer!
Just to quickly confirm: do you mean that we need to further select a smaller, curated subset from mmc4-core-ff to use as the training set for StreamVLN? Also, would it be possible for you to share the selection principles and general approach you used? This would really help us follow your training pipeline correctly.

[chchnii]

It's randomly selected, just maintain the data ratio.

[followingcode]

Thanks so much for your answer!
Do you mean that we need to further select a smaller, curated subset from mmc4-core-ff to use as the training set for StreamVLN?

[followingcode]

The code below is extracted from the official LazyMMC4Dataset implementation. It only keeps the last 1400 shard files and discards all others according to the sampling_strategy: end:1400 defined in the config.

The code only loads the last 1400 shards of the MMC4-core dataset.During training, all remaining shard files are completely ignored and discarded.

It is not randomly selected. Did I misunderstand it?

sampling_strategy = dataset.get("sampling_strategy", "all")
sampling_number = None
rank0_print(f"Loading {json_dir} with {sampling_strategy} sampling strategy")
if ":" in sampling_strategy:
sampling_strategy, sampling_number = sampling_strategy.split(":")
sampling_number = int(sampling_number)
if sampling_strategy == "first" and sampling_number is not None:
json_path_list = json_path_list[:sampling_number]
elif sampling_strategy == "end" and sampling_number is not None:
json_path_list = json_path_list[-sampling_number:]
elif sampling_strategy == "random" and sampling_number is not None:
random.shuffle(json_path_list)
json_path_list = json_path_list[:sampling_number]