Use CacheDataset for DDP

[hw-ju]

Hi!
In brats_training_ddp.py, BratsCacheDataset is implemented to "enhance the DecathlonDataset to support distributed data parallel" and it will "partition dataset based on current rank number, every rank trains with its own data. it can avoid duplicated caching content in each rank, but will not do global shuffle before every epoch".

Can the script still work if we replace the BratsCacheDataset object with a CacheDataset object and a DistributedSampler object, to get a global shuffle before every epoch?

If it can work, then in this way, will the whole intermediate transformed dataset be cached on each GPU memory?

[Nic-Ma]

Hi @hw-ju ,

Yes, if you want to cache and global shuffle, every process will cache the whole data, because it can't shuffle across processes.

Thanks.

[hw-ju]

Thanks @Nic-Ma !