DynUNet pipeline with data outside Decathlon dataset

[bip5]

Hi guys, I'm trying to use the DynUNet pipeline for BraTS' 21 dataset. However I'm finding the process is more involved than I expected it to be as the data pipeline is designed specifically for the decathlon dataset. I've started by modifying the datalist to be in the same format as the output from the built in decathlon loader but it feels like I'm going about this the wrong way.

Edit: I've now transferred my data into the required format except for changing the labels from (0,1,2,4] to (0,1,2,3). Is this necessary with the tutorial pipeline? Which portion of the pipeline do I need to change to achieve this switch at data loading?

Currently I'm getting an index out of bounds error and expecting this to be source of issue.

/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [3022,0,0], thread: [63,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
2022-07-06 22:00:20,656 - ignite.engine.engine.DynUNetTrainer - ERROR - Current run is terminating due to exception: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
2022-07-06 22:00:20,657 - ignite.engine.engine.DynUNetTrainer - ERROR - Exception: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
Traceback (most recent call last):
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/ignite/engine/engine.py", line 840, in _run_once_on_dataset
self.state.output = self._process_function(self, self.state.batch)
File "/scratch/a.bip5/BraTS 2021/nnUNet/./nnu.py", line 830, in _iteration
engine.state.output[Keys.LOSS].backward()
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/torch/_tensor.py", line 255, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/torch/autograd/init.py", line 147, in backward
Variable._execution_engine.run_backward(
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Thank you!

[Nic-Ma]

Hi @yiheng-wang-nv ,

Could you please help provide some comments for this question?

Thanks in advance.

[yiheng-wang-nv]

Hi @bip5 ,

Edit: I've now transferred my data into the required format except for changing the labels from (0,1,2,4] to (0,1,2,3). Is this necessary with the tutorial pipeline? Which portion of the pipeline do I need to change to achieve this switch at data loading?

How many classes does the brats 2021 dataset have? You may need to modify the hyper-parameters of the network in:

tutorials/modules/dynunet_pipeline/create_network.py

Line 52 in b429c88

out_channels=n_class,

[bip5]

There's 4 classes, 1 negative and 3 positive.

For an update, I changed the segmentation mask separately to have 0,1,2,3 instead of 0,1,2,4 and the training is working. I was hoping to be able to achieve this through the data loading pipeline.

Ordinarily, I'd apply the transform inside getitem for the dataset. I'm not sure how I'd apply this change with this pipeline since it uses a CacheDataset and returns a dataloader instead of a dict

tutorials/modules/dynunet_pipeline/create_dataset.py

Line 77 in b429c88

train_ds = CacheDataset(

[yiheng-wang-nv]

Hi @bip5 , you may need to extend the dataset class and write a customized one, for example:

tutorials/kaggle/RANZCR/4th_place_solution/data/seg_data.py

Line 14 in b429c88

class CustomDataset(Dataset):