Cuda Assert Triggered

[dannyhow12]

Hi! Thank you to all contributors for this wonderful repo, really is a great source for beginners like myself

I am making use of the spleen_segmentation_3D.ipynb to perform segmentation on the dataset that I currently have, which is the dataset downloaded from the Kidney and Kidney Tumor Segmentation Challenge 2021. There are 4 classes defined in the dataset, where it includes background, kidney, tumor and cyst. However, in my implementation, I removed the cyst label, thereby only leaving it with 3 classes.

I am using Google Colab to run this code, with the GPU enabled. This is the config printed out:

  MONAI version: 0.9.dev2219
  Numpy version: 1.21.6
  Pytorch version: 1.11.0+cu113
  MONAI flags: HAS_EXT = False, USE_COMPILED = False
  MONAI rev id: c79f0d7e3254a3ea98a5e4cbebf1897f8b0f8069
  MONAI _file: /usr/local/lib/python3.7/dist-packages/monai/init_.py
  
  Optional dependencies:
  Pytorch Ignite version: 0.4.8
  Nibabel version: 3.0.2
  scikit-image version: 0.18.3
  Pillow version: 7.1.2
  Tensorboard version: 2.8.0
  gdown version: 4.4.0
  TorchVision version: 0.12.0+cu113
  tqdm version: 4.64.0
  lmdb version: 0.99
  psutil version: 5.4.8
  pandas version: 1.3.5
  einops version: NOT INSTALLED or UNKNOWN VERSION.
  transformers version: NOT INSTALLED or UNKNOWN VERSION.
  mlflow version: NOT INSTALLED or UNKNOWN VERSION.
  
  For details about installing the optional dependencies, please visit:
      https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

While this is the error generated when I ran the block for Execute a typical PyTorch Training Process.

   RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/spleen_segmentation_3d.ipynb this is the link to the code that I am using.

Note that I only changed the variables such as

1. out_channels = 3 (background, kidney and tumor)
2. and with Include Background = True at DiceMetric
3. to_onehot = 3 for post_pred and post_label

I have also made sure that my labels are starting with 0,1,2,3 instead of 1,2,3,4, where I am able to find some sources that mention this could be causing the issue.

Also, the error code pointed that this line was causing the problem, but I am unsure what is the problem with this

  dice_metric(y_pred=val_outputs, y=val_labels)

Thanks for the attention

Regards,
Danny

[Nic-Ma]

Hi @dannyhow12 ,

Do you still have the issue if removed the metrics computation? I suspect it may be CUDA out of memory, can you reduce your image size in transforms and try again?
And I am not sure whether there is an environment issue, can you run the original spleen notebook successfully? Maybe you can have a try as the spleen dataset is not very big.

Thanks.

[dannyhow12]

Hi @Nic-Ma ,

Surprisingly, I was not able to reproduce the issue when I restarted my Google Colab Jupyter Notebook. Unsure what is causing this issue. Nevertheless, the pipeline is now working fine for my kidney CT scan images. Thank you for the help!

[Nic-Ma]

Sounds good!

Thanks.