Monai Label DeepEdit Out of Memory

[nathanjbeaumont]

Hello,
I am following the MONAI Label DeepEdit Annotation With 3DSlicer tutorial and when I click "next sample" the volume loads but the premade label in Task02_Heart\labelsTr doesn't load.

Then when I click to make a foreground or background point a memory error occurs on the server.

  File "C:\Users\natha\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\functional.py", line 2327, in instance_norm
    return torch.instance_norm(
RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 8.00 GiB total capacity; 4.46 GiB already allocated; 661.75 MiB free; 5.53 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I have an RTX 2070 with 8GiB of memory I would expect it could handle running this AI.

Also I only see an "organ" label in 3D Slicer, whereas the tutorial shows the "left atrium" label, I think because my vtkMRMLSegmentationNode only has an "organ" segment.
Tutorial Slicer:

My Slicer:

Is 8GiB just not enough for DeepEdit?

[diazandr3s]

Hi @nathanjbeaumont,

Thanks for opening this discussion thread.

For demo purposes, we've trained the DeepEdit using an images size of 256x256x128 (https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/deepedit/main.py#L94) on the Medical Segmentation Decathlon.

This image size needs more GPU memory. However, you could re-train the model using the dataset and smaller image size (i.e 128x128x64).

Another way of experimenting with MONAI Label is using Segmentation App. As this is a less complex model and it works on patches, it uses less GPU memory.

Regarding this point:

Also I only see an "organ" label in 3D Slicer, whereas the tutorial shows the "left atrium" label, I think because my vtkMRMLSegmentationNode only has an "organ" segment.

It is just a label name. You could change it here: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/deepedit/lib/infer.py#L44

Hope this helps. Please let us know if more information is needed.