Weird behavior of Dice Metric

[cosminbvb]

Hi, I followed the semantic segmentation tutorial notebooks and I have some questions regarding them:

1. In spleen_segmentation_3d.ipynb, why does the approach use 2 output channels and softmax instead of 1 output channel and sigmoid?
2. Same question for /3d_segmentation/challenge_baseline/run_net.py.

Also, I can't seem to understand the documentation for DiceMetric:

1. It states that it computes the Dice Loss. Shouldn't it be the Dice Coefficient instead?
2. What is the correct way of using it? The docs says "y_preds is expected to have binarized predictions and y should be in one-hot format. ". By that I understand that the ground truth should be one hot formatted while the prediction should't. If that's the case, why are the prediction and gt both not in one hot encoding in 3d_segmentation/torch/unet_training_dict.py for example.
3. I tried testing the DiceMetric with the following code snippet:

a = torch.zeros(1, 5, 5, 5)
a[0, 2:4, 2:4, 2:4] = 1
b = torch.zeros(1, 5, 5, 5)
# b[0, 2:4, 2:4, 2:4] = 1
dice = DiceMetric(include_background=True)
dice(a, b)
coeff = dice.aggregate().item()
print(coeff)  # => 0.0

post_trans = Compose([EnsureType(), AsDiscrete(to_onehot=2)])
dice2 = DiceMetric(include_background=True)
dice2(post_trans(a), post_trans(b))
coeff = dice2.aggregate().item()
print(coeff)  # => 0.9652174115180969

# The following produces the error: y_pred and y should have same shapes.
# dice3 = DiceMetric(include_background=True)
# dice3(y_pred=a, y=post_trans(b))
# coeff = dice3.aggregate().item()
# print(coeff)

In my understanding, the first approach does not follow the documentation. The second one, similar to the way it is used in spleen_segmentation_3d.ipynb gives wrong results. The third one, which comes closer to the documentation, produces an error. Could anyone explain what's the correct way of using it?

I had 3 approaches for solving a binary segmentation on CTs and obtained some very unexpected results, with the same model and the same hyperparameters. The only thing that differ are: last activation function and the post transformations applied before computing dice metric.

1. I trained a 3d UNet with 1 output channel (and sigmoid activation prior to the loss function). At evaluation, after sliding window inference, I do the following post transformations and plug them into dice metric:

post_trans_output = Compose([EnsureType(), Activations(sigmoid=True), AsDiscrete(threshold=0.5, to_onehot=2)])
post_trans_gt = Compose([EnsureType(), AsDiscrete(to_onehot=2)])
outputs = [post_trans_output(x) for x in decollate_batch(outputs)]
gt = [post_trans_gt(x) for x in decollate_batch(gt)]

I achieve a best dice coefficient of 75%.

2. I trained a 3d UNet with 1 output channel (and sigmoid activation prior to the loss function). At evaluation, after sliding window inference, I do the following post transformations and plug them into dice metric:

post_trans = Compose([EnsureType(), Activations(sigmoid=True), AsDiscrete(threshold=0.5)])
outputs = [post_trans(x) for x in decollate_batch(outputs)]

I achieve a best dice coefficient of 60% (with the exact same code except for the two lines above).
Can anybody explain why is there such a difference?

3. I trained another 3d UNet but this time with 2 output channels (and softmax activation prior to the loss function). At evaluation, after sliding window inference, I do the following post transformations and plug them into dice metric:

post_trans_output = Compose([EnsureType(), AsDiscrete(argmax=True, to_onehot=2)])
post_trans_gt = Compose([EnsureType(), AsDiscrete(to_onehot=2)])
outputs = [post_trans_output(x) for x in decollate_batch(outputs)]
gt = [post_trans_gt(x) for x in decollate_batch(gt)]

With this I achieve a best dice coefficient of 78%.

If there is anyone that could take the time to answer the question above and / or explain why are there such differences between the approaches, I would very much appreciate it! Thank you!

[Nic-Ma]

Same as Project-MONAI/MONAI#4310.

Thanks.