Evaluation on Original Image Spacings vs Evaluation on Test Data

[dannyhow12]

Hello

In regards to the function of evaluating original image spacings with the validation data, is there a way that we could also evaluate the metrics of the testing dataset during inferencing? I believe this was also mentioned by #663 .

Currently in the spleen segmentation tutorial, the segmentation map can be generated with the model that I have trained, but I seem to encounter an error when trying to evaluate the metrics of the testing data. Would appreciate if anyone could point out on how to perform this.

Here I share the snippet of the code:

  with torch.no_grad():
      for test_data in test_org_loader:
          test_inputs = test_data["image"].to(device)
          roi_size = (160, 160, 160)
          sw_batch_size = 4
          test_data["pred"] = sliding_window_inference(
              test_inputs, roi_size, sw_batch_size, model)
  
          test_data = [post_transforms(i) for i in decollate_batch(test_data)]
          test_outputs, test_labels = from_engine(["pred"])(test_data)
          # compute metric for current iteration
          dice_metric(y_pred=test_outputs, y=test_labels)
  
      # aggregate the final mean dice result
      metric_org = dice_metric.aggregate().item()
      # reset the status for next validation round
      dice_metric.reset()
  
  print("Metric on original image spacing: ", metric_org)

Error message

  ValueError: not enough values to unpack (expected 2, got 1)

[Nic-Ma]

Hi @dannyhow12 ,

May I know which line raised the error?

Thanks.

[dannyhow12]

Thanks for the prompt reply @Nic-Ma , initially the error popped up because I set to_onehot = None, as when I need to save the prediction map, it would need to be set to None. After setting it back to to_onehot = 3, it solved the problem. Also, I added a feature where I could divide each dice score of the classes, thereby showing their scores respectively.

I am now able to produce respective score for Kidney and Tumor based on the workaround of the Evaluation on Original Image Spacings for the validation data. I attach the code here if anyone would need it.

test_org_transforms = Compose( 
    [
        LoadImaged(keys=["image", "label"]),
        EnsureChannelFirstd(keys=["image", "label"]),
        Orientationd(keys=["image"], axcodes="RAS"),
        Spacingd(keys=["image"], pixdim=(
            1.62, 1.62, 3.22), mode="bilinear"),
        ScaleIntensityRanged(
            keys=["image"], a_min=-79, a_max=304,
            b_min=0.0, b_max=1.0, clip=True,
        ),
        CropForegroundd(keys=["image"], source_key="image"),
        EnsureTyped(keys=["image", "label"]),
    ]
)

  test_org_ds = Dataset(data=test_data, transform=test_org_transforms)
  test_org_loader = DataLoader(test_org_ds, batch_size=1, num_workers=2)
  
  post_transforms = Compose([
      EnsureTyped(keys="pred"),
      Invertd(
          keys="pred",
          transform=test_org_transforms,
          orig_keys="image",
          meta_keys="pred_meta_dict",
          orig_meta_keys="image_meta_dict",
          meta_key_postfix="meta_dict",
          nearest_interp=False,
          to_tensor=True,
      ),
      AsDiscreted(keys="pred", argmax=True, to_onehot=3),
      # SaveImaged(keys="pred", meta_keys="pred_meta_dict", output_dir="./out", output_postfix="seg", resample=False),
      # SaveImaged(keys="pred", meta_keys="pred_meta_dict", output_dir="/content/drive/MyDrive/data_trytotest", output_postfix="seg999", 
      #            resample=True, mode = {"bilinear", "nearest"}, separate_folder = True),
      AsDiscreted(keys="label", to_onehot=3),
  
  ])
  
  from monai.transforms import LoadImage
  loader = LoadImage()
  
  root_dir = "/content/drive/MyDrive/data_try/"
  
  model.load_state_dict(torch.load(
      os.path.join(root_dir, "best_metric_model_1.pth")))
  model.eval()
  
  
  dice_metric = DiceMetric(include_background=False, reduction="mean")
  dice_metric_batch = DiceMetric(include_background = False, reduction = "mean_batch") #splitting metric into corresponding kidney, tumor, background
  print(dice_metric_batch)
  
  with torch.no_grad():
      for test_data in test_org_loader:
          test_inputs = test_data["image"].to(device)
          roi_size = (160, 160, 160)
          sw_batch_size = 4
          test_data["pred"] = sliding_window_inference(
              test_inputs, roi_size, sw_batch_size, model)
  
          test_data = [post_transforms(i) for i in decollate_batch(test_data)]
  
          # To compute dice score
          test_outputs, test_labels = from_engine(["pred", "label"])(test_data)
          # compute metric for current iteration
          dice_metric(y_pred=test_outputs, y=test_labels)
          dice_metric_batch(y_pred = test_outputs, y = test_labels)
  
      # aggregate the final mean dice result
      metric_org = dice_metric.aggregate().item()
      metric_batch = dice_metric_batch.aggregate()
      metric_kidney = metric_batch[0].item()
      metric_tumor = metric_batch[1].item()
  
      # reset the status for next validation round
      # dice_metric.reset()
      dice_metric_batch.reset()
  
  print("Metric on original image spacing: ", metric_org)
  print("Metric on Kidney image spacing: ", metric_kidney)
  print("Metric on Tumor image spacing: ", metric_tumor)
  
  #         # uncomment the following lines to visualize the predicted results
  #         test_output = from_engine(["pred"])(test_data)
  
  #         original_image = loader(test_data[0]["image_meta_dict"]["filename_or_obj"])[0]
  
  #         plt.figure("check", (18, 6))
  #         plt.subplot(1, 2, 1)
  #         plt.imshow(original_image[:, :, 20], cmap="gray")
  #         plt.subplot(1, 2, 2)
  #         plt.imshow(test_output[0].detach().cpu()[1, :, :, 20])
  #         plt.show()