[BUG] Cannot Tag Added Disks

[irishgordo]

Describe the bug
On Harvester master-head (09/20/24) ( master-9d5b63e-head ) after adding tags to main disk & force-formatting / adding a secondary disk to the host, it seems unable to actually add tags to the secondary disks on the host -> causing an inabillity to create secondary storage-class for targetting additional disks on each node.

To Reproduce
Steps to reproduce the behavior:

1. Have a multi or single-node cluster, with the node(s) having a secondary disk
2. Try to force format & tag that secondary disk
3. Seems to be hung just at Force Formatted, node-disk manager logs will just talk about seeing new device path but won't seem to allow it to continue

Expected behavior
Adding secondary disks to work

Support bundle
supportbundle_622ebe9e-a36e-48b2-a11c-e06ac50fc799_2024-09-23T18-11-26Z.zip

Environment

• Harvester ISO Version: master-9d5b63e-head
• Underlying Infrastructure: bare-metal, HP DL160 Gen9s

Additional context
Each node, seems to just show hung at "Force Format" as "Yes".

Log from node-disk-manager:

harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="Node Disk Manager a6b3584 (a6b3584) is starting"                                                                                      
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="Notable parameters are following:"                                                                                                    
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="Namespace: longhorn-system, ConcurrentOps: 5, InjectUdevMonitorError: false"                                                          
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="Starting node disk manager controller"                                                                                                
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="register exclude filters"                                                                                                             
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="register auto provision filters"                                                                                                      
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="Node Disk Manager a6b3584 (a6b3584) is starting"                                                                                      
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="Detected the disk with block device /dev/sdb, id(Name): 9a5450545894c1d39e1be395a4d9e187 on node node7-test"                          
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="Notable parameters are following:"                                                                                                    
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="  - wwn: 0x600508b1001c443e026ec0eb2802e341"                                                                                          
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="  - vendor: HP"                                                                                                                       
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="  - model: LOGICAL_VOLUME"                                                                                                            
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="Namespace: longhorn-system, ConcurrentOps: 5, InjectUdevMonitorError: false"                                                          
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="Starting node disk manager controller"                                                                                                
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="  - SerialNumber: 600508b1001c443e026ec0eb2802e341"                                                                                   
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=warning msg="Device Path should not change with the active device 9a5450545894c1d39e1be395a4d9e187, new device path: /dev/sdb"                  
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="Wake up scanner first time to update CacheDiskTags ..."                                                                               
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:05Z" level=info msg="Waiting new event to trigger..."                                                                                                      
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:06Z" level=info msg="Starting harvesterhci.io/v1beta1, Kind=BlockDevice controller"                                                                        
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="register exclude filters"                                                                                                             
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="register auto provision filters"                                                                                                      
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="Detected the disk with block device /dev/sdb, id(Name): b180f3c7b6365983b50c4e8aa503b866 on node test-node8"                          
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:06Z" level=info msg="Starting harvesterhci.io/v1beta1, Kind=LVMVolumeGroup controller"                                                                     
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:06Z" level=info msg="Skip device 9a5450545894c1d39e1be395a4d9e187 as no provisioner found or not configured"                                               
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:06Z" level=info msg="Start monitoring udev processed events"                                                                                               
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="  - wwn: 0x600508b1001cbe7cfeb925286d40216f"                                                                                          
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="  - vendor: HP"                                                                                                                       
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="  - model: LOGICAL_VOLUME"                                                                                                            
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="  - SerialNumber: 600508b1001cbe7cfeb925286d40216f"                                                                                   
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:06Z" level=info msg="Starting longhorn.io/v1beta2, Kind=Node controller"                                                                                   
harvester-node-disk-manager-bq9pn time="2024-09-23T16:39:13Z" level=info msg="Skip device 9a5450545894c1d39e1be395a4d9e187 as no provisioner found or not configured"                                               
harvester-node-disk-manager-bq9pn time="2024-09-23T18:07:10Z" level=info msg="Skip device 9a5450545894c1d39e1be395a4d9e187 as no provisioner found or not configured"                                               
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=warning msg="Device Path should not change with the active device b180f3c7b6365983b50c4e8aa503b866, new device path: /dev/sdb"                  
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="Wake up scanner first time to update CacheDiskTags ..."                                                                               
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="Waiting new event to trigger..."                                                                                                      
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="Starting harvesterhci.io/v1beta1, Kind=LVMVolumeGroup controller"                                                                     
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="Starting harvesterhci.io/v1beta1, Kind=BlockDevice controller"                                                                        
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="Skip device b180f3c7b6365983b50c4e8aa503b866 as no provisioner found or not configured"                                               
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="Start monitoring udev processed events"                                                                                               
harvester-node-disk-manager-qcbwb time="2024-09-23T16:59:08Z" level=info msg="Starting longhorn.io/v1beta2, Kind=Node controller"                                                                                   
harvester-node-disk-manager-qcbwb time="2024-09-23T18:07:31Z" level=info msg="Skip device b180f3c7b6365983b50c4e8aa503b866 as no provisioner found or not configured"                                               
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="Node Disk Manager a6b3584 (a6b3584) is starting"                                                                                      
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:13Z" level=info msg="Node Disk Manager a6b3584 (a6b3584) is starting"                                                                                      
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="Notable parameters are following:"                                                                                                    
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="Namespace: longhorn-system, ConcurrentOps: 5, InjectUdevMonitorError: false"                                                          
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:13Z" level=info msg="Notable parameters are following:"                                                                                                    
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:13Z" level=info msg="Namespace: longhorn-system, ConcurrentOps: 5, InjectUdevMonitorError: false"                                                          
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:13Z" level=info msg="Starting node disk manager controller"                                                                                                
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="Starting node disk manager controller"                                                                                                
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="register exclude filters"                                                                                                             
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="register auto provision filters"                                                                                                      
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="Detected the disk with block device /dev/sdb, id(Name): 8fb31d167256669f230182713b14eaf1 on node node5-test"                          
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="  - wwn: 0x600508b1001cec28a12a38168f7bb195"                                                                                          
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="  - vendor: HP"                                                                                                                       
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="  - model: LOGICAL_VOLUME"                                                                                                            
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:13Z" level=info msg="register exclude filters"                                                                                                             
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:13Z" level=info msg="register auto provision filters"                                                                                                      
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:14Z" level=info msg="Detected the disk with block device /dev/sdb, id(Name): 4d985371a2c43ec35d46f0f5c5b79ada on node node6-test"                          
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:14Z" level=info msg="  - wwn: 0x600508b1001c89f560d91370f6f778b2"                                                                                          
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:14Z" level=info msg="  - vendor: HP"                                                                                                                       
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:14Z" level=info msg="  - model: LOGICAL_VOLUME"                                                                                                            
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:14Z" level=info msg="  - SerialNumber: 600508b1001c89f560d91370f6f778b2"                                                                                   
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:19Z" level=warning msg="Device Path should not change with the active device 4d985371a2c43ec35d46f0f5c5b79ada, new device path: /dev/sdb"                  
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:19Z" level=info msg="Wake up scanner first time to update CacheDiskTags ..."                                                                               
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:19Z" level=info msg="Waiting new event to trigger..."                                                                                                      
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:19Z" level=info msg="Starting harvesterhci.io/v1beta1, Kind=BlockDevice controller"                                                                        
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="  - SerialNumber: 600508b1001cec28a12a38168f7bb195"                                                                                   
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="Wake up scanner first time to update CacheDiskTags ..."                                                                               
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="Waiting new event to trigger..."                                                                                                      
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="Starting harvesterhci.io/v1beta1, Kind=BlockDevice controller"                                                                        
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="Starting harvesterhci.io/v1beta1, Kind=LVMVolumeGroup controller"                                                                     
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="Skip device 8fb31d167256669f230182713b14eaf1 as no provisioner found or not configured"                                               
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:19Z" level=info msg="Starting harvesterhci.io/v1beta1, Kind=LVMVolumeGroup controller"                                                                     
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="Start monitoring udev processed events"                                                                                               
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:19Z" level=info msg="Skip device 4d985371a2c43ec35d46f0f5c5b79ada as no provisioner found or not configured"                                               
harvester-node-disk-manager-blsn2 time="2024-09-23T16:40:22Z" level=info msg="Starting longhorn.io/v1beta2, Kind=Node controller"                                                                                   
harvester-node-disk-manager-blsn2 time="2024-09-23T18:06:27Z" level=info msg="Skip device 8fb31d167256669f230182713b14eaf1 as no provisioner found or not configured"                                               
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:19Z" level=info msg="Start monitoring udev processed events"                                                                                               
harvester-node-disk-manager-d5dws time="2024-09-23T16:39:19Z" level=info msg="Starting longhorn.io/v1beta2, Kind=Node controller"                                                                                   
harvester-node-disk-manager-d5dws time="2024-09-23T18:06:48Z" level=info msg="Skip device 4d985371a2c43ec35d46f0f5c5b79ada as no provisioner found or not configured"    

[irishgordo]

This doesn't seem to be present on Version: v1.4.0-dev-20240830, at least running virtualized

[Vicente-Cheng]

Thanks @irishgordo,
This is already fixed with harvester/node-disk-manager#135.

[irishgordo]

Tried on Harvester Version: master-aa5ddfc-head -> from 9/23, but guess not recent enough.
Will retry on 9/24 master-head

[irishgordo]

@Vicente-Cheng -this looks good on:

• master-0bde441-head
  I'll go ahead and close this out 😄 👍
  Did notice though:
• [BUG] master-0bde441-head cannot build storage classes #6631

[Vicente-Cheng]

@Vicente-Cheng -this looks good on:

• master-0bde441-head
  I'll go ahead and close this out 😄 👍
  Did notice though:
• [BUG] master-0bde441-head cannot build storage classes #6631

Thanks @irishgordo for taking care of this!