Error when making a backup and in the process the machine turns off unexpectedly

[inode64]

Version used
1.9.42

Describe the bug
Error when a backup is made and in the process the machine turns off unexpectedly, and the following backups generate an error because the checkpoint is not found
This error has been happening to me for a long time, but Windows Server 2012r2 only shuts down unexpectedly about once every month.

Expected behavior
If this problem occurs, the checkpoints will not be updated

Hypervisor information:

• OS: Gentoo
• HV type Libvirt 9.4.0 and libnbd 1.18.1

Logfiles:

First backup
[2023-11-23 01:02:13] INFO lib common - printVersion [MainThread]: Version: 1.9.42 Arguments: /usr/lib/python-exec/python3.11/virtnbdbackup -S /tmp/ -d svrServicios --noprogress -q --compress -l auto -o /srv/backup/svrServicios/2023-47 --checkpointdir /srv/backup/.checkpoint/svrServicios
[2023-11-23 01:02:13] INFO root virtnbdbackup - main [MainThread]: Backup level: [auto]
[2023-11-23 01:02:13] INFO root virtnbdbackup - main [MainThread]: Compression enabled, level [2]
[2023-11-23 01:02:13] INFO root virtnbdbackup - main [MainThread]: Backup mode auto: executing incremental backup.
[2023-11-23 01:02:13] INFO root virtnbdbackup - main [MainThread]: Store checkpoints in: [/srv/backup/.checkpoint/svrServicios]
[2023-11-23 01:02:13] INFO root virtnbdbackup - main [MainThread]: Libvirt library version: [9004000]
[2023-11-23 01:02:13] INFO root disktype - Optical [MainThread]: Skipping attached [cdrom] device: [sda].
[2023-11-23 01:02:13] INFO root virtnbdbackup - main [MainThread]: Backup will save [1] attached disks.
[2023-11-23 01:02:13] INFO root virtnbdbackup - main [MainThread]: Concurrent backup processes: [1]
[2023-11-23 01:02:13] INFO root checkpoint - redefine [MainThread]: Loading checkpoint list from: [/srv/backup/.checkpoint/svrServicios]
[2023-11-23 01:02:13] INFO root checkpoint - create [MainThread]: Checkpoint handling.
[2023-11-23 01:02:13] INFO root checkpoint - create [MainThread]: Next checkpoint id: [4].
[2023-11-23 01:02:13] INFO root checkpoint - create [MainThread]: Parent checkpoint name [virtnbdbackup.3].
[2023-11-23 01:02:13] INFO root checkpoint - create [MainThread]: Using checkpoint name: [virtnbdbackup.4].
[2023-11-23 01:02:13] INFO root virtnbdbackup - main [MainThread]: Local NBD Endpoint socket: [/var/tmp/virtnbdbackup.157352]
[2023-11-23 01:02:13] INFO root virtnbdbackup - main [MainThread]: Temporary scratch file target directory: [/tmp/]
[2023-11-23 01:02:13] INFO root virtnbdbackup - startBackupJob [MainThread]: Starting backup job.
[2023-11-23 01:02:14] WARNING fs fs - freeze [MainThread]: error interno: End of file from agent socket
[2023-11-23 01:02:14] ERROR root virtnbdbackup - startBackupJob [MainThread]: Failed to start backup: [Operation is not supported: cannot perform disk backup for inactive domain]

Next Backup
[2023-11-24 01:04:03] INFO lib common - printVersion [MainThread]: Version: 1.9.42 Arguments: /usr/lib/python-exec/python3.11/virtnbdbackup -S /tmp/ -d svrServicios --noprogress -q --compress -l auto -o /srv/backup/svrServicios/2023-47 --checkpointdir /srv/backup/.checkpoint/svrServicios
[2023-11-24 01:04:03] INFO root virtnbdbackup - main [MainThread]: Backup level: [auto]
[2023-11-24 01:04:03] INFO root virtnbdbackup - main [MainThread]: Compression enabled, level [2]
[2023-11-24 01:04:03] INFO root virtnbdbackup - main [MainThread]: Backup mode auto: executing incremental backup.
[2023-11-24 01:04:03] INFO root virtnbdbackup - main [MainThread]: Store checkpoints in: [/srv/backup/.checkpoint/svrServicios]
[2023-11-24 01:04:03] INFO root virtnbdbackup - main [MainThread]: Libvirt library version: [9004000]
[2023-11-24 01:04:03] INFO root disktype - Optical [MainThread]: Skipping attached [cdrom] device: [sda].
[2023-11-24 01:04:03] INFO root virtnbdbackup - main [MainThread]: Backup will save [1] attached disks.
[2023-11-24 01:04:03] INFO root virtnbdbackup - main [MainThread]: Concurrent backup processes: [1]
[2023-11-24 01:04:03] INFO root checkpoint - redefine [MainThread]: Loading checkpoint list from: [/srv/backup/.checkpoint/svrServicios]
[2023-11-24 01:04:03] INFO root checkpoint - create [MainThread]: Checkpoint handling.
[2023-11-24 01:04:03] INFO root checkpoint - create [MainThread]: Next checkpoint id: [4].
[2023-11-24 01:04:03] INFO root checkpoint - create [MainThread]: Parent checkpoint name [virtnbdbackup.3].
[2023-11-24 01:04:03] INFO root checkpoint - create [MainThread]: Using checkpoint name: [virtnbdbackup.4].
[2023-11-24 01:04:03] INFO root virtnbdbackup - main [MainThread]: Local NBD Endpoint socket: [/var/tmp/virtnbdbackup.737235]
[2023-11-24 01:04:03] INFO root virtnbdbackup - main [MainThread]: Temporary scratch file target directory: [/tmp/]
[2023-11-24 01:04:03] INFO root virtnbdbackup - startBackupJob [MainThread]: Starting backup job.
[2023-11-24 01:04:15] INFO fs fs - freeze [MainThread]: Freezed [2] filesystems.
[2023-11-24 01:04:17] INFO fs fs - thaw [MainThread]: Thawed [2] filesystems.
[2023-11-24 01:04:17] ERROR root virtnbdbackup - startBackupJob [MainThread]: Failed to start backup: [checkpoint inconsistent: missing or broken bitmap 'virtnbdbackup.3' for disk 'vda']

Workaround:
No.

[abbbi]

hi,

i dont see a sane way virtnbdbackup could recover the situation for you. The issue is that the checkpoint has
been defined on libvirt level and is not existant in the qcow image! It seems the system / qemu or any other
component crashed before the checkpoint could be written to the qcow image.

You have to remove the half-defined checkpoint from the libvirt metadata using:

virsh checkpoint-delete svrServicios --checkpointname virtnbdbackup.3 --metadata

First of all you should clarify why your windows 2012 system crashes, it seems to me the crash happens
during filesystem freeze:

WARNING fs fs - freeze [MainThread]: error interno: End of file from agent socket

maybe the filesystem freeze using the qemu agent triggers some issue that leads to the crash. You could
try and see if it helps if you disable the qemu agent in the VM.

Im converting this to a discussion because its not likely a bug in virtnbdbackup, but the situation is caused
by malfunction in your OS or some other component. Additionally it may help to check the qemu / libvirt
logfiles for why the virtual machine crashes the way it does.

[abbbi]

#135 (comment)

[inode64]

Windows actually shows a blue screen of death and stops :-(
We have already updated all the drivers and the qemu-agent

Rather, what I think is if when the copy starts and the virtual machine stops during the backup process, the checkpoint should be deleted because neither the operation nor the backup has been completed. The following backup attempts always fail, the checkpoints must be manually deleted so that a backup can be performed correctly again.

[abbbi]

hi,

Rather, what I think is if when the copy starts and the virtual machine stops during the backup process, the checkpoint should be deleted because neither the operation nor the backup has been completed. The following backup attempts always fail, the checkpoints must be manually deleted so that a backup can be performed correctly again.

yes, it depends where the backup fails. If it fails during startBackup operation (as it does in your case), it could make sense
to remove the checkpoint just created, or at least attempt to remove it. But then it also depends in which state the virtual
machine is. If the virtual machine goes offline, i cannot remove checkpoints correctly because thats not allowed from the
libvirt api (checkpoint operations only allowed if vm is online)

In general it would be nice if virtnbdbackup could cope with it, but then there also situations where things
might get worse..

I need to think through at which points in the backup process it makes sense to remove the checkpoint during Error.
New enhancement ticket: #147

[inode64]

OK, thanks a lot,