Failed to setup SSH connection: [Unhandled exception occurred: [Errno 110] Connection timed out]

[KRK-Bowman]

Setting up a remote backup job and getting this error.

virtnbdbackup -U qemu+ssh://user@server/system --ssh-user user -d vm01 -l full -o /backup/vm01 -v
[2024-10-18 11:56:51] INFO lib common - printVersion [main]: Version: 2.15 Arguments: /usr/bin/virtnbdbackup -U qemu+ssh://user@server/system --ssh-user user -d vm01 -l full -o /backup/vm01 -v
[2024-10-18 11:56:51] INFO root virtnbdbackup - main [main]: Backup level: [full]
[2024-10-18 11:56:51] DEBUG output target - create [main]: Create: /backup/vm01/checkpoints
[2024-10-18 11:56:51] DEBUG virt client - _connect [main]: Libvirt URI: [qemu+ssh://user@server/system]
[2024-10-18 11:56:52] INFO virt client - _connect [main]: Connected to remote host: [server]
[2024-10-18 11:56:52] INFO root virtnbdbackup - main [main]: Libvirt library version: [8000000]
[2024-10-18 11:56:52] INFO root virtnbdbackup - main [main]: NBD library version: [1.10.5]
[2024-10-18 11:56:52] DEBUG virt client - getDomainDisks [main]: Disk [sda]: file notation
[2024-10-18 11:56:52] INFO root disktype - Optical [main]: Skipping attached [cdrom] device: [sdb].
[2024-10-18 11:56:52] DEBUG virt client - getDomainDisks [main]: Device list: [DomainDisk(target='sda', format='qcow2', filename='vm01-sda', path='/var/lib/libvirt/images/vm01-sda', backingstores=[])]
[2024-10-18 11:56:52] DEBUG virt client - getDomainInfo [main]: No setting [loader] found: 'NoneType' object has no attribute 'text'
[2024-10-18 11:56:52] DEBUG virt client - getDomainInfo [main]: No setting [nvram] found: 'NoneType' object has no attribute 'text'
[2024-10-18 11:56:52] DEBUG virt client - getDomainInfo [main]: No setting [kernel] found: 'NoneType' object has no attribute 'text'
[2024-10-18 11:56:52] DEBUG virt client - getDomainInfo [main]: No setting [initrd] found: 'NoneType' object has no attribute 'text'
[2024-10-18 11:56:52] DEBUG virt client - getDomainInfo [main]: Domain Info: [{}]
[2024-10-18 11:56:52] INFO root virtnbdbackup - main [main]: Backup will save [1] attached disks.
[2024-10-18 11:56:52] INFO root virtnbdbackup - main [main]: Concurrent backup processes: [1]
[2024-10-18 11:56:52] INFO root checkpoint - create [main]: Loading checkpoints from: [/backup/vm01/vm01.cpt]
[2024-10-18 11:56:52] INFO root checkpoint - redefine [main]: Loading checkpoint list from: [/backup/vm01/checkpoints]
[2024-10-18 11:56:52] INFO root checkpoint - create [main]: Checkpoint handling.
[2024-10-18 11:56:52] DEBUG root checkpoint - removeAll [main]: Cleaning up persistent storage /backup/vm01/checkpoints
[2024-10-18 11:56:52] INFO root checkpoint - create [main]: Using checkpoint name: [virtnbdbackup.0].
[2024-10-18 11:56:52] DEBUG root checkpoint - create [main]: Checkpoint info: [{'name': 'virtnbdbackup.0', 'parent': '', 'file': '/backup/vm01/vm01.cpt'}].
[2024-10-18 11:56:52] INFO ssh client - connect [main]: Connecting remote system [server] via ssh, username: [user]
[2024-10-18 11:59:02] ERROR ssh client - connect [main]: [Errno 110] Connection timed out
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/libvirtnbdbackup/ssh/client.py", line 69, in connect
cli.connect(
File "/usr/lib/python3/dist-packages/paramiko/client.py", line 349, in connect
retry_on_signal(lambda: sock.connect(addr))
File "/usr/lib/python3/dist-packages/paramiko/util.py", line 279, in retry_on_signal
return function()
File "/usr/lib/python3/dist-packages/paramiko/client.py", line 349, in
retry_on_signal(lambda: sock.connect(addr))
TimeoutError: [Errno 110] Connection timed out
[2024-10-18 11:59:02] WARNING lib common - sshSession [main]: Failed to setup SSH connection: [Unhandled exception occurred: [Errno 110] Connection timed out]
[2024-10-18 11:59:02] ERROR root virtnbdbackup - main [main]: Remote backup detected but ssh session setup failed

Remote server (server) syslog shows:

libvirtd[4019713]: End of file while reading data: Input/output error

Tried 'aa-teardown' on both the local and remote systems and verified DNS connectivity, at a loss at what to try next.

Thanks in advance!

[abbbi]

The ssh session to the Remote system runs into timeout..

[KRK-Bowman]

Can the timeout value be increased somehow? (Thanks for the quick reply BTW!)

[abbbi]

not at the moment, is the target system reachable via regular ssh command?

From the logging it seems its able to reach the libvirt daemon via ssh, but the regluar paramiko ssh session then runs into
timeout. Thats pretty strange since both layers use basic ssh to connect.. You might want to check the ssh / journal logfiles on the remote system for possible reasons.

[KRK-Bowman]

You might want to check the ssh / journal logfiles on the remote system for possible reasons.

Nothing obviously out of sorts there?:

Oct 18 11:48:46 server sshd[3412528]: Accepted publickey for user from 192.168.60.95 port 41506 ssh2: RSA SHA256:Bn5*************************++Ujku+etCqgs
Oct 18 11:48:46 server sshd[3412528]: pam_unix(sshd:session): session opened for user user(uid=1000) by (uid=0)
Oct 18 11:50:56 server sshd[3412628]: Received disconnect from 192.168.60.95 port 41506:11: disconnected by user
Oct 18 11:50:56 server sshd[3412628]: Disconnected from user user 192.168.60.95 port 41506
Oct 18 11:50:56 server sshd[3412528]: pam_unix(sshd:session): session closed for user user

[KRK-Bowman]

It is.

The target is located over a SD-WAN link so the connection is 'slow' compared to a LAN-connected host.
iperf is showing 359 Mbits/sec link between the hosts.
Is that too slow for this to work using the remote backup function?

[abbbi]

well, not.. the ssh session should work, backup would be slow, though.
This looks like a general connectivity issue to the remote system for me.

[KRK-Bowman]

Alright, no worries, I'll work around it.
I appreciate the responses and the work on this project!