Backup failing (flaky CIFS mount)

[jdaviescoates]

@girish yes:

@jdaviescoates said in Backup failing:

Backing up post-reboot worked fine. So my guess is that my Hetzner Storage Box had a wobble and somehow became unmounted mid backup.

Sounds like others have been experiencing similar too, so I'm guessing a (hopefully temporary) issue with Hetzner storage boxes.

Although like @robi said it'd be cool if there was some way Cloudron backups could try to survive such wobbles.

[girish]

@jdaviescoates Yes, at some point we want to make it possible to add mounts easily. When we do that, we can attempt to re-mount automatically, I guess.

[robi]

@girish not sure this is the right context..

this is about existing mounts and their resilience to disconnects.

• monitoring mounts for health
• allowing for self-healing if it exists from the protocol
• if not, try harder, remount command for example
• if still not, try unmount, then remount
• if still not, notify to reboot

[p44]

@robi right suggestions.

@girish I had this problem again: all backup folder this morning where unmounted...

[iqweb]

@p44 said in Backup failing:

@robi right suggestions.

@girish I had this problem again: all backup folder this morning where unmounted...

I've had the same thing again too

[girish]

@iqweb Are you also using CIFS/NFS?

[p44]

@iqweb Still on Hetzner storage?

[iqweb]

@girish said in Backup failing:

@iqweb Are you also using CIFS/NFS?

Yes - storage provider: CIFS Mount

@p44 said in Backup failing:

@iqweb Still on Hetzner storage?

Yes - Hetzner storage box

& again - I had to reboot to be able to remount.

[p44]

@iqweb Maybe Hetzner sometimes reboot Box instance? From my side is about one week CIFS mount is stable...

[nebulon]

I also have to remount my cifs share about once a month. In my case I thought it was related to my residential home internet connection. Either way I think we have to handle such mount points better and maybe even attempt a remount automatically.

[p44]

@nebulon Good idea to add a health-check to keep alive mounting

[girish]

@nebulon Does cifs mounting have some retry flags? (i.e the fstab options)

[nebulon]

@girish not that I have found and https://linux.die.net/man/8/mount.cifs does not mention anything.

Apparently systemd has some automounting feature https://www.freedesktop.org/software/systemd/man/systemd.automount.html which mounts the drive once the mountpoint is accessed by a process! Can't tell if it would reconnect such a mount though.

[girish]

@nebulon Maybe https://askubuntu.com/questions/1210867/remount-cifs-on-network-reconnect ? (just expands on your link)

[imc67]

@girish & @nebulon any news on this issue?

I started yesterday to backup via cifs to a Hetzner storagebox (2 backups a day), today the first backup failed due to

Command failed with exit code 1 (EPERM): df -kPT /backups_cifs

I solved it with a manual:

sudo umount /backups_cifs
sudo mount -a

But do I need to script this with a daily cron or is Cloudron able to recover from this somehow?

[p44]

@imc67 Dear, this happened because Hetzner did some maintenance on their Box Service.

Also a Cloudron restart solved problem.

[nebulon]

@p44 actually I hit the same thing today morning, had to remount the disk. I happily ignored the heads up warning email about the maintenance from hetzner

[p44]

@nebulon Yes, but not all my Cloudron installs has been affected, even if attached at the same Hetzner Box... Just restarted and all was operational again

[nebulon]

@p44 in such cases a remount alone would be enough, no need to reboot the whole server, if that is what you did. This would then not affect downtime. We will soon implement auto-remounting in such cases.

[p44]

@nebulon Yes before rebooting I tried to umount-remount but it seems it has been impossible due to an error... So I decided to restart.

About auto-remounting it will be an amazing feature