Backup failing (flaky CIFS mount)

[robi]

we need better retries for mounts and DB connections during long running processes such as backups.

[p44]

@jdaviescoates I had a similar problem. I don't know if @nebulon reached Hetzner provider to ask more info's about long-time mounting of Box service.

[iqweb]

same here on a Hetzner storage box 2 days back - couldn't manually remount either. Had to reboot & then remounted automatically as per /etc/fstab.

[girish]

@jdaviescoates Did this get sorted out?

[jdaviescoates]

@girish yes:

@jdaviescoates said in Backup failing:

Backing up post-reboot worked fine. So my guess is that my Hetzner Storage Box had a wobble and somehow became unmounted mid backup.

Sounds like others have been experiencing similar too, so I'm guessing a (hopefully temporary) issue with Hetzner storage boxes.

Although like @robi said it'd be cool if there was some way Cloudron backups could try to survive such wobbles.

[girish]

@jdaviescoates Yes, at some point we want to make it possible to add mounts easily. When we do that, we can attempt to re-mount automatically, I guess.

[robi]

@girish not sure this is the right context..

this is about existing mounts and their resilience to disconnects.

• monitoring mounts for health
• allowing for self-healing if it exists from the protocol
• if not, try harder, remount command for example
• if still not, try unmount, then remount
• if still not, notify to reboot

[p44]

@robi right suggestions.

@girish I had this problem again: all backup folder this morning where unmounted...

[iqweb]

@p44 said in Backup failing:

@robi right suggestions.

@girish I had this problem again: all backup folder this morning where unmounted...

I've had the same thing again too

[girish]

@iqweb Are you also using CIFS/NFS?

[p44]

@iqweb Still on Hetzner storage?

[iqweb]

@girish said in Backup failing:

@iqweb Are you also using CIFS/NFS?

Yes - storage provider: CIFS Mount

@p44 said in Backup failing:

@iqweb Still on Hetzner storage?

Yes - Hetzner storage box

& again - I had to reboot to be able to remount.

[p44]

@iqweb Maybe Hetzner sometimes reboot Box instance? From my side is about one week CIFS mount is stable...

[nebulon]

I also have to remount my cifs share about once a month. In my case I thought it was related to my residential home internet connection. Either way I think we have to handle such mount points better and maybe even attempt a remount automatically.

[p44]

@nebulon Good idea to add a health-check to keep alive mounting

[girish]

@nebulon Does cifs mounting have some retry flags? (i.e the fstab options)

[nebulon]

@girish not that I have found and https://linux.die.net/man/8/mount.cifs does not mention anything.

Apparently systemd has some automounting feature https://www.freedesktop.org/software/systemd/man/systemd.automount.html which mounts the drive once the mountpoint is accessed by a process! Can't tell if it would reconnect such a mount though.

[girish]

@nebulon Maybe https://askubuntu.com/questions/1210867/remount-cifs-on-network-reconnect ? (just expands on your link)

[imc67]

@girish & @nebulon any news on this issue?

I started yesterday to backup via cifs to a Hetzner storagebox (2 backups a day), today the first backup failed due to

Command failed with exit code 1 (EPERM): df -kPT /backups_cifs

I solved it with a manual:

sudo umount /backups_cifs
sudo mount -a

But do I need to script this with a daily cron or is Cloudron able to recover from this somehow?

[p44]

@imc67 Dear, this happened because Hetzner did some maintenance on their Box Service.

Also a Cloudron restart solved problem.