Backups times out at "Removing directory /mnt/cloudronbackup/snapshot/mail (mail)"

[joseph]

Have you tried that command manually? Looks like it hangs. Maybe some mount point issue?

[ekevu123]

No, I wanted to get confirmation first that I am supposed to do that. I will do that then and try to restart the backup.

[joseph]

Yes, you can do that . The backup task will be able to handle it.

[ekevu123]

I tried running it via terminal, but I had the same thing - it hangs.

Also, today, the backup timed out saying "packages out of order".

[nebulon]

so rm -rf ... hangs? If so then something is off with that filesystem/disk. Check any system logs for hints, but probably not related to Cloudron itself then.

[ekevu123]

Is there a good way to check backup integrity - to understand what has been backed up and what hasn't? Then I can monitor the backup process better, I think.

[joseph]

You have to list the files in the backups and compare it with the listing in appsdata directory , I guess.

[ekevu123]

Unfortunately, I am still having problems with this - every single time, to be precise.

I went through the logs of the last backup attempts and looked for the last entries each time:

Apr 08 03:59:59 box:backupformat/rsync Adding vmail/alias@domain.com/mail/.Archive/cur/1640450876.M588430P358.cc2ac4cc0c67,S=14771,W=15404:2,S position 323753 try 1

Apr 06 19:51:40 box:backupformat/rsync Adding vmail/alias@domain.com/mail/.Sent/cur/1674390327.M835576P19744.e4334a49cfa6,S=1973,W=2032:2,S position 431623 try 1

Apr 05 09:13:02 box:shell filesystem: rm -rf /mnt/cloudronbackup/snapshot/mail

Apr 05 03:59:58 box:backupformat/rsync Adding vmail/alias@domain.com/mail/.Archive/cur/1742580897.M74343P843.46f5cae5622d,S=19005,W=19405:2,S position 374316 try 1

Apr 03 21:29:13 box:backupformat/rsync Adding vmail/alias@domain.com/mail/.Archive/cur/1742571065.M344721P843.46f5cae5622d,S=19099,W=19494:2,S position 340523 try 1

Apr 02 17:42:33 box:backupformat/rsync Adding vmail/alias@domain.com/mail/.Archive/cur/1742580774.M330309P843.46f5cae5622d,S=5354,W=5484:2,S position 373873 try 1

Apr 01 08:14:45 box:shell filesystem: rm -rf /mnt/cloudronbackup/snapshot/mail

Apr 01 07:27:11 box:backupformat/rsync Adding vmail/firstname@domain.com/mail/.Virtual.All Mail/cur/1742560747.M843702P858.46f5cae5622d,S=10928,W=11122:2,Sa position 465347 try 1

The error is always "Task 10224 timed out" (with varying numbers)

Apps are being backed up normally.

[joseph]

IIUC, you are using CIFS/NFS or some mount point for the backups . This mount is not working reliably, since you are unable to delete from the shell either. If so, this is a kernel/network/VPS level issue since Cloudron is not involved at that level . Is it an option to ask the support of the VPS provider or the mount storage provider?

[ekevu123]

Are we sure we are talking baout the same thing? I am using Hetzner Storage Box via SSHFS for backups. This would explain the email issue. But /mnt/cloudronbackup should be on the disk, right?

[BrutalBirdie]

@ekevu123
Are you using the main account of that storage-box or a sub-account?

@ekevu123 said in Backups times out at "Removing directory /mnt/cloudronbackup/snapshot/mail (mail)":

But /mnt/cloudronbackup should be on the disk, right?

Sorry but there are multiple ways to interprete your question.

• /mnt/cloudronbackup is a directory path on your root server where you can ssh into
• /mnt/cloudronbackup can be local folder if nothing is mounted there
• /mnt/cloudronbackup can be a remote folder if something is mounted there

Such errors with the Hetzner-Storage box can be many things.
Always make sure there is no current maintence done on your Storage-Box, see https://status.hetzner.com/

When using SSHFS you are subjected to the fail2ban of hetzner which is not documented from hetzner anywere, but it exists
This would cut / disrupt the whole connection to the Storage-Box and would only last 15 to 30 Minutes if no more failing requests* are made.
*failing requests? e.g. attempts to mount with wrong password or ssh-key

In your case, please try to mount your Hetnzer Storage-Box locally with the following command:

sshfs -p 23 ACCOUNTID@ACCOUNTID.your-storagebox.de:/home $TARGETMOUNT

Then you can cd $TARGETMOUNT and verify if you can delete stuff locally.
If that works please report back.

---

ps: @ekevu123 no where before have you mentioned that you are using the Hetzner Storage-Box product. @joseph had to assume it is some generic provider.
For the future, try to describe your setup as precise as possible. This way you get better and faster answers.

[ekevu123]

1. I did what you suggested and I can create and delete files using sshfs.
2. I am not using a sub-sccount.
3. I'd like to add that the setup has worked at some point, I have two successful backups in the history, but then it stopped working. Also, apps back up just fine, but it fails during e-mails at varying positions.
4. The issue happens every day now with no exception, so this is not a maintenance issue.

EDIT: When I look at the folders in the Storage Box, I have backup folders since 02nd March, although my backup policy is 7 days. Could this simply be too much for Cloudron to handle? I could perhaps delete these manually.

I could also get a fresh storage box and attempt to perform a new backup there and see if the issue persists.

[BrutalBirdie]

@ekevu123 said in Backups times out at "Removing directory /mnt/cloudronbackup/snapshot/mail (mail)":

but it fails during e-mails at varying positions

Do you have some crazy mail folder structure?
Since rsync is limited to 255 char length for the path.
With deep nested folders and long names this could get triggered.
But would show up in the logs

Can you try with tgz instead of rsync?

[girish]

@ekevu123 in the past, we have had issues with sshfs where many fuse operations result in fuse misbehaving. in an earlier release, we added a workaround to do a recursive copy directly via ssh instead of going through fuse .

Recursive remove probably suffers the same fate. I have now implemented the workaround for that as well - https://git.cloudron.io/platform/box/-/commit/65f066d391c498e07d7dd1fde0766b8bae7fa119 . Feel free to apply the patch into /home/yellowtent/box/src/storage/filesystem.js

[ekevu123]

Creating a new box hasn't helped. The issue persists.

Regarding the patch, it wouldn't play a role anymore as now nothing has to be removed. The timeout happens when adding emails.

[ekevu123]

Using tgz instead of rsync did let the backup run normally, at least once right now. Let's call this a workaround for now, I'll see if this works regularly.