Backup issue in rsync non-encrypted filesystem type. Never moves past status of "Removing directory" for one particular app.

[d19dotca]

Hello,

Sorry for all the backup-related posts lately. Moved to a new infrastructure and as such have been tinkering with the backup configuration for the new environment. Unfortunately, been running into a lot of issues.

1. NFS volume mounted, but get "chown"-related error on backup task

2. Backup using rsync encrypted gives "ENAMETOOLONG" error

And now this third issue. It's been a difficult couple of days for backup configurations and tests in my environment.

When I try using non-encrypted rsync filesystem backup, it continually gets stuck at this task:

Removing directory /cloudron-backups/snapshot/app_a4c34340-d201-4a5e-8d78-d043d717dcfd

When I go to manually try removing the directory, I get the following error in the command line, which I assume is what Cloudron is getting stuck on as well:

rm -rf /cloudron-backups/snapshot/app_a4c34340-d201-4a5e-8d78-d043d717dcfd
rm: cannot remove '/cloudron-backups/snapshot/app_a4c34340-d201-4a5e-8d78-d043d717dcfd/data/_data_/_default_/storage/cfg': Directory not empty

The app, in case it matters, is the webmail Rainloop app. Any ideas why this is happening? What could cause this issue? And what workarounds may exist? Is this a defect in Cloudron?

It's able to remove directories for every other app, but it always gets stuck on this one about half way through the backup process.

For context on all these backup issues in case curious...

I never had issues before because it was using DigitalOcean Spaces, but they do not have a Canada location for the data and I need to now have my backups also in Canada aside from my server (which was always in Canada). So that plus using AMS3 for the region in DO plus DO's rate limiting led me to some bad performance issues with backups. So in the new environment and with the new restriction to needing to keep backups in Canada too, I am now using filesystem backups to a dedicated disk for backups, and thus the change from DO Spaces to Filesystem type. Ideally I'd like to use the encrypted rsync, but that's where I get the second issue above with ENAMETOOLONG. And if I use non-encrypted rsync, I get the error in this post. The only one that seems to work for now is encrypted tgz, but that is less than ideal given the size that needs to backed up every time, and there's roughly 40 GB of data it has to backup, so doing that twice a day is 80 GB which will rollover my backup space quite quickly. My understanding is rsync is much better for this so that my large files (some video files for a customer website for example) are only really backed up once and every other backup uses a hardlink to it so it conserves space. Thus my expectation to use encrypted rsync (or even non-encrypted rsync at the very least). Unfortunately this has been a difficult process for some reason. Not sure if Cloudron or environmental issue or a mix, but it's eroding my confidence in having safe backups right now and so I could use some help if possible, please.

[nebulon]

Not sure where exactly to respond with the various topics open. However guessing from the others, is it correct that this folder is cifs mounted? Both the ENAMETOOLONG and possibly the directory removal might be related to the filesystem type. Can you testwise do a backup of that app to the local filesystem to see if the issues persist?

[d19dotca]

This post is deleted!

[d19dotca]

@nebulon Yes, it is CIFS. Per your response in the other thread, you had stated you test with CIFS, not really as much with NFS so you recommended CIFS over NFS. I mounted it with CIFS and now have these issues. I'm not sure I have enough disk space locally to actually do that, it'd fill it up pretty quickly. Although if I can get away with only doing that one app then that would definitely be fine. I'll try that now and see what happens.

[girish]

This seems to be some filesystem related issue and not related to Cloudron. Specifically, we need to figure out why this fails:

rm -rf /cloudron-backups/snapshot/app_a4c34340-d201-4a5e-8d78-d043d717dcfd
rm: cannot remove '/cloudron-backups/snapshot/app_a4c34340-d201-4a5e-8d78-d043d717dcfd/data/_data_/_default_/storage/cfg': Directory not empty

If you cannot remove from the shell, cloudron cannot remove it either. What is in that directory? Is it empty or some NFS/CIFS bug?

[d19dotca]

@nebulon I was just able to test the Rainloop app backup on a local /tmp directory and yes it worked fine. So it's a combination of that app and CIFS it seems.

[d19dotca]

@girish It's possible, however I think it's odd that this works fine for every other app but this one, so I'd be hesitant to say it's only a CIFS issue. I was able to run a test a moment ago and can verify it works fine on local disk, but why would this one app have a problem on a CIFS-mounted disk and none of the other apps? Seems strange, no?

Btw, if Cloudron can't complete an operation like removing a directory, I'd suggest it fail fast. I left it for hours before thinking it might finally do it and it never did. I'm not sure how long that backup would have kept running for.

[d19dotca]

@girish Here is what's in that directory:

ls -alh /cloudron-backups/snapshot/app_a4c34340-d201-4a5e-8d78-d043d717dcfd/data/data/default/storage/cfg/

total 0
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 .
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 ..
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 al
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 au
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 br
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 cl
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 do
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 du
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 he
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 j.
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 mi
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 ne
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 s.

If I keep going into it, it seems to be the different usernames, as the folders then become <username>@<domain>, and then it becomes the aliases setup in Rainloop, by the looks of it.

[girish]

@d19dotca apps do not touch the backups directory. so it's related to CIFS for sure.

Is this CIFS/NFS on your home server btw? Is it an option to run minio? That might be more reliable than all these file permission problems

[d19dotca]

@girish No, these are all hosted in a dedicated backup partition hosted at OVH in Canada. Running minio I don't think is an option although I'll look into it. Ultimately I need to keep all data in Canada, which is one of the reasons I'm now trying to use the filesystem backup method to keep it in Canada since the other API-related options to my knowledge do not host in Canada for their data (I know DigitalOcean for sure doesn't anyways, and AWS does I believe but it's expensive).

[girish]

@d19dotca Have you seen this note about CIFS mount options - https://cloudron.io/documentation/backups/#sambacifs ?

In your directory listing, can files be individually rmed?

[d19dotca]

@girish Yes, they can. And that's the thing, every other app can remove files and directories without issue. That's why I'm not totally convinced this is only a CIFS issue, it seems more like a combination of that app data and CIFS perhaps. Of course you could be right, it may be entirely CIFS, but then I can't understand why this issue only occurs with one app alone and all others work (well all others up until this one anyways, it's about the 7th app backed up, so at least 6 others work fine).

[d19dotca]

Side note: I just learned that OVH seems to have an S3-compatible storage solution, so if I can make sure they store that in Canada then I may try that S3 API option in the Cloudron backups, maybe that'll be better. Doesn't solve the problem but at least gets me a working backup in Canada. Also seems cheaper than what I'm currently doing.

[nebulon]

Maybe one specific file name/path which just happens to be produced by rainloop causes CIFS to fail?