Backup issue in rsync non-encrypted filesystem type. Never moves past status of "Removing directory" for one particular app.

[girish]

This seems to be some filesystem related issue and not related to Cloudron. Specifically, we need to figure out why this fails:

rm -rf /cloudron-backups/snapshot/app_a4c34340-d201-4a5e-8d78-d043d717dcfd
rm: cannot remove '/cloudron-backups/snapshot/app_a4c34340-d201-4a5e-8d78-d043d717dcfd/data/_data_/_default_/storage/cfg': Directory not empty

If you cannot remove from the shell, cloudron cannot remove it either. What is in that directory? Is it empty or some NFS/CIFS bug?

[d19dotca]

@nebulon I was just able to test the Rainloop app backup on a local /tmp directory and yes it worked fine. So it's a combination of that app and CIFS it seems.

[d19dotca]

@girish It's possible, however I think it's odd that this works fine for every other app but this one, so I'd be hesitant to say it's only a CIFS issue. I was able to run a test a moment ago and can verify it works fine on local disk, but why would this one app have a problem on a CIFS-mounted disk and none of the other apps? Seems strange, no?

Btw, if Cloudron can't complete an operation like removing a directory, I'd suggest it fail fast. I left it for hours before thinking it might finally do it and it never did. I'm not sure how long that backup would have kept running for.

[d19dotca]

@girish Here is what's in that directory:

ls -alh /cloudron-backups/snapshot/app_a4c34340-d201-4a5e-8d78-d043d717dcfd/data/data/default/storage/cfg/

total 0
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 .
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 ..
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 al
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 au
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 br
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 cl
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 do
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 du
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 he
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 j.
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 mi
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 ne
drwxr-xr-x 2 yellowtent yellowtent 0 Apr 1 16:39 s.

If I keep going into it, it seems to be the different usernames, as the folders then become <username>@<domain>, and then it becomes the aliases setup in Rainloop, by the looks of it.

[girish]

@d19dotca apps do not touch the backups directory. so it's related to CIFS for sure.

Is this CIFS/NFS on your home server btw? Is it an option to run minio? That might be more reliable than all these file permission problems

[d19dotca]

@girish No, these are all hosted in a dedicated backup partition hosted at OVH in Canada. Running minio I don't think is an option although I'll look into it. Ultimately I need to keep all data in Canada, which is one of the reasons I'm now trying to use the filesystem backup method to keep it in Canada since the other API-related options to my knowledge do not host in Canada for their data (I know DigitalOcean for sure doesn't anyways, and AWS does I believe but it's expensive).

[girish]

@d19dotca Have you seen this note about CIFS mount options - https://cloudron.io/documentation/backups/#sambacifs ?

In your directory listing, can files be individually rmed?

[d19dotca]

@girish Yes, they can. And that's the thing, every other app can remove files and directories without issue. That's why I'm not totally convinced this is only a CIFS issue, it seems more like a combination of that app data and CIFS perhaps. Of course you could be right, it may be entirely CIFS, but then I can't understand why this issue only occurs with one app alone and all others work (well all others up until this one anyways, it's about the 7th app backed up, so at least 6 others work fine).

[d19dotca]

Side note: I just learned that OVH seems to have an S3-compatible storage solution, so if I can make sure they store that in Canada then I may try that S3 API option in the Cloudron backups, maybe that'll be better. Doesn't solve the problem but at least gets me a working backup in Canada. Also seems cheaper than what I'm currently doing.

[nebulon]

Maybe one specific file name/path which just happens to be produced by rainloop causes CIFS to fail?