Thank you for the quick reply. But even in my testing environment, telling me my backup method is unsafe once a day is just a hassle to weed them out of the Cloudron update (valuable) notifications. I bet I can just trick Ubuntu into seeing the path as a volume to get rid of the notification. But for normal Cloudron users, I bet an option to acknowledge and disable the “backup is unsafe notification” would be valuable for them especially in their first non-production use of Cloudron (where I’m personally at for now).

Those sync.cache files are state files of the rsync backup uploader. That has to run as root to be able to list and read the app's files, the app file permissions here depend on the app itself and how it manages permissions. The yellowtent user as such does not have root rights on the system and uses helper scripts in such cases. The list of scripts which need and thus run with elevated permissions can be found at https://git.cloudron.io/cloudron/box/-/blob/master/setup/start/sudoers

I can confirm that, now that the lifecycle settings are correct, backups get physically deleted from B2.

Reading documentation is sysadmins nightmare Thank you for the clarification. That makes sense and it looks like a clever solution. I guess that in case of a Cloudron trivia night, the question "How many backups do you expect in your storage location?" will be answered in many different ways according to the "important rules" from the documentation

ok. I gave mysql more RAM. Since then the backups have been running smoothly.

If someone knows how other tools handle it would be great. I know rclone has similar restrictions as us - https://rclone.org/crypt/#name-encryption

@girish Great! Thanks again for your help debugging this and adding more configuration. Huge help for larger backups like mine.

For the last release, I did some basic performance tests to get the backup speed to be at par with something like rclone. I did not test sshfs specifically but I am curious how long it takes to copy this amount of data using normal tool in your setup? My guess is that the problem is inherent to sshfs, and is not something specific to Cloudron (sshfs not being fast is pretty widely known/recognized). The only real reason I am reporting this is to perhaps suggest noting that it does not scale well in the docs, or something like that, because I don't think this gives a particularly good impression as-is. sshfs is very sensitive to latency, so any cases where you run it to a remote server on a different network, the slowdown will be very noticable. Using a tool more suited to this style of operation (like, say, rsync) will give better results, but it will still not scale well for large files (or large datasets, over say a few hundred GB of small files). In the other thread, I saw that you were using encrypted backups. Is this the case when you tested this? Encryption requires a decent CPU as well, so a simple cp is not a fair comparison. I wonder if you can enable encryption on the remote disk somehow and not rely on Cloudron encryption? Correct, encryption was enabled, but it wasn't really a factor. CPU was more or less idle while this was going on. Remote disk encryption isn't really an option in this case because I don't have physical control over the remote host, so I want to keep the data hidden

Ok this is fixed, however requires a new Cloudron release, since the tasks API needs a proper pending state.

@d19dotca I have added some docs now. https://cloudron.io/documentation/backups/#schedule has info on the timeout and nice. https://cloudron.io/documentation/backups/#concurrency-settings on the concurrency settings.

Did you try to remount that volume and retrigger the backup? Otherwise do you know if it fails consistently at the same file and also are you using backup encryption?

To give some update on the tech side (if you are interested in this) about the speed issues. Cloudron code run is a very constrained memory environment - 400MB. Even if your server has 10GB RAM, this limit is hardcoded. This limit covers the full platform code (i.e the server side for the dashboard) and also all the tasks (backups, certs, app installs), cron jobs. What I found with some profiling is that if I can make things faster, if we run these tasks with more memory. So, I am reworking the task code in 5.5 so that it runs in it's own cgroup. Currently, this is the only change pending for 5.5. Hopefully, it gets done this week. Apart from that, what I am seeing your proposal is : per-app backup schedule (doable) parallel backups (also doable and is only possible after the above change in 5.5 since running parallel tasks in 400MB won't work) split the file backups and db. I understand the problem you are facing where this one big app is "blocking" every other app from backing up properly. But I don't think this can be fixed by not backing up the files to speed things up since for most apps (except many emby/jellyfin), the files are almost like an extension of db. If you snapshot them at various points in time and stitch them together, it will break... Let's think about other ideas here.

@girish Just provided access and sent the email.

@zjuhasz Guessing that all the S£ compatible providers may all have the same limits. We have 120TB on Wasabi with other systems and always been happy with Backblaze B2 too, also S3 compatible and Cloudron integrated and tested now (I only tested smaller though) if that helps.

@marcusquinn said in Cloudron Backups to GitLab/GitHub Private Repos: Not a high priority - but might be nice for both the really tight and multi-location/provider redundancy aims. I have to think through the rest but multi-location backups is in our radar. We have a long pending issue about this https://git.cloudron.io/cloudron/box/-/issues/528

Thanks @nebulon, my bad! #usererror I omitted part of the path to the backup location - d'oh!

@nebulon Thanks, this was the reason for my problem. After I uncheck "mounted filesystem" Cloudron was updated to 5.4.1

@nebulon That did the trick, thanks a lot again.

Thank you everyone for the clear explanations as always.

@ultraviolet Glad I could help...