Cloudron Forum

@girish I have reports that this just happened again.

@girish Sounds good, thanks.

@girish Hi. I believe I am on: v6.3.5 (Ubuntu 20.04.2 LTS). I actually got it working now using RSYNC instead of a Tarball. However, using RSYNC takes a long time so my question now is why the Tarball method doesn't work - there may be no temp folder to keep the zipped file I suspect, or something like that. My next question is what is the restore process - but I am not there yet. Thanks.

@robi The fsmetadata.json is only used in rsync backups. It's used to keep track of empty directories and 'x' bit of files when using rsync mode. We do this because object storage services (like s3, gcs and friends) cannot track this since they are not a filesystem but an object store. This is not needed in tgz because tgz format can encode this information inside the tarball. In essence, fsmetadata.json is usually owned by root. If it's owned by yellowtent, it's probably because maybe you restored the Cloudron in the past or something. The restore logic changes the permission of all files to yellowtent after downloading so that restore can continue as non-root user.

Thanks, fixed.

borg2 is in the works which reorganizes the storage backend to support any kind of remote storage, recently including everything supported by rclone.

@girish In my case I was trying to mirror the content of a remote DO object storage to a local Minio instance. I can't recall the exact error but I remember it had to do with the handling of filenames.

@mehdi Very much interested in this topic. I love my American friends, but the EU certainly isn't without capability to respond to the data-mining free-for-all antics of the last decade. I share occasional posts on Twitter on these subjects if anyone's interested.

@girish said in Add delete backup Action in Backup List: @robi i think users expect the box backups (the ones listed in the backups view) to be complete backups. If one of the apps is missing, it would be unexpected. Yes, unless one goes and edits them. Why would that be an expectation problem? If I remove an app from a backup that means I don't want to restore it either. Of course, alternately, we have to set expectations... When you restore from such backups, the apps will start our empty or something. Uhh, why not leave it out completely? Just skip that app from the restore, it's been removed, right? The backup, after the app removal, should decrement the app count, right? Neat and tidy.

Exposing this in the backup UI is the idea.. not fishing in the logs.

@marcusquinn Yeah, I have another idea once differential backups become available.. Have the template App be backed up locally, while others remotely, keep things fast. Let's see which we get first.

@thibaud I replied to you on support@ but the issue is that there is some long file name. The current rsync+encryption backup has some file name length limitation - https://docs.cloudron.io/backups/#encryption . There is a feature request at https://forum.cloudron.io/topic/3057/consider-improvements-to-the-backup-experience-to-support-long-filenames-directory-names . Run the following command in /home/yellowtent/appsdata to find the large filenames: find . -type f -printf "%f\n" | awk '{ print length(), $0 | "sort -rn" }' | less

Just to follow up, here's a sample of normal backups followed by a Cloudron upgrade, which itself triggered another backup run, and the corresponding relevant network and disk graphs: [image: 1607915273220-network-traffic-resized.png] [image: 1607915283316-disk-i_o-resized.png] All in all, it's definitely fast-er but not insanely performant. CPU utilization vs load hints that it may in fact be down to inefficient utilization of cores to some extent, but there is definitely a fair bit more bottleneck coming from the network still. [image: 1607915562708-cpu-utilization-resized.png] [image: 1607915565769-cpu-load-resized.png] Nothing earth-shattering either way, and gains were more mild than I would have guessed, but all in all, not a bad outcome.

glad to have helped

@girish Oh, I had the URL copied in the paste buffer but got distracted. It's much simpler - https://odrive.com I've updated the post with the link.

Good points @mehdi @CarbonBee See if you can find another project that already integrates with Glacier and how they handle it. If it's OSS, the code will be reusable, easing integration for devs.

@ei8fdb said in Why is my backup drive full?: I've moved the backup directories from /var/backups/ to my external drive. If I want to restore from those backup directories, do I need the contents of the /var/backups/snapshot directory also? The snapshot directory is not required for restoring. BUT it's required for the actual the backups to work (so think of it as a working directory for backups). It's important to not remove that directory when doing backups! Since, Cloudron will use hard links from the actual backups to the snapshot directory, it's not really taking up extra space.

Thank you for the quick reply. But even in my testing environment, telling me my backup method is unsafe once a day is just a hassle to weed them out of the Cloudron update (valuable) notifications. I bet I can just trick Ubuntu into seeing the path as a volume to get rid of the notification. But for normal Cloudron users, I bet an option to acknowledge and disable the “backup is unsafe notification” would be valuable for them especially in their first non-production use of Cloudron (where I’m personally at for now).

Those sync.cache files are state files of the rsync backup uploader. That has to run as root to be able to list and read the app's files, the app file permissions here depend on the app itself and how it manages permissions. The yellowtent user as such does not have root rights on the system and uses helper scripts in such cases. The list of scripts which need and thus run with elevated permissions can be found at https://git.cloudron.io/cloudron/box/-/blob/master/setup/start/sudoers

I can confirm that, now that the lifecycle settings are correct, backups get physically deleted from B2.