J
Ah. I see.
My apologies. I am very used to being the same user on both the host and the target system. And, I'm thinking in terms of scp or sftp, not an SSHFS mount. The difference matters a great deal; your answer is clear, and I see why I was confused/wrong.
My fog of confusion wafts away in the light of illumination. 
Thank you.
(I have a suspicion that this is a variation on this post from a while back.)
I have configured backups as follows:
| backup set | encr? | target | day | time | files | size |
|---|---|---|---|---|---|---|
| bitwarden | Y | storage box | daily | 20:00 | 800 | 7MB |
| photos | N | storage box | S | 03:00 | 300K | 200GB |
| photos | N | NAS | Su | 03:00 | 300K | 200GB |
| full (- music, -photos) | Y | NAS | MWF | 03:00 | 18K | 12GB |
| music | N | NAS | T | 03:00 | ? | 600GB |
What I'm finding is that my Immich (photos) instance does not want to backup. To be more precise: Immich consistently fails a long way into the backup. In both the case where it is talking to a storage box (overseas, for me) and to my local NAS, it is configured as an SSHFS mount. In each location I have set up a folder called $HOME/backups, and used a subpath for each backup (e.g. photos, so that the full path becomes $HOME/backups/photos, $HOME/backups/vaults, etc.). In all cases, I'm using rsync with hardlinks.
I removed the photos (which is large/has many files) and the music from the full backup set, because I want to target them separately for backup. And, I want to make sure my full backup completes.
I can backup the bitwarden instance, because it is small. I have not yet seen the photos complete. I end up somewhere around 290K files, and there's an SSH error that drops. I don't know what the root cause is. (And, I'm now waiting for another backup, because Immich kicked off an update... so, I have to wait.)
I'll update this thread if/when it fails again. Possible root causes (that would be difficult for me to work around):
rsync would have no problems.The NAS is a TrueNAS (therefore Debian) machine sitting next to the Cloudron host. Neither seems to be under any kind of RAM pressure that I can see. Neither is doing anything else of substance while the backups are happening.
Unrelated: I do not know what happens when Immich updates, because I am targeting it with two backup points. Does that mean an app update will trigger a backup to both locations? Will it do so sequentially, or simultaneously?
I would like the SSHFS backup to "just work." But, I'm aware of the complexity of the systems involved.
Other solutions I could consider:
rsync with many files, I discovered that (on B2) I could end up paying a lot for transactions if I had a frequent backup, because rsync likes to touch so many things. This was the point of getting the NAS.rsync would think it was a local filesystem. This might only be pushing the problems downwards, though.rsync processes. On the other hand, I have confidence that if I set up a local volume, the Cloudron backup will "just work."Ultimately, I'm trying to figure out how to reliably back things up. I think #4 is my best bet.
Hi @james ,
Fair enough. To be clear:
If you make the mistake of using $HOME for the target directory, then the behavior of allow_other changes the permissions on $HOME to 777. The .ssh directory must exist under a home directory that is either 755, 751, or 750. (It probably can be something else...) Point being, "fool me twice," I have made this mistake on more than one system, and wondered why it is so hard to set up an SSHFS mountpoint. It is because it works once, and then not a second time, because the home directory permissions have changed, "breaking" SSH on the target system.
Perhaps this is clearly described in the SSH mount docs, and I missed it, but it is a silent/invisible source of confusion when setting up SSHFS mountpoints.
(An aside: I still don't know why any user other than the user I assign would need to access the mountpoint: I provide a private key and a username. Only that user should be able to carry out the SSHFS mount, and all of the writes should happen as that user. Why would I ever need some other user to be able to read my backups on a remote system?)
We can re-close this as solved, because I more clearly understand Cloudron's behavior. Because two things can be true, I understand the behavior, and I still think it is incorrect: if I provide a private key and username, that is the user I expect all operations to happen as, and I do not expect permissions to be set so that any user of the remote system can read the files. But, expectations are tantamount to assumptions. 
I'm struggling with this problem as well.
I'm finding when I try and SSHFS with my TrueNAS box...
cloudback/home/pool/dataset/cloudback/home/pool/dataset/cloudback and my prefix to fullCloudron always changes the permissions on the directory /home/pool/dataset/cloudback to 777. This seems... grossly insecure. And, worse, it breaks SSH, because you can't have a filesystem above the .ssh directory with permissions that open.
However, I also find that if I set the path deeper into the account (with no prefix), I avoid the permissions issue, and instead, I get backups that hang/lock, especially on Immich. (That could be unrelated.)
My single biggest question is why is Cloudron setting perms to 777 anywhere?
I'm trying again by creating a directory in the homedir, and using that as my base path. Then, within that, I'm using the "path" option to create subfolders. I don't have a reason I think this might help, but given comments above, I'm trying it. 
I've eyed both
and
https://planka.app/community#strategy
as being potentially friendly to Cloudron packaging. kan.bn in particular just wants a Postgres database, which Cloudron happily provides.
I don't have a Synology, but can you use the Task Manager to run an rsync command periodically to pull copies of your backups from the VPS to your local NAS?
Or, if you're backing up to an object store from the VPS, you could use Cloud Sync to sync the S3-based backups to your local NAS?
These are two ideas that come from some googling. Perhaps they'll inspire additional thoughts.
@timconsidine , I'd like to look at combining your package and mine. Should https://git.cloudron.io/timconsidine/cloudron-garages3-ui be public? It just says Retry later.
This is my new favorite sequence of posts in this forum. Thank you, @robi , for the assist. 
@timconsidine , I'll look at Voltron-ing the two repositories together.
Everything I'm about to say is independent of your actual needs and what you want to achieve. Are you hosting for others? Do you need to be able to restore within hours? Then some of what I say is not for you. If you're instead looking for some "I messed up, everything is gone, and I need a way to recover, even if it takes me a few days," then some of what I say will be more applicable.
I have generally built my backups in tiers, for redundancy and disaster recovery.
rsync or direct mount to your instance's local disk.tar files, or tar.gz if you think the data is compressible. Set up a lifecycle rule to move it quickly (one day?) to Glacier if you're thinking about cost.That's perhaps a lot, but it depends on your data/what you're doing.
You could also go the other way: if you think your cloud backup costs will be too high, you could do the following:
restic or similar will be your friend here.That's the... $800 or so solution, but you would weigh that cost against how much you're going to be paying in cloud storage. (That is, if you decide you're going to be paying $200+/year for backups, perhaps the NAS is going to start to look attractive.) The incremental backups should get smaller once you get the initial pull done (in terms of size to pull down).
A version of the NAS is where you buy one external drive, run your backups locally, and pray the drive doesn't go underneath you, or worse, when you have to do a restore. I would personally chuck a single drive out the window, but some people love to gamble.
Recovery from the offline backup will be annoying/painful. You'd have to upload it, and then configure your restore to point at it. However, it would be your "last ditch" recovery approach. This would be to my opening point: your backups are dictated, in no small part, by your budget and needs.
If you have money to spare, use direct- or SSHFS-mounted disk, and just backup to it. If you are looking for some savings, you can price out S3-based storage (B2 tends to be cheapest, I think, but don't forget to estimate how many operations your backup will need---those API calls can get expensive if you have enough small objects in your backup.) Moving to Glacier is possible if you use AWS, which is significantly cheaper per TB. Having at least one disconnected backup (and a sequence) matters in the event of things like ransomware-style attacks (if that is a threat vector for you). Ultimately, each layer adds cost, complexity, and time to data recovery.
Finally, remember: your backups are only as good as your recovery procedures and testing. If you never test your backups, you might discover you did something wrong all along, and have been wasting time from the beginning. I find Cloudron's backups to be remarkably robust, and was surprised (pleasantly!) by a recent restore. But, if you mangle backups via cron, etc., then you're just spending a lot of money moving zeros and ones around...
Hi @timconsidine ,
Good question. I didn't push further, given that @girish suggested this might be positioned to be an addon.
https://forum.cloudron.io/post/116655
@girish , do you think I should finish this as an app package, or do you think this is something that will land in the roadmap? Or, as we say, "two things can be true?"
@timconsidine , I'm happy to bring this app to completion. Or, perhaps, contribute work to the core. I guess I'd look to the product team to provide some guidance.
Restoring the previous version from backup worked for me, and I then disabled automatic updates.
I just migrated my entire Cloudron instance, and there were no problems.
You can, and should, close this ticket after basking in your kudos. 
Specifically, I'm locally hosting, and was very impressed with how seamless the process was. My backups were via SSHFS mount, so when I brought up my new machine, and uploaded the backup config, it happily mounted the backup and began the restore process.
I used the /etc/hosts trick to point to the new machine on the internal network, and when everything came up, I told my router to point to the new host as opposed to the old.
Absolutely wonderful. Thank you.
YOUR_LOGS_GO_HERE
Not this time!
Just wanted to say how grateful I am for the work that goes into this. Thank you.
I moved from a Dell 7040MFF (bare metal) to a UGREEN NAS 2800.
/homeThis should put all of my Cloudron app data on the ZFS array. I backup over SSH to another machine with a similarly sized ZFS mirror.
9.0.15
24.04.03
Manual on a 24.04 Ubuntu Server VM.
I did some explorations in packaging Garage, and I suspect the need for object storage on Coudron with the fading of Minio is on the core team's radar:
Like timka said, I didn't poke RustFS for reliabiliy/newness reasons, and was reasonably pleased that Garage fit a similar niche/seems to fit the Cloudron model (self-hosting/limited-scale hosting).
@girish Glad to serve.
If the goal is an addon, then...
As to a web interface, I would not recommend you create one for end-users. (I suspect this is not what you mean.) You have NextCloud, XBackBone, and other apps that can talk to an S3-compatible backend for file browsing. What people might need/want is a way to:
I suspect you could iterate towards this, if you wanted to. Release it with terminal-only management to start, and work towards an admin interface for common bucket creation/removal-type tasks.
There are things that Garage does not do (e.g. lifecycle rules), so this is not a "full" S3 clone (the way Minio aspired to be). In this regard, SeaweedFS might offer more features (and a built-in admin UI)... so, it might be worth doing a spike to explore it as well. At a glance, it is similar, but it also is intended to scale (e.g. it does Raft consensus and Reed-Solomon EC if you want it) in a way that Garage does not. This might also be a reason to not use Seaweed.
I can poke it with a stick if it would be valuable.
Update, a few hours later: I would recommend sticking with Garage.
Hi @scooke ,
To your question, I followed the instructions on the Garage website, and I added a near-copy of that documentation to the README in the package I've developed. This is all documented in the git repository for the package that I linked to. I have Garage running on my Cloudron instance, and was able to create buckets, add content, and even serve static web content from within those buckets. I'm sorry your prior experience with this did not work for you.
I am now trying to do that in a way that it would be considered for inclusion as a standard package, if the Cloudron team thinks it is worthwhile. If they don't, perhaps I'll do it for myself.
Not much I can do with that statement. I have it packaged and working. I'm now in the weeds of how best to handle data backup and restore on Cloudron. Given that the Minio package must go away, this is at least a possibility that can be evaluated.
Hm. Maybe I am wrong about the URL rewrites. I'll sleep on it.
Because Garage puts the different APIs/functions on different ports... you might be right, @robi.
I still think it would be nice to have aliases exposed in the manifest.