@marcusquinn I suppose the other thought with rsync is that the retention period isn't so bad on storage space since its all incremental. Tarballs would be though.
So many variables to think through eh! So a worthy post and conversations to have evolve here for others I'm sure.
@lucidfox Haha, yeah, that happens - but some Apps have file-versioning or Trash features too, some allow users to flag a delete but the DB or file system doesn't delete until a sys admin purges. So you might be taking a sledgehammer to crack a nut.
If there's any apps in particular you think that might save you from backup exponentials and you're not sure, ping their names in here and I might or others might have a quicker answer.
@lucidfox Sounds like a solid strategy to me. Certainly right to have a multi-provider & encrypted setup.
One of the biggest risks I see with anything nowadays isn't technical but Ts & Cs and provider lock-out.
Their platforms, their rules, they can change at any time and you can be an accidental victim of bad actors in ways you'll never imagine before it happens.
From what I understand storage boxes have hardware redundancy, so I don't think that would be a point of failure - most backup recovery needs are user or software caused.
In my experience, no-one ever needs a backup past a week old, what is usually needed is the most recent recoverable.
So I think start with shorter intervals to get it all working confidently, then extend to longer intervals and don't store too old for the sake of it if you don't have some regulatory needs to.
This is my personal Cloudron setup now (rsync encrypted to Wasabi) + the 7 daily provider snapshots:
Happy with this for balancing costs, cruft and security.
Does anyone else get "KeyTooLongError: Your key is too long" errors on using rsync & encryption for backups with EspoCRM?
It happens with the basic installation and nothing changed other than the logo and localisation settings:
(The backups above that are Tarballs but that's impractical for larger servers.)
^This also has additional cover in that when there's App or Cloudron Updates, the App or whole system has a backup triggered before those are installed.
Basically, daily off-site backups are probably over-kill for almost all of us that have VPS backup snapshots enabled too, and for anyone that doesn't, a provider storage box is probable going to be more efficient for daily backups and S3 should really be more for weeklies.
I've had another thought on this, and something I'm thinking will work for our needs:
Hetzner and most others have daily snapshot backups, likely that those have the same resilience as any of their storage boxes, and using that has no CPU cost to your VPS, although in Hetzner's care a 20% extra cost on the VPS, which I feel has value in that there's no reliance on your own software to do this.
Which makes me think than any other off-site backups only need to be Weekly, for whatever retention period your needs require.
It doesn't solve the Wasabi 90-day charges for deleted data issue completely, or the Backblaze ingress costs, which aren't that bad after the first Rsync is complete - but it does reduce the data storage and cycling by a factor of 7, can be scheduled for weekends or days/nights when servers are least used, and still fulfils the provider and geo-replication aims.
@lucidfox Maybe try a Hetzner Storage Box, sounds like that might fit your needs. Encrypted, definitely. Probably Rsync from the sounds of things.
I agree though, it's difficult to know without trying, and I've tried many of the things you have too and still not 100% sure I won't change things again for that perfect setup 
@humptydumpty I think low-to-medium traffic is the key in that since it's latency that has the most difference on perceived speed with web browsing & app interfaces (as opposed to things like downloading/streaming).
updown.io is my go-to for monitoring TTFB times from various locations and the cheapest way I've found of doing that.
I usually target <200ms total TTFB from the nearest location and <1s from the furthest to be considered satisfying for all.
Much of the TTFB effect is from the app itself.
As an example; we have Windows servers on Contabo, accessed via Remote Desktop on, and the screencasting is as responsive and quick as any.
If you wanted higher port speed, you'd still probably find upgrading the package on Contabo cheaper than going to something that has the higher port speeds on the lower tier packages anyway.
For me the decisions are: Contabo when I need the cost-efficient CPU & RAM but Hetzner when I want the VPS to be collaborated on, or potentially moved between accounts without migrating the underlying VPS.
I tried pretty much every other provider over the years, and when you find one or two that have a solution for all you need, perform well, have the lowest prices, and all the features you need to get things done - they become a set & forget utility and I'm grateful not to have to think about moving anything again for the foreseeable.
@girish Sorry, one question as I'd not tested this. Did you try without and with gaga added as a Cloudron User?
As-in, would I solve it by adding all EspoCRM Regular Users as Cloudron Users with permissions for that app?
@girish Ahh, OK, I guess not many people use the LDAP integration as I couldn't see anything on their forum reporting any issue. I'll report on there and see what comes back. Thanks for looking!
@humptydumpty I couldn't find any good reviews on that one, felt like actual too good to be true pricing.
@jsuto Not checked but it's probably something referenced to pull from a CDN than your own server.
Anything with one of these URLs in the code would be what that extension is designed to use a local version of:
ajax.googleapis.com
ajax.aspnetcdn.com
ajax.microsoft.com
cdnjs.cloudflare.com
code.jquery.com
cdn.jsdelivr.net
fonts.googleapis.com
yastatic.net
yandex.st
apps.bdimg.com
libs.baidu.com
cdn.staticfile.org
cdn.bootcss.com
mat1.gtimg.com
lib.sinaapp.com
upcdn.b0.upaiyun.com
stackpath.bootstrapcdn.com
maxcdn.bootstrapcdn.com
netdna.bootstrapcdn.com
use.fontawesome.com
ajax.cloudflare.com
akamai-webcdn.kgstatic.net
gitcdn.github.io
vjs.zencdn.net
cdn.plyr.io
sdn.geekzu.org
ajax.proxy.ustclug.org
unpkg.com
pagecdn.io
cdn.css.net
cdnjs.loli.net
ajax.loli.net
fonts.loli.net
lib.baomitu.com
cdn.bootcdn.net
Not a big deal as anyone using that extension would probabaly know to try again with it bypassed ("HTML Filtering"), just for your awareness really.
I have an odd one:
Setup users just in EspoCRM, "Regular" (as opposed to "Admin").
Not added as Cloudron LDAP Users.
They can't seem to login when set to "Regular",
They can if set to "Admin", and can't again if set back to "Regular".
Might not be a Cloudron integration thing and me just missing something obvious.
Any experience of or fresh eyes on this? 
@jsuto I'm interested if clients are, it's certainly a feature Cloudron doesn't already cover, lends itself better to being an app for multi-provider mail archiving, and is important for business compliance expectations.
Most are on either Microsoft Exchange or Google Vault for these things. Difficult to cost-compare when they obfuscate feature pricing by bundling but It certainly looks like you have a capable alternative.
For interest, I tested the Demo, and have the localcdn.org extension installed by default, so had to enable "HTML Filtering" for the UI to work, maybe something to look at, although I admit I'm a less common setup as most users aren't concerned about CDN privacy.
@humptydumpty Dunno but very happy with the performance, interface and features.
Finland is the closest they offer, and I expect most wouldn't notice perceptible latency.
The real question for me is; how is Digital Ocean that expensive with their economies of scale?
If you want even cheaper and with a US data centre, I've been happy with contabo.com too. Doesn't have all the control panel eye-candy and features but their support is responsive, capable and will go above and beyond the call of duty if its a reasonable request.
@jsuto Nice project and welcome to Cloudron!
I can see value in this, are you packaging it as a Cloudron App or hoping for the community to?
Kinda what Upcloud.com has had for years at 20% less cost and Hetzner.com for almost half that price. that.
Good markup for DO though 
I used it once, nice app store and dashboard.
@privsec Maybe odrive.com will help if you want to multi-cloud for redundancy.
syncthing.net sounds like a decent Cloudron solution too, although not tried it yet.
cyberduck.io or mountainduck.io with backblaze.com/b2 could also be an option.
I like the free tier with sync.com too, trust them more than the bigger names in that space. Decent mobile app too.
Main priority right now though is recovering those photos, feeling your pain on that.
@girish The most interesting thing about this is the way most hosts charge less per CPU/RAM for the smaller VPS tiers to capture people price-comparing.
It might be something that pays for itself, where growing single VPS CPU/RAM can be exponentially costly. 