Update: The backup was successful just 55 minutes ago, without me needing to do anything this time. Still very interested in getting this resolved though as I'm sure it'll come back at some point again since this is several times this has happened this year now.
@girish Is there anything I need to be doing to help you guys out before I need to erase the backups again and start fresh? It's failed a second time now so I suspect this will always be like this until I force clear it again but I don't want to take away anything that'd help you narrow down what the issue is either.
It's happening again, I'm afraid. Just got notified a short bit ago of this.
Aug 06 13:46:42 box:shell copy (stdout): /bin/cp: cannot create hard link '/cloudron-backups/2020-08-06-204501-046/box_2020-08-06-204642-043_v5.4.1/mail/vmail/user@example.com/mail/.A - PC + FD Referrals INBOX/cur/1596744663.M967808P27952.6cc2d7e3254e,S=2825,W=2866:2,S' to '/cloudron-backups/snapshot/box/mail/vmail/user@example.com/mail/.A - PC + FD Referrals INBOX/cur/1596744663.M967808P27952.6cc2d7e3254e,S=2825,W=2866:2,S': Operation not permitted
Aug 06 13:46:50 box:shell copy code: 1, signal: null
If you think it's helpful to remote in or anything, just let me know and I can enable that option for you.
@simon Have you tried with rsync and hardlinks enabled? Hardlinks preserves space and maybe it's just in my head but I swear it speeds things up a bit too. This will all depend a bit though too on how fast your external disk is accessible.
I have about 29 apps running right now, and my backups are very quick (after the initial backup anyways) using rsync and hardlinks to an ext4 hard disk that's mounted directly (as opposed to CIFS or NFS, etc.). Of course, I don't have 200 GB of data either, so maybe you've got way more data to backup so the number of apps may not matter in that case, but I can say that for me, running a server with 29 apps and using roughly 35 GB of data on the primary hard disk, it backups very quickly. I'd estimate my backups generally take about 2 minutes or less, depending on if there's a WordPress app package update as that means way more gets backed up after those are updated.
If I was to extrapolate from my usage, let's say I had 200 GB of data which is about 5 times what I consume now, then it'd still potentially be just about ten minutes for the backup to complete (assuming it's a linear trend) which seems quite speedy to me still.
How long is it taking the backups to complete for your system? If it's a lot longer than ten minutes, then maybe there's performance issues on the disk you're backing up to or you're not using rsync with hardlinks?
I believe some of this is already planned for 5.5 or 6.0. See this announcement here: https://forum.cloudron.io/topic/2918/what-s-coming-in-5-5 and you'll see a note that reads "Backup upload/download speed - Currently, backups can be quite slow but we have some ideas to speed it up" -- This only addresses part of what you're reporting, but I think some of what you're asking for is already in the pipeline.
A second alternative to KittySplit is an app called ihatemoney.
GitHub: https://github.com/spiral-project/ihatemoney
Website: https://ihatemoney.org/
(Recently I had to use KittySplit, and it got me thinking what self-hosted alternatives exist. The two above seem the best.)
Freecount is a self-hosted app that replaces apps such as KittySplit.
Freecount's slogan is "Your free shared expenses tracking solution".
GitHub: https://github.com/ssimono/freecount
Website: https://freecount.s10a.dev/
@girish Not to sidetrack but this is probably another great example of where the config should be exposed in the Cloudron UI (and of course persisted), related to many of the email improvements many of us are waiting to see in the near future.
As I have a few mailboxes growing quickly for their business, while I have plenty of space left on my main SSD, I'd like to be able to dedicate (and move mail) storage to an external mounted disk. From what I can tell, this isn't possible to do yet in Cloudron (but please correct me if I've missed it).
It seems like a natural extension of the already-existing app storage in custom directories feature, so I naturally assumed I could do this already for mail but it doesn't appear to be the case. I would love to see the ability to move and store mailbox data on an external disk in the near future. 
@alejandrolengua This documentation page is mostly geared towards setting up with different DNS providers, but this is a good place to start too as you'll need this to add new domains to your Cloudron server: https://cloudron.io/documentation/domains/
@girish I used rspamd a year or two ago as part of hardware/mailserver project on GitHub, it was great. With that said, I had not specifically installed rspamd since it came with that project (own container), but it pretty much just worked well out of the box, didn't have to do too much tinkering with it. It gave great reports and such too which I really liked, seeing how much of my email was deliverable, how much was flagged as spam (incoming) vs legitimate, greylisted, etc.
@MooCloud_Matt I think I may see now where some of the confusion is coming from now. What you're talking about is effectively an entirely new functionality (integrating Cloudron with third-party spam services), and what most of the rest of us are talking about in this thread is simply asking for Cloudron to expose existing functionality to Cloudron admins so we can improve how our mail servers act for our users.
The title of this thread is very generic but I think it might be helpful if you started a dedicated thread specific to requesting the addition of third-party spam services and having Cloudron integrate with those APIs, because I think that's a very large topic all on it's own that may require a lot of discussion. Just my two cents anyways.
I'm certainly in favour of your suggestion for a third-party integration for spam filtering - the more options we have the better (as long as it's still maintainable for the Cloudron team) - although I respectively disagree with the implication that a third-party service is the only solution to improving spam filtering "effectiveness" in Cloudron. Plenty of large corporations (receiving far more mail than any of us Cloudron users likely are) deal with spam in-house, though admittedly they will also have far more resources to throw at it.
There is still a lot more we can do within the confines of what Cloudron already does to improve spam filtering effectiveness, we just need those functions exposed to us to better modify them to our liking.
@girish Definitely in agreement with that. I assume you mean "should not be" rather than "should be" though. 
haha But yes, exposing what's already working on the lower levels is mostly what people are asking for and that's what I'd like to see happen and I believe most users would say the same thing.
And that's understandable to be hesitant for now on opening up the SA config if you're pondering moving to rspamd down the road, I think most of us would be okay waiting a bit longer for that decision to be made and then investing the time on the solution identified.
@MooCloud_Matt "optimization" and "management of resources" are important factors in anything for sure, but most of what is being asked for is just exposing already functioning components, so naturally those shouldn't really take any more resources. It isn't so much adding entirely new features that we're asking for (although perhaps a couple), just exposing what's already there (for most of the suggestions).
Most Cloudron users are likely running on at least a 4 GB VPS, probably 8 GB or even 16 GB is the sweet spot depending on how many apps they're running.
As an example on memory... running the Mastodon app is a 1 GB requirement for example, and I'm betting that more users use Cloudron for email than running the Mastodon app, so presumably there'd be a lot of people happy to dedicate 1 GB of memory to additional features to managing their emails better.
Again though, I don't even think additional resources are even necessary here for most of the requests from Cloudron users, unless adding an AV scanner is important (personally I don't think AV is needed but it would be a nice-to-have as a feature we can enable or disable).
I think what most of us are asking for are simply exposing existing functionality through the GUI, and a little bit of new functionality perhaps such as queue management and rate limiting. That's about it. As a result, I'm confused by the recent comments for some off-box spam solution. Again, not against that as an additional feature - more options are a good thing, but it should not be the solution as it doesn't really address what most of us have been asking for, IMO.
My fear is we're looking to off-box spam control entirely and I don't think many of us are wanting that to happen, at least not as the only solution to improving spam control.
I'm a little worried we may be getting off track with the request for more mail server control versus what's being proposed, but maybe I've just misunderstood the more recent proposals (in which case I apologize for any confusion).
I'm definitely not against a third-party service to integrate with if others can take advantage of it (I don't think I could unfortunately but that's fine) and it improves things. But what's being requested by active users are things that should be ideally all local to Cloudron without the need for a third-party service.
What's being asked for by users isn't really any different than what other "all in one" mailbox platforms/solutions have been offering for many years (in some cases decades)... such as the following:
Admins of a mail server should:
Now I know the last few comments have been more specific to spam control, and a few of the ones above are also on spam control and really should all be done local to Cloudron (such as exposing Spamassassin configurations, greylisting capabilities, editing the denylists and allowlists, etc.). I just hope that with the recent proposals to use some sort of proxy to offload to a third-party spam service, that we're not offloading these user requests and chalking it up as done. I hope that we're still going to be doing spam filtering local to Cloudron, and that more controls are coming local to Cloudron too. This is most important for those who aren't even able to use a third-party service that views the messages if the service provider doesn't operate in the country needed for the data processing to be done in (such as healthcare providers who generally need to host their data in the country they operate in and remove as many services from the equation that don't operate in the same country).
My two cents. And again, my full apologies if I've misconstrued the direction this is going in, I just wanted to make it clear that I think most of us (if not all of us) still would like these improvements we've requested to be made local to the Cloudron app rather than offloading to a third-party service. With that said, a third-party service would be a nice addition - but it should be just that, an addition rather than a replacement.
@zjuhasz Shouldn't it just be backup/snapshot/app.*.tar.gz.enc (one asterisk)?
@girish Yes, it seems all hardlinks were created properly this time. I'll let you know if it fails again, for sure. Definitely seems like an odd issue and at first I assumed it was my infrastructure because of the other thread back in April, but this seems at this point to be more of a Cloudron issue. I can't quite get past the fact that this always is an issue on emails too, it's never on a file in an app or anything like that, always email. Maybe a coincidence since mail is also taking up most of the disk space, but still... just very strange to me. Hopefully this can be fixed in a future release once we know the root cause. Thanks again for the help to get me out of that jam.
@girish Okay I can report that did the trick in terms of at least getting a proper full backup now, so I have at least one backup. It was 17 GB in size. Thanks for that suggestion.
My ext4 hard disk is not mounted with CIFS or NFS, so I'm not certain that's a factor in this case. Here is what the line is in /etc/fstab:
/dev/vdc /cloudron-backups ext defaults 0 2
Since this worked fine after switching from no-op to Filesystem again (although I had also rm -rf * the /cloudron-backups/ directory before that too), I think this lends more evidence to this being an odd Cloudron issue (or at the very least a rare inconsistency/compatibility issue) rather than an infrastructure issue.
Thoughts?
@girish Ah okay that'd explain why it isn't backing it all up right now. I'll do that quickly and test. And yes this is rsync. I use rsync with hardlinks to save disk space and to speed up the backup process.