Cloudron email: feature improvements/ideas

[MooCloud_Matt]

@d19dotca Your questions/notes are all very interesting and relevant; Our problem lies in the optimization and management of resources.

Our problem lies in the optimization and management of resources.
To work properly if you control from Mailcow to iredmail they ask for 2GB to 4GB of RAM (I am not considering using more effective ClamAV databases, no OCR filters or anything else.).
This is something that almost none of our customers (moocloud customer not cloudron) can have, moreover there is a reason for the increasing use of proxy or MTAs by professional antispams instead of spamassasin plugins, spam must be fight together, collaboratively and this is not possible if you install an antispam on each server.

What MooCloud has proposed to cloudron (this is one of our 2 proposal) is to offer integration with a popular antispam proxy, we are talking about proxmox gateway that you can all download and install and use to manage all your incoming mail.
What MooCloud will do is offer this package/proxy with additional filters, various optimizations to improve effectiveness.
Proxmox come with many interesting functions already available as "Object-Oriented Rule System":

Rules:
ACTIONS - object: Defines what should happen with the email.
WHO - object: Who is the sender or receiver of the e-mail?
WHAT - object: What is in the e-mail?
WHEN - object: When is the e-mail received by Proxmox Mail Gateway?

Or the second proposal is to offer a set of tools that collaborates with the cloudron server, spamassasin and haraka to improve the score.
In this case we will ensure that only the email signatures are sent to our servers and not other information (This is 100% compatible with any privacy policy of any state.).
Obviously in this case if we could offer this service; Cloudron will allow others to enter the competition and in some time you will have the opportunity to take advantage of the choice of multiple providers.
This second setup, however remains that it is not as effective as the first solution and will take a long time to develop.
because it will take time to integrate Box (cloudron backend) with haraka, spamassasin, clamav demon, pyzor, geoip repository, sieve, etc. etc.
Remember that all the software you mentioned earlier took years to become usable and complete as they are now.

What I can tell you is that: I understand your concern about not being able to use certain services, but unfortunately antispam is something really complex to manage locally (in fact microsoft's exchnge no longer does antispam, and is the first mail server used my enterprise) and I am sure that any solution choosed by cloudron (among ours or others that will be made); if this needs to have external servers/service they will be distributed globally.

[d19dotca]

@MooCloud_Matt "optimization" and "management of resources" are important factors in anything for sure, but most of what is being asked for is just exposing already functioning components, so naturally those shouldn't really take any more resources. It isn't so much adding entirely new features that we're asking for (although perhaps a couple), just exposing what's already there (for most of the suggestions).

Most Cloudron users are likely running on at least a 4 GB VPS, probably 8 GB or even 16 GB is the sweet spot depending on how many apps they're running.

As an example on memory... running the Mastodon app is a 1 GB requirement for example, and I'm betting that more users use Cloudron for email than running the Mastodon app, so presumably there'd be a lot of people happy to dedicate 1 GB of memory to additional features to managing their emails better.

Again though, I don't even think additional resources are even necessary here for most of the requests from Cloudron users, unless adding an AV scanner is important (personally I don't think AV is needed but it would be a nice-to-have as a feature we can enable or disable).

I think what most of us are asking for are simply exposing existing functionality through the GUI, and a little bit of new functionality perhaps such as queue management and rate limiting. That's about it. As a result, I'm confused by the recent comments for some off-box spam solution. Again, not against that as an additional feature - more options are a good thing, but it should not be the solution as it doesn't really address what most of us have been asking for, IMO.

My fear is we're looking to off-box spam control entirely and I don't think many of us are wanting that to happen, at least not as the only solution to improving spam control.

[girish]

Don't mean to side track but there should not be problems exposing more controls (like bl/wl, mail queue etc). From technical point of view, we don't want to expose SA specific configuration too much because we want to keep the option for rspamd open. So a first pass is just exposing things that we already can do easily.

edit: missed an important word @d19dotca

[d19dotca]

@girish Definitely in agreement with that. I assume you mean "should not be" rather than "should be" though. haha But yes, exposing what's already working on the lower levels is mostly what people are asking for and that's what I'd like to see happen and I believe most users would say the same thing.

And that's understandable to be hesitant for now on opening up the SA config if you're pondering moving to rspamd down the road, I think most of us would be okay waiting a bit longer for that decision to be made and then investing the time on the solution identified.

[MooCloud_Matt]

@d19dotca
I'm not saying that some functions like blacklist and whitelist should not be included in cloudron, but these will not improve antispam effectiveness, they will make it more functional.

But to make it more effective you need some things, and one of them is Clamav, because many spam repositories are included in the clamav or other antivirus signature-repo.
For this there are repositories for ClamAV updated every hour with the largest and most common spam attacks at the time on the web.

And this becomes complex especially when you need to use networks like pyzor, razor or DCC.
This means that a server loses at least 2 GB and becomes complex to manage, because in order to function properly they need to be used on a network.

What I am convinced: is that @girish can make cloudron more functional, but to improve effectiveness you need to find a different solution.
This is not my belief, if you look at all the control panels they always offer external solutions or special addons for spam, because protecting from spam is a high cost, in term of time, money, resources.
We modify the words and phrases that spamassasin must observe more carefully every week, during the period of the covid pandemic almost every day.

[d19dotca]

@MooCloud_Matt I think I may see now where some of the confusion is coming from now. What you're talking about is effectively an entirely new functionality (integrating Cloudron with third-party spam services), and what most of the rest of us are talking about in this thread is simply asking for Cloudron to expose existing functionality to Cloudron admins so we can improve how our mail servers act for our users.

The title of this thread is very generic but I think it might be helpful if you started a dedicated thread specific to requesting the addition of third-party spam services and having Cloudron integrate with those APIs, because I think that's a very large topic all on it's own that may require a lot of discussion. Just my two cents anyways.

I'm certainly in favour of your suggestion for a third-party integration for spam filtering - the more options we have the better (as long as it's still maintainable for the Cloudron team) - although I respectively disagree with the implication that a third-party service is the only solution to improving spam filtering "effectiveness" in Cloudron. Plenty of large corporations (receiving far more mail than any of us Cloudron users likely are) deal with spam in-house, though admittedly they will also have far more resources to throw at it.

There is still a lot more we can do within the confines of what Cloudron already does to improve spam filtering effectiveness, we just need those functions exposed to us to better modify them to our liking.

[mehdi]

@girish said in Cloudron email: feature improvements/ideas:

we don't want to expose SA specific configuration too much because we want to keep the option for rspamd open

I completely understand your approach, but I do not think exposing the configuration more would prevent you from changing the spam engine later, as long as the documentation makes it clear that customizing said configuration may stop working later and that you only make the change in a major version.

[girish]

@mehdi Yeah, just trying to reduce some work for ourselves. I think we just need to take some time to evaluate rspamd

Does anyone here used rspamd in prod for long periods and willing to share their experience?

[d19dotca]

@girish I used rspamd a year or two ago as part of hardware/mailserver project on GitHub, it was great. With that said, I had not specifically installed rspamd since it came with that project (own container), but it pretty much just worked well out of the box, didn't have to do too much tinkering with it. It gave great reports and such too which I really liked, seeing how much of my email was deliverable, how much was flagged as spam (incoming) vs legitimate, greylisted, etc.

[jimcavoli]

Love the conversation this has kicked off, and I think there's a few good things to consider here:

• The main thing I was originally aiming at first was better visibility into the mail subsystem and second to that more control over those various components
• Expanding the capability of the mail system to include additional processing by farming out additional processing to external servers like CipherMail, Proxmox, etc. could be done easily for even sparsely-resourced servers. However, optionally enabling certain processing like Antivirus on the same machine is still likely a good idea.

A lot of us are also running fairly beefy servers for large production setups and it seems disingenuous to rule out completely more resource-intensive operations from the core packaging just because some machines at the lower end couldn't support them. Making them optionally available, even subject to certain resource constraints and so on, is exactly what I'm hoping we can get to - a lot more control over the mail system so that administrators can choose how much in terms of resources to invest in that system. A combination of more inspectable, optional services available on the core offering as well as allowing external smtp-based components to slot into the workflow sounds like the best of all worlds.

[paenzy]

A very nice feature for cloudron mail is a per user/mail signature.
With this feature you can managing all disclaimers central in cloudron and don't need to change each disclaimer in the client. All mails would have the same corporate disclaimer - no matter if the mail sends from outlook/thunderbird or mobil app.

In my opinion, a central managing company mail disclaimer for SMTP server is definitly a unique selling point.
(many companies dont't use exchange server)

What do you think?

[girish]

@paenzy currently, disclaimer can be set a domain level - https://docs.cloudron.io/email/#signature . BTW, this feature request post is very broad, feel free to open a specific request as a separate post. It's easier to track and vote.

[jimcavoli]

@girish To that end, perhaps this one is better locked at this point