Is there a way to add in more DNSBL / RBL sources?
-
-
Example: Recently had an IP of
81.30.27.198send clear spam to my server, and it's blocked on several lists already but not yet on Spamhaus.Having an additional one for maybe SpamCop or SORBS (though in my experience SORBS tends to be over-aggressive - though I could be mixing that up with SpamCop, one of them is over aggressive lol).
https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3A81.30.27.198&run=toolpage
If this isn't possible (I suspect it's not) then I'll file a feature request. I have a bunch of mail server improvements I hope we can see in 7.0 as there's so much room for improvement in the mail server functionality, some of which has been requested for quite a long time now.
-
The list is hardcoded - https://git.cloudron.io/cloudron/box/-/blob/master/src/mail.js#L371 . The list is derived from https://raw.githubusercontent.com/jawsome/node-dnsbl/master/list.json . One danger of adding lesser known lists is that they suddenly disappear and get replaced by ad pages. I am happy to add any missing popular provider though.
-
@girish Ah interesting, thanks for sharing that link. A couple of things then:
-
I've only ever seen mail blocked in the logs by Spamhaus Zen, never any others. Curious if that's seen by others in there too. Wondering if there's an issue there at all.
-
I'm wondering why this message got through for example when it's listed on so many blocklists (seems 3 or 4 of which from the screenshot above are present in the Cloudron configuration then). I suppose it's technically possible it came in before being added to the blocklists, but I really doubt this is the case as I looked this up just minutes after it was in the logs and it already showed all those blocklists with it.
I'm starting to wonder if there's a possible defect here where the other lists aren't actually being checked? Mostly because this message still passed, and also I've yet to ever see anything blocked by anything other than Spamhaus Zen in the mail logs, so I'm thinking the others may not be working properly.
Example of recent "blacklisted by" items in the logs:
I think we may have uncovered a possible bug here.
PS - I agree being hesitant to add any lesser-known ones as they tend to be "fly by night", but the UCEPROTECT, SORBS, and SpamCop have been around forever for example, and thankfully you have those added in Cloudron which I didn't realize before, but I still haven't seen any of those in the logs for any messages blocked from being processed further in Cloudron (see image above). I'm actually happy to see the current configuration you have as it's a solid list (if anything I'd have actually only enabled two or three to avoid false-positives, haha, so I think your configuration is even more aggressive than I was wanting, but that's okay for now since I haven't seen any false-positives yet - though again I haven't even seen the others actually work yet either).
-
-
@d19dotca Ah, I see your confusion. The list is posted is only for the DNSBL checks in the UI (mail -> status). It doesn't affect mail delivery, it's really just a check to inform the user and nothing else.
As for accepting mail, we only check against Spamhaus Zen in the mail server. If it's black listed, then we delay the rejection until the user logs in. Sometimes, you will see in the logs that the IP is blacklisted but then it proceeds to accept email . This is because the user logged in and when a user logs in, we ignore the spamhaus checks (because most residential and user IPs are blacklisted anyways).
Zen has worked out pretty well so far since it's very actively maintained. I don't really know much about the lists but if we add it to the mail server, then we have to be 500% sure it's reliable because atleast for support rejecting email is way more work than accepting some junk
-
-
@girish Oh, I see, so the others are checked for adding in headers to identify it as spam, but only Spamhaus Zen is actually used for blacklisting it completely, right?
So I guess that's back to my original question then (but I guess you've answered it now), is how to add more than just Spamhaus Zen for the connection blocking.
Ultimately on a related topic... I'm focusing on my mail server and see so many instances of clear abuse of the spam and the Cloudron server doesn't seem to be doing enough, IMO here as the list of sources to check for denying access isn't customizable for example, and plus the other area I've seen and filed a feature request for already which is to block messages identified as spam from being forward on to mailing list recipients. Basically my server is handling far more spam that it should be, IMO, and it'd be great to see more customizations here. I'll file a feature request for it though
Thanks for clarifying the current behaviour, Girish.
-
@girish A LOT slip through, haha, yeah. I mean most get filed correctly to the Spam folder and maybe 5% passes to the inbox, which is reasonable IMO, but there are too many instances where the mail server is processing clear spam and passing it forward still when I'd prefer that it just be outright denied. My bigger concern is for the mailing list ones as I'm fearful Apple and Google and Microsoft will eventually blocklist my own server IP from all the mail that's being forwarded to their addresses which are clear spam. There are instances already of Google not blocking but greylisting my server because of the spam volume. There's lot of instances where I'm looking at the IP for a spam assault on my server haha and they're blacklisted, but not in Spamhaus Zen yet so they pass through still.
-
@d19dotca good points. I think allowing setting custom bl servers is complicated. I prefer adding them to the code itself if those lists are reliable. Haraka supports adding multiple servers, so we can easily add it if we can identify another reliable source. For example, I thought spamcop was very reliable and in fact just 2-3 months ago, the site disappeared (and then re-appeared).
But I will look into not forwarding emails marked as spam. For some reason, I thought we already did this since I remember reading gmail policy about this.
-
@girish
+1 for not let add additional DNSBL.i think that DNSBL check is good, but for example, spamhaus.org have multiple modules that you can use, some DNSBL use custom score, so I think that let people add list as they like just create more mess then use just a good one.
And you should add them also to the unbound server so that he can cache the requests.
-
@d19dotca
i advise you to update regularly your custom rules on SpamAssassin to add new words used and improve your filtering base on your own experience with spam.You can also use a Domain provider that protects your email and contact data on whois, which is the main cause of spam.
Matteo. R.
Founder and Tech-Support Manager.
MooCloud MSP
Swiss Managed Service Provider -
@moocloud_matt I think you may misunderstand a couple of items here in my messages. I'll try to clarify below.
update regularly your custom rules on SpamAssassin to add new words used and improve your filtering base on your own experience with spam
The SpamAssassin rules are fine, they properly identify spam and the vast majority of spam make it into the spam box away from the inbox. Identifying spam inside of SpamAsssin isn’t the issue in this case.
You can also use a Domain provider that protects your email and contact data on whois, which is the main cause of spam
This is something I already do currently when possible, but this is also not the issue here because that spam would be going to me then in that case as it's my contacts on the WHOIS. In my case here, it's my clients being sent the spam. It's basically an attack as far as I can tell on my mail server given the domains that point to my same IP so they just fire away at my server hoping to get some. In some cases, it's definitely a targeted recipient, but it's pretty generic and nothing to do with WHOIS spam in this case.
The issue I'm raising is basically two-fold...
-
that there's currently no way to prevent messages marked as spam by SpamAssassin to mailing list recipients, and
-
that there's no way to deny message processing either with custom lists. Right now Spamhaus Zen is the only list that mail gets outright denied from, despite there being plenty of other lists that marked these mail servers as spammy.
I believe an administrator should have the ability to manually add in new checks though to have it denied just like it currently does for Spamhaus Zen. I don't see any reason why that shouldn't be enabled.
I'm not wanting Cloudron to make new blocklists mandatory in code where a bunch are used to deny connections, that's agreeably bad, but we should at least have the option. In fact, while I’d certainly enable Spamhaus to deny messages outright, some mail admins may not want to be denying mail at all and it’s a little weird to me that even that is forced in code, honestly. Running a run a mail server is unique to everyone, and that’s why there needs to be customization abilities. I ran a mail server for a few years before moving to Cloudron, and before Cloudron when I was running my own mail server spam was practically never an issue because I took a lot of time to set it well to suit my clients needs and stayed on top of it. Unfortunately much of what I did on my mail server before Cloudron can’t be done in Cloudron.
-
As seen above, my server is frequently getting "spam assaulted" from a few mail servers in particular that rotate every so often. Spamhaus Zen gets some of them but not many. The ability to add in SORBS for example would be a huge help on my server to outright deny processing of the message. Currently the only workaround I have is to manually add in the IPs of the mail servers to the Network IP Blocklist function, but this is an ever-rotating list I'd basically have to update every few days manually.
-
Also, since currently half my clients just have mailing lists setup to forward to their own personal email address, and Cloudron doesn'tr currently prevent messages identified as Spam from just the SpamAssassin headers to be denied forwarding, it continues onwards to the Gmail.com or iCloud.com address for example which is not good and easily avoidable if we had the ability to prevent forwarding spam messages, or the ability to add in custom DNSBL checks to outright deny the message to begin with.
Ultimately, my problems would be solved with better customization for spam filtering. SpamAssassin rules alone are insufficient due to the inability to prevent messages identified as spam onwards to mailing lists.
-
-
@d19dotca For SORBS (which seems to be reliable), can you try this:
- docker exec -ti mail /bin/bash
- edit /run/haraka/config/dnsbl.ini
- Change zones to "zen.spamhaus.org;dnsbl.sorbs.net"
- supervisorctl restart haraka
This won't survive restarts, but it will give us a good idea of how effective this is. I put this in our cloudron.io mail server as well (but then again, we don't really get that much spam).
-
@girish Certainly, I’ll try that today and see how it goes and will report back.
Thanks for being open minded about this.Side note: I know you wrote above earlier that you want to control this list in code, but that part I actually would respectfully disagree with. Example: some mail server admins never want to deny anything and simply classify as spam and deliver - but that’s also not possible currently given it’s hard coded. Conversely, others may want it just a tad more aggressive at denying connections (such as me especially during “spam attacks” like the one that started a few weeks ago on mine) and I can’t add any new lists. I’d hope that if it’s as easy as running those commands to change the behaviour that we’d be able to expose that in the UI.
-
@girish Quick update: I changed from SORBS to SpamCop and am trying again, as I already found a false-positive when on SORBS. I checked the IP and it was basically only on SORBS and Backscatter, not SpamCop which means it would have passed as expected. I think this jives with what I thought earlier too but couldn't remember which one, I recall one of them was a bit too aggressive in years past as it often would block even Gmail and Hotmail mail servers which is just not feasible to do since so much legit email comes from them too.
I think it was both SpamAssassin and SpamCop I used on my mail server before Cloudron, so I've set it accordingly now. It now reads
zen.spamhaus.org;bl.spamcop.netfor the zone. -
Relevant but slightly off-topic, but wanted to share: https://www.intra2net.com/en/support/antispam/index.php
That list essentially monitors many DNSBLs for effectiveness and inaccuracies too (false-positives) using their own network for running the tests on. I find it quite interesting and stumbled into it today again, and I remember seeing it many years ago too. It's always up-to-date data which is interesting.
-
@d19dotca
ok,
that's the main reason that pushes us to use a centralized mail gateway, having control over incoming and outgoing traffic is fundamental for provider, and learning+settings are easier to do.
Cloudron with Haraka can't be a replacement of good email proxy or antispam, if you think all the service to prevent spam been send or received are some kind of proxy, for example, rspamd is build to have a demon on the mail server but all the elaboration is done in an external server.I'm sure that with better setup of DNSBL, URIBL,DCC, and SURBL will be better, but will not resolve the issue and make the setup harder for newbies.
For the fwd issue use a sieve forward from imbox this should prevent email marked as spam to been sent out.
Sorry if miss some point, I'm in paternity leaving so sleep is not a thing. (not for a baby. for now just an adorable husky that doesn't understand that he can sleep at night)
-
@moocloud_matt Congratulations on the new baby, that's awesome news!
will not resolve the issue and make the setup harder for newbies
I don't think I agree with that. Nothing will "resolve the issue" of spam itself (if that's what you meant by "the issue"), spam will never be 100% blocked and there will always be false-positives too. The goal is simply to reduce the level of spam and reducing the level of false-positives (or at least keeping it at an acceptable level), and that's where the extra customization comes into play.
I also don't think it will make it harder for "newbies" at all, because the out-of-the-box Cloudron setup would not change (at least I don't envision it would). Having the ability to set extra DNSBL checks for denying messages before they get processed shouldn't make anything harder for anyone - I should be able to setup a new Cloudron instance just as easily as I can today. Nothing should change there. Only the option to add new DNSBL checks to deny messages for example would be added as a completely optional feature to enable - it'd basically only be touched by "power users" and actual mail administrators who are comfortable making those tweaks and already looking to make such changes in the first place.
I agree though that there's plenty of different ways to improve spam filtering and this is just one of many possible ways that I hope to see (and many others from the community too judging by how many mail improvements / feature requests exist in the Cloudron forum).
For the fwd issue use a sieve forward from imbox this should prevent email marked as spam to been sent out
That is an interesting approach, and I'll consider it. My first thought though is... while it may technically be a valid workaround, in my case I don't think this option is feasible though as I have too many accounts to do this for. I have roughly 20+ recipients on my server who only are setup for mailing lists to forward to their own personal email accounts on common domains (no mailboxes on Cloudron). This workaround means I'd need to setup about 20+ mailboxes, not only that but also set them all up consistently and accurately. This leaves a lot of room for human error in my case and a lot of overhead if I ever wanted to make a quick tweak and keep it consistent across them all. If I only had a few, that'd be no problem, but I think I have too many for that to be feasible in my case, unfortunately. I appreciate the thought there though, I hadn't really considered that as a possible workaround before.
