Firewall / Spamassassin: Automatic list update

[necrevistonnezr]

@imc67 explained how to block IP ranges from certain countries
I think it would be useful if such lists (e.g. from https://www.ipdeny.com/ipblocks/) could be kept updated automatically, e.g. by running an update script regularly.

The same applies for spamassassin lists, e.g. the excellent Spamassassin lists from Heinlein who runs www.mailbox.org

[ruihildt]

We were talking with @rmdes about it, it would be the best to have a default updated list you can subscribe to instead of each Cloudron admin going over the same process on their own.

[jdaviescoates]

@ruihildt yeah, it'd be great if Cloudron could just subscribe to these lists for us.

[d19dotca]

@necrevistonnezr Fully support this, this would be awesome. Definitely been looking into this recently and realized there's no way to keep it updated automatically from lists like that in Cloudron. Some ability to enter a URL to subscribe to or even just have a few built-in ones that need to be enabled would be fantastic.

[girish]

Would love to see this as well. The license of ipdeny is also fairly permissive.

You may NOW re-distribute our zone files and you can freely use our generated IP zone
files in  your commercial or freeware solutions or services, please read our
Copyright policy and please comply with our Usage limits policy. Linkback to our
service is always appreciated and  recommended.

[girish]

BTW, for mail server, the mail server already uses zen spamhaus. This is enabled by default (and in fact, no way to disable this).

[robi]

The output from mail-tester.com provides ~20 different RBL/SBL lists to check. Could be useful to integrate if they have an API.

[rmdes]

@girish There is the risk those lists are too broad, I just had to remove taiwan zone because one of our forum users could not access it.

I think if we ever have automatic list update, the source should be a vetted spam ip list not just IP list blocking entire countries..

[rmdes]

@girish @nebulon This repo here seems to be interesting to build on this feature request :
https://github.com/firehol/blocklist-ipsets
official site :https://iplists.firehol.org/

[MooCloud_Matt]

@necrevistonnezr said in Firewall / Spamassassin: Automatic list update:

Spamassassin lists from Heinlein

I think that this config file is mostly oriented on German email, probably if an incoming email is in Italian or French or English wold be completely useless, this is one of the biggest issue in spam protection and having a 0% ham.

You need custom rules for every language and if you use Rspamd you also need different AI/db for every language.

[necrevistonnezr]

@moocloud_matt said in Firewall / Spamassassin: Automatic list update:

@necrevistonnezr said in Firewall / Spamassassin: Automatic list update:

Spamassassin lists from Heinlein

I think that this config file is mostly oriented on German email, probably if an incoming email is in Italian or French or English wold be completely useless, this is one of the biggest issue in spam protection and having a 0% ham.

You need custom rules for every language and if you use Rspamd you also need different AI/db for every language.

I don't think so. Looking into their body cf file, you see entries like this, I'd say more than half is English:

body HS_BODY_2021		/Today is the last day to order a custom print for Christmas delivery/
describe HS_BODY_2021		Heinlein Support Spamschutz Body-2021 Spam
score HS_BODY_2021		10

body HS_BODY_2023		/Need a little help finding the right piece? Our curators are here to help/
describe HS_BODY_2023		Heinlein Support Spamschutz Body-2023 Spam
score HS_BODY_2023		10

body HS_BODY_2026		/Shop now - no minimums or promo required. Sale ends Tuesday/
describe HS_BODY_2026		Heinlein Support Spamschutz Body-2026 Spam
score HS_BODY_2026		10

body HS_BODY_2028		/globalgallery.us5.list-manage.com/
describe HS_BODY_2028		Heinlein Support Spamschutz Body-2028 Spam
score HS_BODY_2028		10

body HS_BODY_2029		/ausmisten, die Ablage wegsortieren, den Jahresurlaub buchen oder auch/
describe HS_BODY_2029		Heinlein Support Spamschutz Body-2029 Spam
score HS_BODY_2029		3

body HS_BODY_2032		/Ciao!  https:..betcb.com/
describe HS_BODY_2032		Heinlein Support Spamschutz Body-2032 Spam
score HS_BODY_2032		5

body HS_BODY_2037		/If you ignored this email your account will be Officially Permanently disabled the next/
describe HS_BODY_2037		Heinlein Support Spamschutz Body-2037 Phishing
score HS_BODY_2037		5

body HS_BODY_2040		/from AppleID./
describe HS_BODY_2040		Heinlein Support Spamschutz Body-2040 pHISHING
score HS_BODY_2040		5

body HS_BODY_2043		/http\:\/\/datingx\.co/
describe HS_BODY_2043		Heinlein Support Spamschutz Body-2043 Spam
score HS_BODY_2043		5

body HS_BODY_2045		/as one of the final recipients of the Mega million Bonanza funding/
describe HS_BODY_2045		Heinlein Support Spamschutz Body-2045 Spam
score HS_BODY_2045		5

body HS_BODY_2048		/A Sophisticated Automated Database to Randomly select/
describe HS_BODY_2048		Heinlein Support Spamschutz Body-2048 Spam
score HS_BODY_2048		5

body HS_BODY_2050		/ We Embarked on a worldwide promotion for Disabled, Employed and Unemployed Workers, Retired, Young and Old people/
describe HS_BODY_2050		Heinlein Support Spamschutz Body-2050 Spam
score HS_BODY_2050		5

body HS_BODY_2051		/the On-line director of the Mega millions Bonanza funding imposed by the United State Government/
describe HS_BODY_2051		Heinlein Support Spamschutz Body-2051 Spam
score HS_BODY_2051		5

body HS_BODY_2052		/Ihr Paket ist gerade in unserer Zentrale eingetroffen, aber wir k/
describe HS_BODY_2052		Heinlein Support Spamschutz Body-2052 Spam
score HS_BODY_2052		3

body HS_BODY_2053		/Content-Disposition: attachment; filename=DHL.*PDF.iso;/
describe HS_BODY_2053		Heinlein Support Spamschutz Body-2053 Phishing
score HS_BODY_2053		5

body HS_BODY_2054		/http.*\.icu\/ub\.php\?/
describe HS_BODY_2054		Heinlein Support Spamschutz Body-2054 Spam
score HS_BODY_2054		2

body HS_BODY_2056		/http...www.db-onlinemarketing.net/
describe HS_BODY_2056		Heinlein Support Spamschutz Body-2056 Spammer
score HS_BODY_2056		4

body HS_BODY_2059		/Firmendatenbank GC-Contact/
describe HS_BODY_2059		Heinlein Support Spamschutz Body-2059 Spam
score HS_BODY_2059		2

body HS_BODY_2061		/Eine Investition in die Firmenadressen macht sich sofort bezahlt. Sie erwerben das Nutzungsrecht am kompletten Adressenpaket./
describe HS_BODY_2061		Heinlein Support Spamschutz Body-2061 Spam
score HS_BODY_2061		2

[d19dotca]

@rmdes said in Firewall / Spamassassin: Automatic list update:

source should be a vetted spam ip list

My two cents... Cloudron should not be responsible for vetting the list. It should be (ideally) as simple as admins enabling/disabling lists that are pre-packaged by Cloudron in case they can't just allow any dataset to be used, or we'd be able to throw in our own links to files updated by various vendors such as those from Firehol for example.

I think that's what you meant, but wanted to clarify in case, as I would hate to see Cloudron being responsible for doing any kind of manual vetting, that should definitely be on admins to do. Cloudron just needs to allow access to the lists and we then go from there as admins.

[d19dotca]

There's even a list for Cloudron team to use for this forum haha

https://iplists.firehol.org/?ipset=stopforumspam

Now imagine if that could be used in the firewall automatically. Would be awesome.

[rmdes]

@d19dotca Yes that's what I meant, hence the lists I suggested: https://forum.cloudron.io/post/20010

[rmdes]

The more I read about FireHol the more I wish this was backed into cloudron install directly, it seems to me that Firehol is a great source to rely for blocking bad IP's

[necrevistonnezr]

@girish
Did anything come of this...?

[necrevistonnezr]

Can we re-visit this?

[necrevistonnezr]

I guess since the blocklist is in /home/yellowtent/platformdata/firewall/blocklist.txt, one could build something with a script & cron?

[necrevistonnezr]

@girish is the way I described feasible? Is that txt file the actual list the firewall accesses to check blocked IPs or is this txt file e.g. used to feed into a database?

[girish]

@necrevistonnezr Updating the txt file is not enough. The txt file is actually just a "cache" , the real value is stored in the database.