Firewall / Spamassassin: Automatic list update

[robi]

The output from mail-tester.com provides ~20 different RBL/SBL lists to check. Could be useful to integrate if they have an API.

[rmdes]

@girish There is the risk those lists are too broad, I just had to remove taiwan zone because one of our forum users could not access it.

I think if we ever have automatic list update, the source should be a vetted spam ip list not just IP list blocking entire countries..

[rmdes]

@girish @nebulon This repo here seems to be interesting to build on this feature request :
https://github.com/firehol/blocklist-ipsets
official site :https://iplists.firehol.org/

[MooCloud_Matt]

@necrevistonnezr said in Firewall / Spamassassin: Automatic list update:

Spamassassin lists from Heinlein

I think that this config file is mostly oriented on German email, probably if an incoming email is in Italian or French or English wold be completely useless, this is one of the biggest issue in spam protection and having a 0% ham.

You need custom rules for every language and if you use Rspamd you also need different AI/db for every language.

[necrevistonnezr]

@moocloud_matt said in Firewall / Spamassassin: Automatic list update:

@necrevistonnezr said in Firewall / Spamassassin: Automatic list update:

Spamassassin lists from Heinlein

I think that this config file is mostly oriented on German email, probably if an incoming email is in Italian or French or English wold be completely useless, this is one of the biggest issue in spam protection and having a 0% ham.

You need custom rules for every language and if you use Rspamd you also need different AI/db for every language.

I don't think so. Looking into their body cf file, you see entries like this, I'd say more than half is English:

body HS_BODY_2021		/Today is the last day to order a custom print for Christmas delivery/
describe HS_BODY_2021		Heinlein Support Spamschutz Body-2021 Spam
score HS_BODY_2021		10

body HS_BODY_2023		/Need a little help finding the right piece? Our curators are here to help/
describe HS_BODY_2023		Heinlein Support Spamschutz Body-2023 Spam
score HS_BODY_2023		10

body HS_BODY_2026		/Shop now - no minimums or promo required. Sale ends Tuesday/
describe HS_BODY_2026		Heinlein Support Spamschutz Body-2026 Spam
score HS_BODY_2026		10

body HS_BODY_2028		/globalgallery.us5.list-manage.com/
describe HS_BODY_2028		Heinlein Support Spamschutz Body-2028 Spam
score HS_BODY_2028		10

body HS_BODY_2029		/ausmisten, die Ablage wegsortieren, den Jahresurlaub buchen oder auch/
describe HS_BODY_2029		Heinlein Support Spamschutz Body-2029 Spam
score HS_BODY_2029		3

body HS_BODY_2032		/Ciao!  https:..betcb.com/
describe HS_BODY_2032		Heinlein Support Spamschutz Body-2032 Spam
score HS_BODY_2032		5

body HS_BODY_2037		/If you ignored this email your account will be Officially Permanently disabled the next/
describe HS_BODY_2037		Heinlein Support Spamschutz Body-2037 Phishing
score HS_BODY_2037		5

body HS_BODY_2040		/from AppleID./
describe HS_BODY_2040		Heinlein Support Spamschutz Body-2040 pHISHING
score HS_BODY_2040		5

body HS_BODY_2043		/http\:\/\/datingx\.co/
describe HS_BODY_2043		Heinlein Support Spamschutz Body-2043 Spam
score HS_BODY_2043		5

body HS_BODY_2045		/as one of the final recipients of the Mega million Bonanza funding/
describe HS_BODY_2045		Heinlein Support Spamschutz Body-2045 Spam
score HS_BODY_2045		5

body HS_BODY_2048		/A Sophisticated Automated Database to Randomly select/
describe HS_BODY_2048		Heinlein Support Spamschutz Body-2048 Spam
score HS_BODY_2048		5

body HS_BODY_2050		/ We Embarked on a worldwide promotion for Disabled, Employed and Unemployed Workers, Retired, Young and Old people/
describe HS_BODY_2050		Heinlein Support Spamschutz Body-2050 Spam
score HS_BODY_2050		5

body HS_BODY_2051		/the On-line director of the Mega millions Bonanza funding imposed by the United State Government/
describe HS_BODY_2051		Heinlein Support Spamschutz Body-2051 Spam
score HS_BODY_2051		5

body HS_BODY_2052		/Ihr Paket ist gerade in unserer Zentrale eingetroffen, aber wir k/
describe HS_BODY_2052		Heinlein Support Spamschutz Body-2052 Spam
score HS_BODY_2052		3

body HS_BODY_2053		/Content-Disposition: attachment; filename=DHL.*PDF.iso;/
describe HS_BODY_2053		Heinlein Support Spamschutz Body-2053 Phishing
score HS_BODY_2053		5

body HS_BODY_2054		/http.*\.icu\/ub\.php\?/
describe HS_BODY_2054		Heinlein Support Spamschutz Body-2054 Spam
score HS_BODY_2054		2

body HS_BODY_2056		/http...www.db-onlinemarketing.net/
describe HS_BODY_2056		Heinlein Support Spamschutz Body-2056 Spammer
score HS_BODY_2056		4

body HS_BODY_2059		/Firmendatenbank GC-Contact/
describe HS_BODY_2059		Heinlein Support Spamschutz Body-2059 Spam
score HS_BODY_2059		2

body HS_BODY_2061		/Eine Investition in die Firmenadressen macht sich sofort bezahlt. Sie erwerben das Nutzungsrecht am kompletten Adressenpaket./
describe HS_BODY_2061		Heinlein Support Spamschutz Body-2061 Spam
score HS_BODY_2061		2

[d19dotca]

@rmdes said in Firewall / Spamassassin: Automatic list update:

source should be a vetted spam ip list

My two cents... Cloudron should not be responsible for vetting the list. It should be (ideally) as simple as admins enabling/disabling lists that are pre-packaged by Cloudron in case they can't just allow any dataset to be used, or we'd be able to throw in our own links to files updated by various vendors such as those from Firehol for example.

I think that's what you meant, but wanted to clarify in case, as I would hate to see Cloudron being responsible for doing any kind of manual vetting, that should definitely be on admins to do. Cloudron just needs to allow access to the lists and we then go from there as admins.

[d19dotca]

There's even a list for Cloudron team to use for this forum haha

https://iplists.firehol.org/?ipset=stopforumspam

Now imagine if that could be used in the firewall automatically. Would be awesome.

[rmdes]

@d19dotca Yes that's what I meant, hence the lists I suggested: https://forum.cloudron.io/post/20010

[rmdes]

The more I read about FireHol the more I wish this was backed into cloudron install directly, it seems to me that Firehol is a great source to rely for blocking bad IP's

[necrevistonnezr]

@girish
Did anything come of this...?

[necrevistonnezr]

Can we re-visit this?

[necrevistonnezr]

I guess since the blocklist is in /home/yellowtent/platformdata/firewall/blocklist.txt, one could build something with a script & cron?

[necrevistonnezr]

@girish is the way I described feasible? Is that txt file the actual list the firewall accesses to check blocked IPs or is this txt file e.g. used to feed into a database?

[girish]

@necrevistonnezr Updating the txt file is not enough. The txt file is actually just a "cache" , the real value is stored in the database.

[necrevistonnezr]

@girish Ok, so simple scripting is out of the question.

[girish]

@necrevistonnezr you can still use the api though

[necrevistonnezr]

Well, the "setBlockList" operation allows to add a range of IPs but not a list of IPs in a file or am I wrong?

curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $CLOUDRON_TOKEN" "https://$CLOUDRON_DOMAIN/api/v1/network/blocklist" --data '{"blocklist":"# Spammy network\n10.244.0.0/16"}'

as per: https://docs.cloudron.io/api.html#tag/Network/operation/setBlockList

[girish]

@necrevistonnezr it's a "Newline separated list of IP entries" . So, it can be # Spammy network\n10.244.0.0/16\n1.2.3.4\n3.4.5.6\n172.4.0.0/16

[necrevistonnezr]

@girish I guess there's no mechanism to avoid duplicate entries when using the "setBlockList" operation, correct?

In general, I guess something like this should work:

#!/bin/bash
curl https://www.ipdeny.com/ipblocks/data/countries/kz.zone --output iplist.txt
while read -r line; do
    curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $CLOUDRON_TOKEN" "https://$CLOUDRON_DOMAIN/api/v1/network/blocklist" --data $line"
done < iplist.txt
rm iplist.txt

I don't have know yet how to avoid duplicates in the database..

[imc67]

It should be default functionality to have country block/allow in the Cloudron GUI just like all Synology NAS’s have. It’s 2023 and too dangerous to have everything accessible for everyone. That’s why many Cloudron users (read the forum) are using Cloudflare for this kind of functionality (like I have to do).