Entering too many IP addresses to be blocked at network level by Cloudron results in error: "setBlocklist (stderr): ipset v7.15: Hash is full, cannot add more elements"

[d19dotca]

I have a client site that's getting a ton of spammy user registrations which I'm finding difficult to clean up but more importantly preventing it in the first place.

I know there are a few spam-related plugins that could help, but I figured since it may affect other areas too like email spam and other client sites seeing as these IPs are hijacked / maliciously used, I would try to use the StopForumSpam IP blocklist to block traffic (I compared the IPs that are hitting the registration endpoint and they exist on the list.

However when I try to paste these IPs and save it, I get an error which shows as this in the logs (and the IPs are not added):

Nov 29 00:07:06box:shell setBlocklist (stderr): ipset v7.15: Hash is full, cannot add more elements

[d19dotca]

Correction: The IPs do seem to save, but took a refresh of the page to see that confirmation.

In the UI I now see the following as I'd expect:
Blocked IPs & Ranges 122095 IP(s) blocked

[d19dotca]

For the record, the three main IPs I'm seeing frequently are the following and all present as blocklisted at CleanTalk and StopForumSpam resources:

• 5.42.64.62
• 37.139.53.17
• 46.161.15.14

I mention this in case others are seeing this too. May be a good reminder to update those network block lists.

[girish]

Looks like this has to do with the ipset being created with too small a hashsize. Investigating, what we can do here. Looks like if you add a lot of IPs, it will take a LOT of memory.

[girish]

@d19dotca said in Entering too many IP addresses to be blocked at network level by Cloudron results in error: "setBlocklist (stderr): ipset v7.15: Hash is full, cannot add more elements":

Blocked IPs & Ranges 122095 IP(s) blocked

I started looking into this and something seems off. https://raw.githubusercontent.com/X4BNet/lists_stopforumspam/main/ipv4.txt itself only has only 4267 addresses. I guess you are adding lot more addresses? But 120k addresses seems like a lot.

[girish]

Even https://iplists.firehol.org/ has only 1890 entries or so. Blocking those many IPs seems a bit off.

[d19dotca]

@girish said in Entering too many IP addresses to be blocked at network level by Cloudron results in error: "setBlocklist (stderr): ipset v7.15: Hash is full, cannot add more elements":

I started looking into this and something seems off. https://raw.githubusercontent.com/X4BNet/lists_stopforumspam/main/ipv4.txt itself only has only 4267 addresses. I guess you are adding lot more addresses? But 120k addresses seems like a lot.

Oh that’s a great catch! I’m not sure how that happened. Let me test again with clearing out the setting entirely and saving blank, then pasting them in again and see what it says.

[girish]

@d19dotca One thing I found is that when adding the stopforumspam list and the firehol list together, my system gets hosed! I will debug this further tomorrow. As in, some iptables/ipset issue. Maybe this needs a lot more memory at kernel level.

[JLX89]

Just throwing this out there for what it is worth... I started experimenting, but got side tracked a while back. But I was looking at using CrowdSec to block traffic. There is at least a few Firhol lists and some other items.

It would be interesting to create one for forums and consolidate data from some of the other sources to create a blocklist. I did this to offset the "blocking" to an outside source, so that my server didn't quite take the hit.

[necrevistonnezr]

@JLX89 There‘s this guide: https://forum.cloudron.io/topic/6224/crowdsec-install-guide-for-cloudron-purposes but I‘m not sure it‘s still up-to-date - and I‘d very much prefere to have such functionality built-in

[JLX89]

@necrevistonnezr Yeah, I've seen that before. I'm looking a few things over, hopefully today -- see what I can come up with.

[girish]

OK, i have debugged this now. The issue was that Firehol lists includes both 127.0.x.x and the 172.16.x.x networks which ends up blocking database and docker access . Commenting out those two works fine.

So, @d19dotca the issue was really just that you have somehow pasted 120k addresses. The max size of ipset is 65536. I will put a validation check in place but otherwise there is no bug.

[robi]

@girish can the validation check include those internal Cloudron addresses too?

[girish]

@robi good idea. But just digging deeper, this looks like a bug in our code. Those IP addresses never appear from outside but looks like the firewall rule ends up getting applied for all network connections (internal and external).

[d19dotca]

@girish said in Entering too many IP addresses to be blocked at network level by Cloudron results in error: "setBlocklist (stderr): ipset v7.15: Hash is full, cannot add more elements":

The max size of ipset is 65536. I will put a validation check in place but otherwise there is no bug.

This may mean that using popular blocklists like this one won't work then (there is over 120k IPs, so I don't think it was a mistake earlier btw as you can see the number of unique IPs in the header comments on this list): https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/stopforumspam_90d.ipset

Perhaps this can be looked at for expansion in the future?

I was able to use their 30 day one (which still included the few IPs that seem persistent for my WordPress sites as trying to spam them) thankfully though as it's less than the limit in Cloudron.

[girish]

@d19dotca right, the 30d one seems to be 54975 size. I have increased the size of the ipset now to 262144 elements. If these things are growing more, we can look into making this size dynamic .