How to proctect instances from Bot, Crawlers, Requests, & Co?

[p44]

Only recently I realized that about 20% of my Cloudron instances’ resources are being “given away” to various parties.

Once upon a time, only search engines accessed sitemaps, but now web-facing instances – like WordPress – are constantly bombarded.

At this point, I took action to block these requests, and where possible, I worked at the .htaccess level (again, WordPress).

Nevertheless, I realized that it might be better to take a centralized approach and have a single point of control.

I don’t want to be forced to use external applications (like Cloudflare).

How could this aspect be improved on Cloudron?

This consideration might also be useful for adding one or more feature requests to Cloudron, given how the web is evolving, to improve existing blocking features.

In the meantime, I had thought of using Fail2Ban and setting rules to read the various logs of specific installed apps, and from there, setting limitations.

I’ve already read about all limitations about Fail2Ban on Cloudron, but, for example, in WordPress, I would block all 404 requests originating from requests like xyz.php pages. Or I would block access from very aggressive bots like AhrefsBot, Semrush, MJ12bot, Sentibot.

I’d be interested in understanding how you block anomalous requests centrally (not on individual apps).

Thanks a lot for your patience.

[timconsidine]

Interesting questions. I shall be watching hing for answers - wished I had them.

[p44]

@timconsidine Great, I’m glad that I’m not alone. I also saw your posts in this discussion related to specific problem of DDOS attacks.

I started to approach to this problem examining how VPS resources are “wasted” on daily bases when migrated from bare metal to VPS... In some peaks, I had a connection timeout on incoming 25 port, and then slowly I saw what was going on... most of accesses on that time they weren’t “human”...

[fbartels]

If the bots are compliant to it https://en.wikipedia.org/wiki/Robots.txt would be the tool you are looking for. This file can already be managed through the Cloudron UI.

When it comes to preventing bad actors then https://docs.crowdsec.net/ could be worthwhile to look into.

[p44]

@fbartels Yes, Robots.txt, .htaccess, all good... but it could be great to manage rules in a central (and simple) way, special on Cloudron instances with multiple apps installed.

It seems to be little bit complicated for my skills. I had a look on this post.

Are you using Crowdsec?

[jdaviescoates]

I always install Wordfence on all my WordPress sites. Blocks most stuff.

[joseph]

Is there a service out there that provides AI block lists? crowdsec only has a platinum list at 3500/month ?

[p44]

@jdaviescoates @joseph Thanks, I wouldn't want to rely on outside services.

[jdaviescoates]

@p44 TBH wrt WordPress i'd expect Wordfence would likely do a much better job that you'd ever be able to do manually. They have very long blocklists and know about many more bad IPs than you do. I just use the free version.

[p44]

@jdaviescoates Thanks, it seems that they don’t block AI bots...

[robi]

Can you extract the blocklists from Wordfence to have it populate the Cloudron deny list?

[p44]

@robi Thanks for advice.

I don’t know where and if Wordfence has a public list, but I think that blocklists has a lot of data that can be huge to handle from CPU.

[robi]

@p44 those are solved problems with lookup tables.

[jdaviescoates]

@p44 I think Wordfence adds stuff to .htaccess

[p44]

@robi Thanks a lot.

I found and applied specific rules in Wordpress .htaccess:

• 8G Firewall
• Ultimate Block List to Stop AI Bots

I think is a good start.

Both filters it seems working fine. Of course, it would better to manage and deploy centrally.

Thanks again for your advices Robi.

[robi]

@p44 You are very welcome.

Now making a tool to parse those IPs for the Cloudron block list is something an eager LLM agent could do.

[p44]

@robi Amazing!!!

Something like a “bridge” that every day take from the source and update Cloudron block list...