How to proctect instances from Bot, Crawlers, Requests, & Co?

[p44]

Only recently I realized that about 20% of my Cloudron instances’ resources are being “given away” to various parties.

Once upon a time, only search engines accessed sitemaps, but now web-facing instances – like WordPress – are constantly bombarded.

At this point, I took action to block these requests, and where possible, I worked at the .htaccess level (again, WordPress).

Nevertheless, I realized that it might be better to take a centralized approach and have a single point of control.

I don’t want to be forced to use external applications (like Cloudflare).

How could this aspect be improved on Cloudron?

This consideration might also be useful for adding one or more feature requests to Cloudron, given how the web is evolving, to improve existing blocking features.

In the meantime, I had thought of using Fail2Ban and setting rules to read the various logs of specific installed apps, and from there, setting limitations.

I’ve already read about all limitations about Fail2Ban on Cloudron, but, for example, in WordPress, I would block all 404 requests originating from requests like xyz.php pages. Or I would block access from very aggressive bots like AhrefsBot, Semrush, MJ12bot, Sentibot.

I’d be interested in understanding how you block anomalous requests centrally (not on individual apps).

Thanks a lot for your patience.

[timconsidine]

Interesting questions. I shall be watching hing for answers - wished I had them.

[p44]

@timconsidine Great, I’m glad that I’m not alone. I also saw your posts in this discussion related to specific problem of DDOS attacks.

I started to approach to this problem examining how VPS resources are “wasted” on daily bases when migrated from bare metal to VPS... In some peaks, I had a connection timeout on incoming 25 port, and then slowly I saw what was going on... most of accesses on that time they weren’t “human”...

[fbartels]

If the bots are compliant to it https://en.wikipedia.org/wiki/Robots.txt would be the tool you are looking for. This file can already be managed through the Cloudron UI.

When it comes to preventing bad actors then https://docs.crowdsec.net/ could be worthwhile to look into.

[p44]

@fbartels Yes, Robots.txt, .htaccess, all good... but it could be great to manage rules in a central (and simple) way, special on Cloudron instances with multiple apps installed.

It seems to be little bit complicated for my skills. I had a look on this post.

Are you using Crowdsec?